Event Cognition: Psychology, Neurophysiology, and Just a Bit of Computation
Washington University in Saint Louis, MO, USA
Everyday activity is continuous, dynamic, and high bandwidth—yet we seem to have the subjective experience of a modest number of meaningful events that stand in structured relations to each other.
In this talk, I will describe a theory that relates the subjective experience of events to computational mechanisms of prediction error monitoring and memory updating.
Briefly, Event Segmentation Theory proposes that perceivers maintain a working memory representation of the current event and use it to guide predictions about what will happen in the near future. When prediction error spikes, they update their model. Data from individual differences, neuropsychology, and neuroimaging suggest that this mechanism is functionally significant for memory and that it can be impaired by neurological injury or disease.
New results indicate that it is possible to improve the encoding of event structure and that this may improve subsequent memory. Such results have implications for technology design and for the remediation of memory disorders in conditions including healthy aging, Alzheimer’s disease, and post-traumatic stress disorder.
Jeffrey M. Zacks is Professor and Associate Chair of Psychological & Brain Sciences, and Professor of Radiology, at Washington University in Saint Louis. He received his bachelor’s degree in Cognitive Science from Yale University and his PhD in Cognitive Psychology from Stanford University in 1999.
He is the recipient of scientific awards from the American Psychological Association and the American Psychological Foundation, and is a fellow of the American Association for the Advancement of Science, the Association for Psychological Science, the Midwest Psychological Association, and the Society of Experimental Psychologists.
Zacks is the author of two books, he has published more than 80 journal articles, and he has also written for Salon, Aeon, and The New York Times.
Events in Early Nervous System Evolution and the Origin of Event-Based Neural Computation
Joseph Cahill-Lane, Michael Paulin
Department of Zoology, University of Otago
Nervous systems evolved near the boundary between the Ediacaran period (635-543MA) and the Cambrian period (542-485MA). Nervous system design was essentially complete by the time that we, the vertebrates, appeared around 520MA. Human brains are ancient technology operating in a modern environment. In this paper we will argue that spiking neurons and the algorithms of neural computation for perception and decision-making can be reverse-engineered by studying the behavioral ecology of the Ediacaran-Cambrian transformation and asking why neurons evolved at that time. The short answer is that Ediacaran animals led uneventful lives in slowly-varying uniform environments, while Cambrian animals episodically encountered discrete threats and opportunities in fragmented environments. Nervous systems are stochastic event-based prediction machines that can anticipate events by causal inference from sense data, and affect consequences by generating actions. This paper will focus on the behavioral ecology and anatomy of Ediacaran animals, whose epithelium was co-opted to form sense organs and nervous systems in modern animals.
In a companion paper (Paulin, Pullar, Hoffman and Cahill-Lane, this meeting) we will present a computational neural model of causal inference and decision-making in a hypothetical ancestor, derived by considering the energetics of sensory acquisition and decision-making during the Ediacaran-Cambrian transition. Molecular-genetic evidence supports the view of 19th century anatomists that we are derived sponge larvae. Sponges (Porifera) still exist, as do a group of algal mat grazers lacking nervous systems, called Placozoa. Placozoans resemble Ediacaran animals such as Dickinsonia and may be a remnant of that group.
We’ll use the anatomy and behavior of poriferans and placozoans, along with fossil evidence and comparative developmental anatomy, to summarize the transformation of the ciliated epithelium of Ediacaran animals into the special sense organs and nervous systems of modern animals. Net energy uptake is proportional to time spent feeding and the free (available) energy density of the food source. The optimal strategy for Ediacaran mat grazers is to distribute themselves spatially in proportion to the local free energy productivity of the mat. This can be done by stochastic gradient search using cues detected at the body surface, without neurons or a nervous system. However, if some mat grazers evolve an ability to digest mat grazers, they become localized free energy sinks and other animals become localized patches of very high free energy density. The fossil record indicates that carnivores evolved on algal mats in the late Ediacaran, after c560MA. This leads to sudden switches in free energy density associated with episodic encounters between animals.
We propose that spiking sensory neurons initially evolved to signal contact with other animals, triggering either a rapid strike to capture a packet of food or rapid withdrawal to avoid becoming a packet of food. Despite lacking a nervous system, Trichoplax exhibits expansion-withdrawal behavior mediated by contractile cells and controlled by sensing local nutrient quality. Thus the first neuron may have been one small step for an animal, but one giant leap for animals.
Joseph Cahill-Lane is an M.Sc. student in Neuroscience interested in computational models of perception and cognition.
An STDP Rule Combined with BCM-like Fast Homeostasis Accounts for LTP and Concurrent LTD in the Dentate Gyrus
Cliff Abraham, P Jedlicka, L Benuskova
Department of Psychology, University of Otago
Long-term potentiation (LTP) and long-term depression (LTD) are widely accepted as mechanisms of learning and memory. It remains uncertain, however, which activity rules and mechanisms driving such synaptic plasticity are actually utilized by hippocampal neurons to generate LTP and LTD in behaving animals.
Recent experiments in the dentate gyrus of freely moving rats revealed that 400 Hz theta-burst stimulation (400-TBS) and 400 Hz delta-burst stimulation (400-DBS) elicited substantial LTP of the tetanized medial perforant path input and concurrent LTD of the non-tetanized lateral perforant path input. In contrast, 100 Hz theta-burst stimulation (100-TBS) was not able to produce significant LTP or concurrent LTD. Here we show in a computational dentate granule cell model (NEURON platform) that these data can be accounted for by a spike-timing-dependent plasticity (STDP) rule combined with a relatively fast Bienenstock-Cooper-Munro (BCM)-like homeostasis / metaplasticity rule, all on a background of ongoing spontaneous activity in the input fibres. Our results suggest that the interplay of STDP-BCM plasticity rules and ongoing pre- and postsynaptic background activity is sufficient to replicate qualitatively the experimentally observed patterns of input-specific LTP and concurrent LTD of granule cell synapses across the three different tetanisation protocols.
These findings should inspire experimental testing of whether granule cells, as well as other cell types in the event processing network, actually utilise these rules to induce such balanced synaptic plasticity and whether these mechansims are critical for event processing and storage.
Professor Cliff Abraham has research interests in the neural mechanisms of memory. He has played a leading role in promoting neuroscience research and teaching at the University of Otago.
Currently he is Co-Director of Brain Research NZ - Ranghau Roro Aotearoa, and is Director of a major HRC-funded research programme investigating biomarkers and therapeutic agents for Alzheimer’s Disease.
Cliff received a BA with Distinction in Psychology from the University of Virginia, and a PhD in Neuroscience from the University of Florida. He then undertook five years of postdoctoral research at the University of Otago and the University of Gothenburg, Sweden, before taking up a Lectureship in Psychology at Otago. In 1997 he was elected as a Fellow of the Royal Society of New Zealand, and in 2007 was awarded a James Cook Fellowship. He chaired the Department in 2003-2005, and in 2009 was awarded the University of Otago Distinguished Research Medal.
The Role of Errors in Predictive Coding in Schizophrenic Disorders
Andreas Fallgatter, Sabrina Schneider, Ann-Christine Ehlis
Department of Psychiatry at the University of Tübingen, Germany
Schizophrenic disorders are characterized by three dimensions of symptoms, which are positive symptoms (delusions, hallucinations, disordered thoughts/speech), negative symptoms (apathy, anhedonia, poverty of speech) and neurocognitive symptoms (“hypofrontality”, executive dysfunctions). Many of these symptoms, in particular positive and neurocognitive symptoms, may at least partially result from errors in predictive coding, i.e. false prediction errors (e.g. Fletcher and Frith, 2009; Martin, 2013). These false prediction errors may lead (i) to increased attention towards irrelevant stimuli, which can in turn result in the misunderstanding of ambiguous (e.g. social) stimuli (→ theory of mind deficits), (ii) a reduced filtering of self-generated stimuli (→ hallucinations) and (iii) an inappropriate perception of own behavioral outcomes (→ deficits in action-monitoring). We investigated these processes with NIRS measurements during processing of different “emotional styles” of human gait, resulting in differences between schizophrenia patients and healthy controls on the behavioral and on the brain functional level including connectivity patterns (Schneider et al., 2014, Schneider et al., submitted). Furthermore, we investigated pragmatic language comprehension based on the N400 event-related potential (ERP) component (Schneider et al., 2013) and action-monitoring with the error-related negativity (ERN/Ne) and positivity (Pe) components (Fallgatter et al., 2004; Herrmann et al., 2004, 2009, 2010; Ehlis et al., 2005, 2011), all indicating aberrant information processing in patients with schizophrenic disorders. This series of studies points to the importance and relevance of predictive coding theory, which promises to help in developing a better understanding of the underlying pathophysiology of these schizophrenia symptoms – and thus aspects of schizophrenia in general – as a prerequisite for the development of more individualized and effective therapies.
Andreas J. Fallgatter is head of the Dept. of Psychiatry at the University of Tübingen, Germany. Besides obligations in the treatment of patients and teaching at the Medical School, his current research topics are as follows: To elucidate the neurobiological basis as well as the treatment response including psychotherapy of psychiatric disorders including schizophrenia, depression, bipolar disorder, anxiety disorders, addiction disorders, dementia, ADHD and OCD; the methodological spectrum comprises brain activation measurements (EEG, ERP, NIRS, fMRI), non-invasive brain stimulation methods (rTMS, tDCS), neuropsychological assessments as well as molecular analyses.
Learning from Multiple Video Streams Without Annotations
Hendrik Lensch, Patrick Wieschollek
University of Tuebingen
In the past few years, we witnessed a tremendous increase of available online media content - especially video data. This is caused by the people's urge to share information on social media platforms and has led to an increased academic interest in automatically understanding video content. In comparison to traditional computer vision, which uses still images, a continuous stream of frames is able to encode temporal context and avoid ambiguities of events. For examples, the actions “stand up” and “sit down” contain very similar appearances between the event-boundaries (standing and sitting).
Recent success on computer vision tasks for single images is fueled by the existence of highly improved GPU (Graphics processing unit) hardware in combination with algorithms that optimally exploit their parallel processing power. Unfortunately, applying these techniques to entire image sequences still exceeds the currently available computational power many times over. However, when working on videos one essentially deals with data that contains much redundancy. This redundancy is apparent on two levels: intra-video (between frames) and inter-video (between videos). The first describes the fact, that any frame covers a significant amount of information about the subsequent frames. While any two videos can differ in the appearance -- they might share the same main features of an event. This inter-video redundancy is used in recent work to classify human action.
Unfortunately, most data-driven approaches require a profoundly annotated dataset, which is out-of-reach for large video collections. In this work we deal with detecting events from multiple videos simultaneously by synchronizing them. Hereby, we leverage the nature of the videos containing a continuous stream of information with causal ordering. Learning a feature embedding that expresses semantic similarity between frames allows us to compute similarity matrices between videos from which synchronization can be derived by shortest path calculation. This yields an approach which autonomously manages its training data for learning to understand video content. In the fashion of curriculum learning, the approach starts using basic assumptions and iteratively captures more diverse training examples by evaluating its own performance so far.
Applying this approach leads to interesting results and might give ideas about how do we perceive the world and connect similar events across different appearances. Which are the most prominent features we are probably looking for when perceiving human actions? How can we detect overlapping events?
Hendrik P. A. Lensch holds the chair for computer graphics at Tübingen University and is currently the head of the computer science department and the vice-spokesperson of the International Max Planck Research School for Intelligent Systems.
He received his diploma in computer science from the University of Erlangen in 1999. He worked as a research associate at the computer graphics group at the Max-Planck-Institut für Informatik in Saarbrücken, Germany, and received his PhD from Saarland University in 2003.
In his career, he received the Eurographics Young Researcher Award 2005, was awarded an Emmy-Noether-Fellowship by the German Research Foundation (DFG) in 2007, and received an NVIDIA Professor Partnership Award in 2010.
His research interests include 3D appearance acquisition, computational photography, machine learning, global illumination and image-based rendering, and massively parallel programming.
A Neural Network Model of Event Representations: Sensorimotor Sequencing, Place Coding, Self-organization, and Bayesian Inference
Martin Takac, Alistair Knott
Comenius University, Slovakia
In our talk we will present a model of how events are stored in working memory (WM).
Firstly, the model assumes a particular account of how episodes are experienced in the sensorimotor (SM) system. The key idea is that the process of experiencing an episode has a canonical sequential structure: the observer attends first to the agent, then to the patient (if there is one), and then classifies the action.
Assuming sequential attention to the agent and patient allows us to posit a WM medium that holds features of 'the currently attended individual', in particular its location and its identity (as a type or a token). This 'WM individual' medium is shown on the left of the figure below, along with a medium whose units hold short-term associations between location and type. The WM individual medium is occupied first by the agent, and then by the patient.
This sequential structure also allows an interesting method for encoding episodes. We assume a 'WM episode' medium, with separate fields for agent, patient, and action. Crucially, the agent and patient fields hold copies of representations in the WM individual medium - in particular, representations of type and token identity. Models where representations of the agent and patient are coded 'by place', in separate media, are normally prone to obvious problems: there is nothing in common between the representation of 'John as agent' and 'John as patient'. But in our case, where the place-coded representations are just pointers into a single WM individual medium, these problems do not arise.
The place-coded representations in the WM episode medium have several advantages. In particular, they allow the learning of localist representations of whole episodes, in units encoding associations between 'agent', 'patient', and 'action' representations. These are held in a self-organizing map (SOM; Kohonen, 1982), which learns to represent the most commonly occurring episodes. Since its inputs represent participants as both object tokens and object types, SOM units can come to represent a mixture of token episodes and episode types - another interesting advantage.
Episodes arrive sequentially in the WM system. The boundaries between events have recently been the subject of much interesting research (see Radvansky and Zacks, 2014 for a review). Our model features another SOM – this time a recurrent one - that is updated by each episode as it arrives. Units in this SOM come to represent the situations that are most frequently encountered, that is, the most commonly experienced sequences of episodes. A final interesting feature of our architecture is that we can train a function to predict the next episode in the candidate episodes’ buffer. Since this function learns to predict localist episode representations, after training it predicts a distribution over possible episodes – a very rich and useful structure.
Constructive Episodic Simulation Hypothesis: How the Brain Simulates Experience
Donna Rose Addis
The School of Psychology & Centre for Brain Research, The University of Auckland
Over the past decade, episodic memory has been reconceptualised to consider its role in imagining the future. In 2007, Schacter and I proposed the constructive episodic simulation hypothesis to account for emerging findings from cognitive neuroscience that remembering past events and imagining future events engage the same brain networks, and that loss of episodic memory is associated with a corresponding deficit in imagining the future. We argued that despite a proclivity for memory errors and distortions, the constructive nature of the episodic memory system provides the perfect neurocognitive architecture for future simulation, enabling flexible extraction and recombination of mnemonic details into simulations of hypothetical events.
In this talk, I present a novel refinement of the constructive episodic simulation hypothesis.
I argue for a change in emphasis, from the notion that imagination relies on episodic memory to considering these abilities as different instantiations of the same process – constructive episodic simulation.
Although the resultant representations may have some differences, such as temporal direction or phenomenological properties, these differences are superficial. At a fundamental level, memories differ little from imaginings; both require the flexible integration of informational elements stored in content-specific areas of the cortex to create complex event representations. I bring together ideas from various contemporary theories in cognitive neuroscience to yield a model of how both memories and imaginings are produced by the same constructive process and subserved by the same brain networks.
In particular, the default mode network (DMN) is the brain’s event simulator: it flexibly interacts with brain regions representing event content and schemas – and when required by situational demands, with networks mediating attention and cognitive control – to construct, encode, and reconstruct all varieties of mental simulations of experience.
Professor Donna Rose Addis is a cognitive neuroscientist and Associate Director of the Centre for Brain Research at The University of Auckland. Her research on memory and future thinking has been recognized with prestigious awards including NZ’s Prime Minister’s Emerging Scientist Prize, the Cognitive Neuroscience Society Young Investigator Award, and election as a fellow of the Royal Society of New Zealand.
BabyX: A virtual framework for embodied cognition
Laboratory for Animate Technologies,
Auckland Bioengineering Institute, The University of Auckland, New Zealand
BabyX is an autonomously animated virtual infant embodying a virtual nervous system comprised of interconnected anatomical models and physiologically based neural network models of sensory, motor, emotional and cognitive systems.
BabyX is designed to wholistically combine the wide range of factors contributing to behaviour in face to face social learning interactions.
The virtual nervous system for BabyX is constructed in a lego-like way with a modelling language BL which interconnects modules representing individual neurons or classes of larger neural networks, motor systems, sensors, and virtual anatomy. It supports modelling vision, speech and real time sensorimotor learning, and allows the construction of interactive models of cognition which take continuous input data from the real world and converts to successively discretized representations through event segmentation to be in a form accessible to and manipulatable by working memory.
Emotion coordinates brain-body states and is a key factor in modelling behaviour, learning and decision making. The virtual nervous system also supports virtual neurotransmitters and neurohormones, which modulate the behaviour and plasticity of neural circuits.
The BabyX project aims to provide a flexible “large functioning sketch” of how various ideas in cognitive science and neuroscience might work together to create realistic autonomous behaviour and learning. The models used in BabyX can be flexibly interchanged, and extensive use of computer graphics is made to be able to monitor the virtual nervous system model, from the neuron to system level.
Double Academy Award winner Dr. Mark Sagar is the CEO/co-founder of Soul Machines and director of the Laboratory for Animate Technologies at the Auckland Bioengineering Institute at the University of Auckland.
Mark and his team are pioneering new animation technologies combining biologically based models of neural systems and realistic faces and to create live interactive systems, capable of real-time learning and emotional response, to create the next generation of human interaction with cognitive systems.
Mark has a Ph.D. in Engineering from the University of Auckland, and was a post-doctoral fellow at M.I.T. He previously worked as the Special Projects Supervisor at Weta Digital and Sony Pictures Imageworks and developed technology for the digital characters in blockbusters such Avatar, King Kong, and Spiderman 2. His pioneering work in computer-generated faces was recognized with two consecutive Scientific and Engineering Oscars in 2010 and 2011.
A Model of Spike-based Inference and Decision-Making in the First Nervous Systems Predicts Functional Architecture in Brains of Modern Vertebrates
Mike Paulin, Kiri Pullar, Larry Hoffman, Joseph Cahill-Lane
Department of Zoology, University of Otago
In a companion paper (Cahill-Lane and Paulin, this meeting) we described the ecological context in which nervous systems evolved near the Ediacaran - Cambrian boundary. We introduced Porifera and Placozoa as extant models of ancestral animals lacking nervous systems, and outlined the historical process by which the undifferentiated ciliated epithelia of Ediacaran animals was transformed into nervous system designs of modern phyla more than half a billion years ago.
In this paper, we use Trichoplax as a model organism to examine what nervous systems evolved to do at that time. We assume that a rogue mutant capable of digesting Ediacaran mat grazers appeared in a population of Ediacaran mat grazers, as indicated in the fossil record. A model of why sensory neurons transmit information in stochastic spike trains, how spike trains represent information, and how populations of central neurons can infer causes of such sense data and make fitness-maximizing decisions accordingly, emerges from simple physics and probability theory in this scenario.
A grazer acquires net energy proportional to time spent grazing, while the expected loss rises sharply as a predator approaches. The expected net benefit of continuing to graze in the face of an approaching predator falls below zero at a critical distance. Travis Monk showed that evolutionary fitness is maximized by grazing until the last possible moment before withdrawing. The critical escape time, which when the critical state occurs, can be computed by a threshold trigger whose input is a distance-dependent signal generated by the predator. We propose that spiking sensory neurons evolved to do this on a millisecond timescale, as fast as efficient biophysical mechanisms permit. Given the 1-1 correspondence between causal states, spikes, and fitness-maximizing action as a function of state, no nervous system was required to intervene in generating the required action when spiking neurons first evolved.
We will show how stochasticity in sensory spike trains emerges from thermal noise in receptor mechanisms as selection pressure acts to increase the range of predator-detecting senses. In the physical limit, the threshold trigger generates random events that are random samples from a Poisson process parameterized by the distance to the predator. Serendipitously, the likelihood function for inferring the Poisson parameter is a negative exponential, which is the electrical impulse response of a neuronal cell membrane (indeed of any cell membrane). This makes neurons themselves natural computers for inferring the distance to a signal source given the output of noisy threshold triggers operating at the physical limits of sensitivity. We’ll show how this can be done, and how spiking neurons can represent the probability distribution of states in a way that makes state-dependent optimal decision-making straightforward. The necessary circuit architecture resembles the neuroanatomy of vertebrate brain structures that are involved in tracking target motion, and the predicted statistical properties of sensory spike trains are found in electrophysiological data. This suggests that our model of a simple, ancient nervous system may be relevant to understanding algorithms of perception and decision-making in modern nervous systems.
Mike Paulin is an Associate Professor in Zoology, a computational evolutionary neuroscientist interested in the role of cerebellum in tracking and controlling movements.
Communication between the ACC and VTA during decision-making
Thom Elston, David Bilkey
Department of Psychology, University of Otago, New Zealand
The anterior cingulate cortex (ACC) has previously been implicated in several high-level processes, including value-based decision-making and exploration.
It is, however, unclear how ACC activity might be transformed into motivational signals. One possibility is through the region’s connections to the ventral tegmental area (VTA), a dopaminergic region implicated in motivation.
We tested this hypothesis by monitoring ACC single-units and local-field-potentials (LFPs) as well as LFPs in the VTA of rats performing a cost-benefit reversal task which elicited both economic, value-based decisions and decisions which appeared to be exploratory.
A combination of behavioural, electrophysiological, and modelling analyses revealed that transient increases in 4Hz ACC modulation of the VTA corresponded with the pre-reversal selection of a less-favoured low-cost, low reward option. The temporal pattern of this behaviour suggested that it may reflect exploration and/or decision implementation.
This interpretation was consistent with our observation that the ACC-to-VTA signal was initially high but decreased exponentially during habituation to a novel open field. Increased ACC-to-VTA signalling was also observed after the reversal as animals as the rats sustained their responses to the new optimal side.
In sum, our results suggest that ACC-to-VTA signalling is important for the initiation and persistence of non-default behaviour.
Thomas has recently completed his PhD in David Bilkey's lab where he studied the electrophysiological relationship between the anterior cingulate cortex and the ventral tegmental area during adaptive behaviour.
He is currently interested in postdoc opportunities.
Altered Temporal Coding of Event Memory Consolidation in a Model of Schizophrenia Risk
David Bilkey, Tara Hayward, Elena Cavani, Kirsten Cheyne
Department of Psychology, University of Otago
The hippocampus encodes information relevant to both event memory and spatial navigation. Both alterations in hippocampal function, and memory deficits, are observed in schizophrenia. It is, however, unclear how these hippocampal changes might contribute to the memory problems. To address this question we have modelled aspects of schizophrenia using prenatal maternal immune activation (MIA), which is a known risk factor for the disease, in an animal model. MIA was induced in pregnant rat dams with a single injection of the synthetic cytokine inducer polyinosinic:polycytidylic acid (poly I:C) on gestational day 15. Control dams were given a saline equivalent. Firing activity and local field potentials (LFPs) were recorded from the CA1 region of the hippocampus in adult male offspring of these dams as they moved freely around a rectangular track.
A large proportion of neurons recorded from both groups displayed characteristic spatially-modulated ‘place cell’ firing activity and the burst firing activity that occurs during sharp-wave ripple (SWR) LFP events. SWR bursts have been linked to memory consolidation and anticipatory ‘preplay’ processes. Initial data from our studies indicates that while classic place cell ‘rate code’ firing is relatively unaffected by the MIA intervention, ‘temporal code’ firing is altered. In particular, an analysis of the timing of SWR bursts revealed that they had less coherent structure than was observed in control animals. This effect has the potential to impact on both decision-making and memory consolidation processes. Furthermore, when viewed from a Complementary Learning System approach, a reduction in coherent episodic-semantic memory consolidation might underlie some of the semantic memory deficits observed in schizophrenia.
David received a PhD from the University of Otago in 1987. After working as a postdoctoral fellow at the University of Washington in 1988 he returned to Otago as a lecturer, and served as the Head of Department from 2009 to 2013.
Posterior Middle Temporal Gyrus –a Cortical Hub of Knowledge about Tools and their Usage?
Marc Himmelbach, Mareike Gann
Center for Neurology, Hertie Institute for Clinical Brain Research, University of Tuebingen
Our modern civilization provides a huge number of tools, which have been optimized for a specific purpose and goal. Some of these tools and their ways of usage are well known to each adult even if they have little practical experience with a respective tool. The integration of such highly familiar tools in sensorimotor and cognitive processing and planning can thus not only draw on currently available sensory information but also on long-term procedural and semantic memories associated with these tools. A so-called cortical tool network, consisting of the dorsal supramarginal gyrus (SMG), the inferior frontal cortex (IFC), and the inferior lateral occipito-temporal cortex (LOCT) has already been associated with the visual presentation, naming, and usage of tools. However, it is unknown through which structures or connections this tool network interacts with memory retrieval, thereby integrating semantic knowledge about tools.
We conducted three fMRI experiments comparing the processing of familiar with unfamiliar tools. In experiments 2 and 3, healthy participants decided whether a presented tool could be used or not used for a given task (e.g. crack a nut – nutcracker). For this task, we hypothesized and found increased signals in the SMG, IFC, LOTC for unfamiliar tools relative to familiar tools, most likely associated with visual and sensorimotor analysis of a currently presented tool. Additionally, we found signal changes at the anterior fronto-median cortex, a region which is associated with decision-making and error monitoring. A signal increase for familiar tools in the posterior middle temporal gyrus suggested a specific role for this region in the retrieval of previously established knowledge about such tools.
We confirmed these findings in a second experiment that looked more into the temporal sequence of activations across these networks using the same task but modified timing of events. In a third experiment, our participants were not explicitly instructed to evaluate the function of a given tool, but simply to categorize a variety of tools (e.g. kitchen vs. workshop). In agreement with our interpretation of experiments 1 and 2, we found signal increases at the posterior MTG in this categorization task relative to a sensory discrimination task. We assume that the posterior MTG connects long-term knowledge with ongoing motor cognition processing, representing a crucial hub in the integration of procedural and semantic memory, visual analysis, and motor cognition.
Marc Himmelbach received his Diploma in Psychology from the University of Düsseldorf in 2000. He obtained a doctoral degree in Neural and Behavioral Sciences at the University of Tübingen in 2005. He received a ERC Starting Grant in 2007 and established a research group on Neuropsychology of Action Control at the Hertie Institute for Clinical Brain Research in 2008.
The Dynamics of Human Action: Perception of Kinematics and Body Shape
Queens University in Kingston, Ontario
The overall size of the body, the length of its limbs, and the distribution of weight over the body all affect the kinematics and the underlying dynamics of human action. When looking at other people in order to assess who they are and what they do, the visual system applies internal models that describe the relations between body shape and body motion. What kind of knowledge do such models contain?
I will present a number of behavioral studies that investigate to which extent the visual system is able to exploit relations between kinematics and body shape. Our stimuli are based on motion capture data from individuals which lift boxes of different weight, push heavy sledges along the floor, or throw small items at different distances. From these motion capture data, we reconstruct the kinematics of the body, but also the individual body shape of the actors. Hybridizing shape and motion from actors with very different body weights allows us to create stimuli with varying degrees of inconsistencies between body shape and body motion.
Participants are displayed with these stimuli and are asked to discriminate the veridical from the hybrid stimuli, to attribute properties such as attractiveness and realism to the renderings, and to estimate physical properties of the manipulated objects. Our results show, that observers are not able to discriminate veridical from hybrid stimuli and that there are no systematic differences in the way they attribute properties to the actors. However, we observe systematic differences in perceived object properties.
The results demonstrate that changes in the relation between body shape and body kinematics are perceived, but that they are interpreted as changes in the physical properties of the manipulated objects rather than being perceived as explicit inconsistencies in the renderings.
Nikolaus Troje is a vision researcher with appointments in Psychology, Biology, and Computer Science at Queen's University in Kingston, Ontario. He received his PhD from Freiburg University, conducted post-doctoral work at the MPI for Biological Cybernetics, and taught at the University of Bochum, Germany, before becoming a Canada Research Chair at Queen's University.
Predictive Event Segmentations: a Key Towards the Formation of Behaviorally-Useful Hierarchical Cognition?
Martin V. Butz
Cognitive Modeling, Department of Computer Science, University of Tübingen
Hierarchical structures are ubiquitous in our minds. Sensory processing abstracts away from sensory details towards entity as well as body-relative and environment-relative spatial encodings. Motor processing streams yield directional motion, goal-oriented motion, as well as dynamic motion primitive encodings. Finally, multisensory and sensorimotor dynamics are analyzed and encoded in the form of events and event transitions.
I propose an integrative modeling perspective by means of the anticipatory behavioral control principle. When acknowledging that in the end all encodings should serve suitable behavioral control purposes – including abilities of adaptation, directing attention, social interaction, versatile planning, and reasoning – it soon becomes apparent that behavior needs to be co-controlled by considering desired future states. However, to avoid excessive computational burden, (i) compact encodings of events and event transitions as well as (ii) selective activation mechanisms are essential.
Frequently encountered and typically motivationally relevant sensorimotor sequences may be encoded as compact events, with enclosing even boundary conditions. Recursively then, frequently encountered event sequences themselves may form compact event encodings, thus building event hierarchies. With such event and event boundary encodings in hand, particular event transitions can be imagined and strived-for. Similarly, event sequences can be replayed – manifesting themselves as abstracted and partially re-constructed episodic memory recollections – but also novel, plausible event sequences can be invoked – manifesting themselves as imaginations. With sensible hierarchical even structures in place, more abstract, complex, and far reaching episodic recollections and imaginations become possible.
Even with such compact, hierarchical encodings, excessive considerations of possible future developments are still highly computationally demanding. Thus, the activation of recollections and imaginations need to be prioritized in a reward-oriented manner, that is, in a manner that depends on the own motivational system. Computationally, this implies that prioritization depends on (expected) reinforcement. Predictive event encodings essentially allow the implementation of reinforcement learning based on selective episodic replays and imaginations. In a given situation, a limited number of future scenarios can be selectively considered, thus generating more adaptive action choices. Similarly, offline memory recollections, counterfactual considerations, as well as imaginations of event progressions can furthermore lead to the optimization and compaction of event encodings and associated behavioral decisions.
Along these lines, I will give an overview of the critical role of predictive event segmentations, behavioral evidence for the existence and relevance of such encodings, and a short glimpse at according neuro-computational models in artificial systems.
With degrees in computer science (Würzburg University & University of Illinois at Urbana-Champaign), Prof. Butz was appointed professor in cognitive modeling at Tübingen University in 2011. He has published more than 50 journal articles covering topics of artificial neural networks and machine learning, experimental psychology, and neuro-computational cognitive models. His recent book on “How the Mind Comes into Being” (Oxford University Press, 2017) offers a computational introduction to cognitive science from a highly integrative, interdisciplinary perspective.
Linguistic Representations of Events, and their Relevance to Cognitive Models
Dept. of Computer Science, University of Otago
Linguists - in particular syntacticians and semanticists - spend much of their time developing models of event representations. Sentences are a vehicle for conveying event representations. (Sentences can also convey ‘states’, which are not the main focus of the current workshop - but conveying events is certainly one of their main functions.) Most linguists developing accounts of sentences see themselves as modelling aspects of cognition, rather than ‘surface sentences': they consider their primary object of study to be the cognitive mechanisms responsible for creating and interpreting sentences, and representing their meaning, rather than sentences themselves. Given this fact, it is surprising that most linguistic models do not make more contact with the tools and methods of modern neuroscience.
In this talk I will outline some of the main findings about sentence syntax and semantics that have emerged in linguistics - and try to express these in a way that connects with models of event representations in psychology and neuroscience. I will try to be agnostic between the various alternative syntactic theories, but in places I’ll adopt one particular theoretical perspective, namely Chomsky’s ‘Government-and-Binding’ theory. While the syntactic representations posited within this theory are motivated through linguistic argumentation, and make no reference to brain mechanisms, I will argue they are amenable to an interesting interpretation in neuroscientific terms, that sheds useful light on cognitive models of event representations.