Gazzaniga and Mangun 2014. The Cognitive Neurosciences
Gazzaniga and Mangun 2014. The Cognitive Neurosciences
Gazzaniga and Mangun 2014. The Cognitive Neurosciences
NEUROSCIENCES
THE COGNITIVE
NEUROSCIENCES
Fifth Edition
A BRADFORD BOOK
THE MIT PRESS
CAMBRIDGE, MASSACHUSETTS
LONDON, ENGLAND
© 2014 Massachusetts Institute of Technology
10 9 8 7 6 5 4 3 2 1
CONTENTS
Preface xiii
contents v
II PLASTICITY AND LEARNING
vi contents
23 Computational Models of Attention Laurent Itti and
Ali Borji 245
25 Are Neural Circuits the Best They Can Be? Fred Rieke 271
contents vii
37 Basal Ganglia Function Robert S. Turner and
Benjamin Pasquereau 435
VI MEMORY
viii contents
VII LANGUAGE AND ABSTRACT THOUGHT
60 The Cognitive Neuroscience of Fear and Its Control: From Animal Models
to Human Experience Catherine A. Hartley, Justin M. Moscarello,
Gregory J. Quirk, and Elizabeth A. Phelps 697
contents ix
65 Sensory Foundations of Socioemotional Perception
Timothy R. Koscik, Nicole White, Hanah A. Chapman, and
Adam K. Anderson 751
IX CONSCIOUSNESS
X ADVANCES IN METHODOLOGY
x contents
80 Using Computational Models to Understand Robustness and Variability in
Nervous Systems Tilman J. Kispersky, Timothy O’Leary, and Eve
Marder 949
Contributors 1079
Index 1085
contents xi
PREFACE
With some amusement we note that the preface to the last edition of this book
carried a hilarious error. One of us (MSG) wrote that preface and buoyantly noted
that the first meeting at Lake Tahoe, which underlay that book, had occurred 20
years earlier. After all, it was the fourth edition and the book does come out every
five years. Do the math, right? Upon first glance, it seems correct and yet like most
everything else in our field, what seems right is not good enough. We have to think
about the problems we study again and again.
Of course, it seemed right to MSG because the Summer Institute in Cognitive
Neuroscience had already been up and running for the four years preceding the
1993 Tahoe meeting. It had started off at Harvard, moved to Dartmouth for two
years and then to UC Davis. The fifth year of that cycle was the Tahoe meeting,
which resulted in the first edition of this volume. Obviously, the volume summa-
rized the previous four years of research presented at the Summer Institutes, not
the future SI’s.
So, in this 2014 edition, the result of the fifth Tahoe meeting, we can say that it
really was 20 years ago that we met for the first time in Tahoe! Not only that, the
fifth meeting was one of the best. Many of the speakers were former fellows, and
many of the current fellows will undoubtedly be future speakers and authors.
Indeed, this book and the 2013 Tahoe meeting reflect a transition. As JFK famously
said, “The torch has been passed to a new generation. …” In this case, young cogni-
tive neuroscientists are exploring the logic of old ideas and, in some cases, finding
them wanting and presenting new ideas. While some old hands were around to
steady the transfer, a new era is launched with this book and is bursting with new
talent.
The Institute started with a bang, a perspective on some pitfalls in the brain sci-
ences. It was pointed out that while human brains are extremely large, neuroscien-
tists have lacked the necessary methods both to compare humans to other animals
and to determine how evolution modified the internal structure of the brain. It
was noted that in the absence of such methods, neuroscientists have focused
on the study of relatively few model species and have viewed those results as appli-
cable to humans with little qualification. Fortunately, new techniques, especially
MRI techniques and the methods of comparative genomics, have started to be
used to study humans and other animals and have been game changers.
Particularly exciting is the application of diffusion tensor imaging (DTI), a tech-
nique that has been used to compare humans to chimpanzees (the animals most
closely related to humans, which makes them critical species for identifying human
specializations). These studies have revealed, for the first time, differences in
systems of connections between cortical areas and have provided the foundations
for understanding the neural bases of human cognitive and behavioral
specializations.
preface xiii
In the area of development, research in the past 20 years or so has generated
profound insights into the infant mind and has revealed that the cognitive and
social cognitive capacities of very young children are much more sophisticated than
was once believed. While developmental psychology made great progress in the
later part of the last century, it remained relatively removed from developmental
neuroscience. In the past decade, the field of developmental cognitive neurosci-
ence has undergone unprecedented expansion. Once again, the use of MRI and
fMRI has propelled our knowledge of how the human brain develops, and the data
from developmental imaging studies have in turn spurned new interest in the
changing structure and function of the brain over the lifespan.
The study of sensation and perception is also being revolutionized by new experi-
mental techniques that provide deeper insights into old problems. New tools are
improving our ability to test the causal role of specific neurons and brain regions
in shaping sensory perception. They are allowing us to record from increasingly
large ensembles of neurons, revealing the ways that information about the state of
the world is untangled from population activity evoked at the sensory epithelium.
New data are also revealing the competing constraints that sculpt the architecture
of sensory systems. Finally, new conceptual approaches and computational methods
are beginning to reveal how the brain’s limited hardware can give rise to fast, flex-
ible, and robust perceptual categories. In short, new tools are exploding our field
of view, pushing us to develop better theoretical frameworks.
People are astounding in their ability to create goal-directed behaviors that trans-
form the world. This vast behavioral repertoire creates theoretical challenges for
defining the fundamental principles of selection, planning, and control that the
motor system must employ to achieve these ends. The section on motor systems
and action addresses some of these challenges by first emphasizing the basics of
motor system anatomy, particularly at the circuit level. This architecture includes
cortical connectivity between many cortical motor areas and subcortical systems,
particularly the basal ganglia and cerebellum. Functional contributions of the dif-
ferent motor areas are considered in terms of optimal control and the retrieval of
learned motor behavior. The section then introduces some of the most important
correlation methods for identifying dynamic representations of movement based
on multiple neuron recordings. The section closes with two chapters on problems
of human action selection. The first considers how we choose among a variety of
goal-directed behavior when there are competing risks and rewards. The second
evaluates the way that possible actions are planned automatically, in advance of any
decision.
Approximately 20 years ago, the memory section of the first edition of The Cogni-
tive Neurosciences highlighted two major themes central to theory and research at
the time. First, to understand memory, one must characterize the processes govern-
ing memory encoding, storage, and retrieval. Second, memory is a nonunitary
entity, as it consists of multiple forms of knowledge expression that depend on
distinct neural substrates and computations. In the intervening years, technological
advances in human and nonhuman neuroscience have enabled increased anatomi-
cal and computational precision in these core areas, while simultaneously sparking
novel themes and challenges to earlier ideas. The memory section of this volume
documents the field’s progress toward establishing increasingly precise models of
memory, such as specifying the distinct computations and forms of remembering
supported by regions in the medial temporal lobe, delineating the multiple mecha-
nisms that enable memory performance over short time scales, and illuminating
how context is neurally represented and shapes memory behavior. The chapters in
this section also document some of the newly emerging themes that have shaped
the field in recent years, such as the convergence of theorizing about memory and
xiv preface
decision making, the unexpected role of parietal cortex in memory, and the fun-
damental link between memory and prediction. The remarkable strides summa-
rized in these chapters evidence past progress and predict a future rich with
continued discovery.
One flourishing area in cognitive neuroscience revolves around trying to under-
stand the affective basis of the social brain. In solving adaptive problems, the brain
relies on emotional processes to solve challenges to successful adaptation. For
humans, many of the most challenging adaptive problems result from the necessity
of interacting with other humans, such as selecting mates, cooperating in hunting
and gathering, forming alliances, competing over scarce resources, and even
warring with neighboring groups. Interacting with other humans produces emotion,
and these emotions serve as guidelines for successful group living. Any true under-
standing of human nature requires a full consideration of both the emotional and
social aspects of brain functioning. There is remarkable convergence in this section
that highlights the important role of the amygdala across animal species and para-
digms. In addition, it is now clear from more than a decade of research that various
prefrontal regions are crucial for successful functioning within social contexts. This
section demonstrates that the study of emotion and social cognition has become
established as a core area of cognitive neuroscience.
The session on language and abstract thought was a testimony of where the field
has moved since the previous edition of this book. First, the connectivity profile of
the perisylvian language areas has been mapped out in much more detail, as have
the cytoarchitectonic and receptor architectonic properties of language areas such
as Broca’s area. It has become clear that the language network, beyond any doubt,
is much more extensive than Broca’s area, Wernicke’s area, and the arcuate fascicu-
lus (the classical Wernicke-Lichtheim-Geschwind model). Second, studies record-
ing event-related brain potentials have established firmly that prediction plays an
important role during language comprehension. Predictions are made at multiple
levels about general event structure, a word’s grammatical characteristics, and even
about the upcoming word sound. Finally, we now realize that communication with
language is a multimodal system in which speech is accompanied by gestures and
other information. Indeed, sign language is a form of language based solely on the
visual-gestural modality. How this difference in modality of expressing language
and how the multimodal aspects co-determine the neural architecture of the system
are starting to be understood. Moreover, understanding language is more than
decoding what is written or said. In communication we try to infer the intention
behind the utterance. The step from coded meaning to speaker meaning, we now
know, involves the ToM network. Hence, it is clearly shown that an account of lan-
guage comprehension based solely on the mirror neuron system will fail
hopelessly.
The session on consciousness showed, in no uncertain terms, how far we have
moved since the days when the c-word was an outcast from psychology and neuro-
science (but one that had gained early admission within cognitive neurosciences,
as witnessed by the previous editions of this volume). There are now bold attempts
at characterizing consciousness theoretically, with respect to both quantity and
quality, which do not shy away from the “hard problem” of what consciousness is
and how it is generated by the brain. Impressive new work using TMS/EEG has
demonstrated that the level of consciousness can be measured in individual subjects
and across brain states as diverse as wake, sleep, anesthesia, and brain disorders.
We are also learning a great deal about how and when consciousness is lost in dif-
ferent kinds of seizures—an area of study that was nearly nonexistent even 10 years
ago. A strong case is now being made for the experimental and conceptual dissocia-
tion between consciousness and attention on one side and between consciousness
preface xv
and executive function/report on the other. Intriguingly, initial results suggest that
some of the so-called neural correlates of consciousness demonstrated in past
studies have to do with stimulus processing or response preparation rather than
with consciousness itself. Other prominent areas of progress include the relation-
ship of consciousness with cognitive and neural access, the conscious control of
action, and the relevance of body awareness to consciousness in general.
Tackling such profoundly difficult questions will be aided by the introduction of
powerful new methodologies to analyze brain function. What were leading tech-
nologies 10 years ago are not the leading technologies of today. Accordingly in this
edition of The Cognitive Neurosciences we include a new section devoted to new
methods that will shape and guide our field into the future. Moving from new, more
exacting methods of measurments for fMRI to deeper probings of the massive data
collected, today’s cognitive neuroscientist must be well versed in both several new
methodologies and concurrent research in separate related fields. Sophisticated
models of causality in brain function are now beginning along with the application
of network theory to massive data sets on diffusion imaging knowledge about brain
connections.
Finally, other people are mysterious. We do not understand how they work, and
we do not know how to treat them. Neuroscience is starting to help us unravel these
mysteries. To improve our understanding of society, recent work in cognitive neu-
roscience has revealed some of the psychological and neural processes through
which we perceive people as people, through which we determine which mental
states other people have, and through which we form moral judgments about
actions and choices by other people. In addition, knowledge of the brain can be
used as a tool to change how we relate to other members of society. It can affect
policies regarding addicts, psychopaths, and perhaps even “normal” criminals. It
can also be used to develop drugs to enhance cognitive and emotional performance
in people without illnesses and to understand and control economic transactions
through neuromarketing. This power of neuroscience to illuminate and affect
society can, of course, be used for good or for ill, so it poses profound moral quan-
daries that we are only beginning to face. The goal of the section on neuroscience
and society in this volume is to raise these issues, not to resolve them.
It was a glorious three weeks of science interrupted by hikes, swimming, and
pondering the lectures in the sylvan beauty of Lake Tahoe. This book and the linked
videos of the lectures are our attempt to let one and all enjoy the richness and
scholarship of the great scientists involved in this effort. The complex and intense
planning, editing, logistics, and time put in by dozens of people should be obvious.
Needless to say, the section editors and the participants make the book what it is
and all deserved our profound thanks. The flawless execution of the meeting itself
was due to Jayne Rosenblatt, and once again the managing editor of the book,
Marin Gazzaniga, made it all seem easy. Finally, we are indebted to MIT Press, which
year after year serves our field in ever so many ways. And all of this, of course, would
not have been possible without the generous support of the National Institute of
Mental Health.
xvi preface
I
DEVELOPMENTAL
AND EVOLUTIONARY
COGNITIVE
NEUROSCIENCE
Chapter 1 BAILLARGEON, SETOH, SLOANE, JIN,
AND BIAN 7
3 CRONE 23
6 WISE 51
Fifty years ago, very little was known about how the
human brain develops. Research on brain development
in the second half of the twentieth century focused
almost entirely on nonhuman animals and revealed a
great deal about early neuronal and synaptic develop-
ment (Hubel & Wiesel, 1977). These advances in animal
research were paralleled by pioneering research in
developmental psychology, particularly by Piaget (1977)
and Vygotsky (1986). Their studies, which involved
observing and analyzing children’s behavior in meticu-
lous detail, changed contemporary thinking about chil-
dren’s minds. Through such important behavioral
studies, children were increasingly seen as more than
just miniature versions of adults. Research in the past
20 years or so has generated profound insights into the
infant mind and has revealed that the cognitive capaci-
ties of very young children are much more sophisti-
cated than was once believed. Infant social cognition is
the focus of chapter 1 by Baillargeon, Setoh, Sloane,
Jin, and Bian in this section.
While developmental psychology made great prog-
ress in the later part of the last century, it remained rela-
tively removed from developmental neuroscience.
Research on human neural development was heavily
constrained by the technical challenges of studying the
living human brain and, until fairly recently, was limited
to the study of post-mortem brains. Yet, in the past two
decades, the field of developmental cognitive neurosci-
ence has undergone unprecedented expansion. This
can at least in part be attributed to technological
advances. Electroencephalography, or EEG, has been
used for several decades to study the temporal dynamics
of brain function in infants and children. However, in
ABSTRACT Infant social cognition depends on at least two Mary’s and Jane’s individual actions by inferring the
evolved systems: the psychological- and sociomoral-reasoning mental states underlying these actions (e.g., Mary wants
systems. Each system has at its core a distinct explanatory
to stack all the rings, she noticed that the last one fell,
framework of principles and concepts. The psychological-
reasoning system enables infants to interpret agents’ inten- and she is trying without success to reach it; Jane under-
tional actions and is constrained by a principle of rationality stands what Mary is trying to do, and she purposefully
(with its corollaries of consistency and efficiency). When walks to the ring and picks it up). The sociomoral-
infants observe an agent act in a scene, the psychological- reasoning system would allow infants to form expecta-
reasoning system infers the mental states that underlie the tions about what Jane should do next (e.g., is it
agent’s actions; if the scene changes, infants use these mental
states—together with the rationality principle—to predict the obligatory for Jane to give the ring to Mary? is it permis-
agent’s likely actions. Recent evidence indicates that infants sible for Jane to drop the ring back to the floor? is it
are already capable of sophisticated mentalistic reasoning and impermissible for Jane to steal the ring?) and how Mary
can attribute to agents not only motivational states (e.g., should respond to Jane’s actions.
goals) and epistemic states (e.g., ignorance), but also coun- In this chapter, we review findings on psychological
terfactual states (e.g., false beliefs). The sociomoral-reasoning
system guides infants’ expectations about how individuals
and sociomoral reasoning in infancy (i.e., before age
should act toward others and is constrained by several prin- 2). These findings are important not only for develop-
ciples, including reciprocity, fairness, and ingroup (with its mental psychology but also for cognitive science gener-
corollaries of loyalty and support). When infants observe two ally, because they inform and constrain theoretical
or more individuals interact in a scene, the sociomoral- models about the nature and causal etiology of human
reasoning system determines what actions are obligatory, what
intuitive social cognition.
actions are permissible, and what actions are impermissible.
This chapter reviews key findings concerning each system in
infancy. Psychological reasoning
Beginning in the first year of life, infants attempt to Agents When infants encounter a novel entity, they
make sense of the actions of others. At least two causal- gather evidence about its ontological status. If the entity
reasoning systems contribute to this process: the is capable of autonomous motion (e.g., begins to move
psychological- and sociomoral-reasoning systems (for a com- or reverses course on its own), infants categorize it as
prehensive review, see Baillargeon et al., 2014). Each self-propelled and endow it with internal energy (e.g.,
system operates largely without explicit awareness and Luo, Kaufman, & Baillargeon, 2009). If the entity has
has at its core a distinct explanatory framework com- autonomous control over its actions (e.g., uses varied
prising key principles and concepts. The psychological- means to achieve the same goal or responds purpose-
reasoning system infers agents’ mental states (e.g., fully to changes in its environment), infants categorize
Leslie, 1994), and the sociomoral-reasoning system it as agentive and endow it with mental states (e.g.,
determines what is obligatory, permissible, and imper- Csibra, 2008; Johnson, Shimizu, & Ok, 2007). Finally, if
missible in social interactions (e.g., Dwyer, 2009). the entity gives evidence that it is both self-propelled
To illustrate the purview of each system, imagine that and agentive, infants categorize it as an animal and
infants are observing the following scene. Mary is stack- endow it with biological properties, such as filled insides
ing rings on a table; the last ring rolls to the floor, and (Setoh, Wu, Baillargeon, & Gelman, 2013).
Mary vainly attempts to reach it. At that point, Jane By 3 to 5 months of age, infants can already identify
enters the scene; she observes Mary’s efforts, walks over and reason about novel nonhuman agents (e.g., Hamlin
to the fallen ring, and picks it up. The psychological- & Wynn, 2011; Luo, 2011; Luo & Baillargeon, 2005).
reasoning system would enable infants to interpret These findings make it unlikely that infants gradually
ABSTRACT The past 15 years have seen a major expansion in With the increased availability of MRI, a number of
research on the structural development of the maturing large-scale studies of typical structural brain develop-
human brain in childhood and adolescence. In this chapter,
ment have demonstrated not only that the human brain
we summarize the major findings from histological and mag-
netic resonance imaging (MRI) studies looking at the struc- continues to undergo substantial structural remodeling
tural development of the child and adolescent brain. There throughout childhood and adolescence and into adult-
is substantial evidence for the ongoing development of many hood, but have also documented region-specific differ-
regions of the brain into adulthood, both microscopically and ences in the timing and extent of these changes. It is
macroscopically. Histological investigations have shown that important to note that common MRI techniques are
myelination continues throughout childhood and adoles-
cence, and appears to correspond to the increasing white limited to investigations of macroscopic changes, since
matter volumes seen in MRI studies. Post-mortem studies the images produced are typically made up of voxels of
investigating synaptic density have documented a dynamic between 1 mm3 and 3 mm3. By comparing MRI investi-
process of synaptogenesis and synaptic pruning that pro- gations with post-mortem histological studies, we are
gresses in a region-specific manner during this developmental better able to describe the major macroscopic develop-
period. MRI studies have found that most cortical regions
show evidence of increasing gray matter volume during child-
mental patterns of the human brain and link these to
hood, before peaking in late childhood/early adolescence the likely microscopic changes that underlie them.
and decreasing in volume during adolescence, although there In this chapter, we summarize the major findings
is wide variation in the timing and extent of this change. In from histological and MRI studies looking at the struc-
contrast, subcortical regions show varying trajectories, which tural development of the child and adolescent brain.
differ between structures but also across studies. Possible
future directions to tackle many of the unanswered questions
that remain are discussed at the end of the chapter. Histological changes in brain structure during
childhood and adolescence
The past 15 years have seen a major expansion in
research on the structural development of the matur- Since the late 1960s, studies have shown evidence of
ing human brain in childhood and adolescence, cellular age-related differences in particular regions of
based largely on the results of cross-sectional and lon- the human brain in childhood and adolescence (Hut-
gitudinal magnetic resonance imaging (MRI) studies tenlocher, 1990, 1979; Huttenlocher & Dabholkar,
(Brain Development Cooperative Group [BDCG], 1997; Miller et al., 2012; Petanjek et al., 2011; Yakovlev
2012; Brown et al., 2012; Giedd et al., 1996; Lenroot & Lecours, 1967). These developments can largely be
et al., 2007; Østby et al., 2009; Raznahan et al., divided into changes in myelination of axons and
2011a; Sowell, Trauner, Gamst, & Jernigan, 2002; axonal caliber, and changes in the organization and
Sullivan et al., 2011). Prior to the development of density of synapses.
MRI, our understanding of the maturation of the
human brain was largely limited to data from post- Changes in Myelination Myelination, the process
mortem studies. While such histological studies provide of laying down myelin by oligodendrocytes, is important
invaluable information regarding the basic processes for the maturing connectivity within the developing
underlying brain development, they are unable to brain, allowing the synchronization of information
inform us fully about how individuals change over transfer within and between brain networks (Fields &
time, how structures vary in their development both Stevens-Graham, 2002). Myelination occurs as a result
within and between individuals, and how these struc- of reciprocal communication between neurons and oli-
tural changes relate to functional correlates and behav- godendrocytes (Simons & Trajkovic, 2006). Humans
ioral patterns. are born with relatively low levels of neocortical myelin
ABSTRACT One of the major steps in cognitive development Social Affective Sensitivities At the onset of ado-
during childhood and adolescence is an increase in cognitive lescence the influence of social and affective context
control functions, such as working memory, response inhibi-
starts to impact the decisions that adolescents make
tion, adaptation, and complex reasoning. Yet, at the same
time adolescence is characterized by a relative dominant focus (see chapter 4, this volume). Adolescence starts with
on rewards and short-term outcomes at the cost of long-term puberty around 10 to 11 years of age, although there is
consequences. This chapter reviews neurocognitive studies variance between children at the onset of puberty
showing a change in recruitment of prefrontal cortex and (Forbes & Dahl, 2010). During pubertal development
parietal cortex over time when performing cognitive control there are substantial changes in hormone release; these
tasks and heightened sensitivity of subcortical areas in
response to emotional stimuli in mid-adolescence. Together, changes propagate alterations in both bodily character-
these findings have been interpreted in terms of different istics and in social-affective sensitivities, such as an
developmental trajectories for brain regions involved in increased tendency towards risk-taking and a greater
making complex decisions, which creates sensitivities but also sensitivity to peer group influence (Crone & Dahl,
important opportunities for adaptation and explorative 2012). Most changes in social sensitivity and increases
learning.
in risk-taking behavior are adaptive, to stimulate explor-
ative learning, and to eventually obtain mature social
goals. However, in some cases such changes can have
Cognitive and affective decision making serious consequences, including accidents, drug abuse,
in adolescence and in extreme cases suicide attempts (Dahl & Gunnar,
2009).
Fast Improvement in Cognitive Control One of These developmental patterns have inspired neuro-
the most consistent observations in the development of scientists to investigate how different regions in the
cognitive capacities across childhood and adolescence brain work together when children, adolescents, and
is the rapid increase in cognitive-control functions, oth- adults make decisions.
erwise referred to as “executive functions.” Cognitive-
control functions are capacities that allow one to keep The neurocognitive development of
relevant information in mind in order to obtain a future cognitive control
goal (Huizinga, Dolan, & van der Molen, 2006). In early
development, there is a marked improvement in a wide Basic Cognitive Control: Working Memory and
range of cognitive-control functions, such as the ability Response Inhibition The basic components of
to store and manipulate information in one’s mind to cognitive control consist of several processes. Most
inhibit responses, to filter irrelevant information, and developmental cognitive-control research has focused
to switch between tasks (Zelazo, 2004). This improve- on the processes of working memory and response
ment continues during school-aged development, when inhibition.
children refine their cognitive-control functions, with
adult levels achieved around mid-adolescence. These Working memory Drawing from the adult literature
cognitive-control functions are of crucial importance (D’Esposito, 2007), several developmental neuroimag-
for all kinds of daily activities, and for school perfor- ing studies have examined the role of the lateral pre-
mance. For example, working memory, a key compo- frontal cortex (Brodmann area [BA] 44 and BA 9/46)
nent of cognitive control, predicts future academic and posterior parietal cortex (BA 7) in working memory
performance in areas such as reading and arithmetic performance and development. These studies have
(St Clair-Thompson & Gathercole, 2006). reported that when 8- to 12-year-old children and adults
adolescents and 25- to 30-year-old adults (Kleibeuker et make decisions, research has often focused on how
al., 2013). Participants were instructed to find solutions children, adolescents, and adults process rewards.
for Match Stick problems by removing matches to find Reward processing has been examined in the context
new spatial shapes (e.g., remove six matches to make of risk-taking, based on the observation that adoles-
two squares). Adolescents slightly outperformed adults cents are more prone than children and adults to take
in finding spatial solutions and showed more activation risks in daily life (Steinberg, 2011). Laboratory studies
in the dorsolateral prefrontal cortex while solving these have demonstrated an age-related reduction in risk-
problems. This same area correlated positively with the taking (Crone, Bullens, van der Plas, Kijkuit, & Zelazo,
number of solutions found. Such findings suggest that 2008; van Duijvenvoorde, Jansen, Bredman, & Huiz-
the flexible recruitment of the lateral prefrontal cortex enga, 2012), but also nonlinear age effects, suggesting
in adolescence is adaptive for successful performance, that adolescents take more risks than children and
such as in the case of spatial creativity. adults when there is a strong affective context (Burnett,
Bault, Coricelli, & Blakemore, 2010; Figner, Mackinlay,
The neurocognitive development of affective Wilkening, & Weber, 2009). The specificity of these
decision making developmental differences has been studied in more
detail using neuroimaging, for example, during studies
Risks and Rewards To understand how affective that aim to differentiate between different phases of
context influences the way we control our actions and risk-taking.
sensitive time for risk-taking, but also brings opportuni- Conclusion and future directions
ties for exploration and adaptive learning.
Crone and Dahl suggested that puberty could be a This chapter has described the neural correlates of
driving force for heightened affective sensitivity in ado- cognitive and affective decision making in school-
lescence (Crone & Dahl, 2012). Extensive animal research aged children, adolescents, and adults. Literature
has pinpointed the timing of puberty in rodents and relating to cognitive control shows that the develop-
reported specific effects of puberty on brain function and ment of basic to complex levels of control follows a
structure (Spear, 2011). Furthermore, pubertal hormones pattern of specialization with age in the prefrontal
were found to have a steering influence on the structural cortex and the posterior parietal cortex, such that
development of the human brain (Ladouceur, Peper, these areas are more selectively recruited for specific
Crone, & Dahl, 2012). Finally, puberty influences affec- tasks. The transition from widespread to focused net-
tive responses to reward in the ventral striatum, inde- works takes place during adolescence, a period of
pendent of age (Forbes et al., 2010; Op de Macks et al., change in explorative learning. This development
2011). Puberty has a strong influence on the way we coincides with increased affective sensitivity in mid-
process affective and social information, preparing ado- adolescence to affective cues, pinpointing nonlinear
lescents to obtain independence and adapt quickly to contributions of control and affective brain regions
changing social contexts. Therefore, pubertal develop- in development. This integrative approach, in which
ment may be an important contributor to the increased the development of cognitive control and decision
sensitivity to affective information, which, together with making are studied in combination, is expected to
flexibility in recruitment of prefrontal cortex, may facil- allow for a richer description of adolescent brain
itate explorative learning (figure 3.2). development.
ABSTRACT Adolescence is a period of formative biological and complex, such as understanding others’ mental states
social transition. Social cognitive processes involved in navi- (theory of mind/mentalizing; Blakemore, den Ouden,
gating increasingly complex and intimate relationships con-
Choudhury, & Frith, 2007), social emotion processing
tinue to develop throughout adolescence. Here, we review
neuroimaging studies of social cognitive processes across (Burnett, Bird, Moll, Frith, & Blakemore, 2009), and
development during adolescence. These studies show that negotiating complex interpersonal decisions (Crone,
areas of the social brain undergo both structural changes and 2013). In this chapter we focus on the more complex
functional reorganization during the second decade of life, social cognitive processes. Recent behavioral and neu-
possibly reflecting a sensitive period for adapting to the social roimaging studies have shown that these processes con-
environment.
tinue to develop past childhood and throughout
adolescence (Apperly, 2010; Blakemore, 2012).
Adolescence is defined as the period between the onset
of puberty and the achievement of relative self- Adolescent social environment
sufficiency. The adults who emerge from adolescence
must be equipped to navigate the social complexities of Adolescents go through a period of “social reorienting,”
their community. Therefore, it has been proposed that during which the opinions of peers become more
adolescence is a time of particular sensitivity to social important than those of family members (Larson &
signals (Choudhury, 2010; Fiske, 2009) and that the Richards, 1991; Larson, Richards, Moneta, Holmbeck,
impact of puberty on the brain makes adolescents par- & Duckett, 1996). This enhanced sensitivity to the social
ticularly sensitive to their social environments (Crone environment and the rewarding nature of peer accep-
& Dahl, 2012; Peper & Dahl, 2013). Knowledge about tance is reflected in studies showing that adolescents
the development of the adolescent brain has signifi- are particularly susceptible to the negative conse-
cantly increased in the last 20 years, and there is now quences of social exclusion (Sebastian, Viding, Wil-
extensive evidence that the brain undergoes substantial liams, & Blakemore, 2010). Adolescents aged 13–17
structural and functional change during this transition years reported that peer evaluations affected their feel-
from childhood to adulthood, including in networks of ings of social or personal worth, and that peer rejection
brain regions involved in social cognition (Blakemore, indicated their unworthiness as an individual (O’Brien
2008; Mills, Lalonde, Clasen, Giedd, & Blakemore, & Bierman, 1988). While the adolescents and children
2013). This chapter reviews neuroimaging studies of aged 10–13 years similarly felt that peers provided com-
social cognitive processes across development, with panionship, stimulation, and support, the younger
emphasis on the changes occurring in mentalizing— group did not indicate that peer acceptance impacted
that is, the attribution of mental states to other people— self-evaluation. O’Brien and Bierman suggest that
during adolescence. increasing abilities to form abstract representations, as
well as increasing motivation for peer acceptance, might
Social cognitive development account for the influence of peer evaluations on self-
evaluations in adolescence (O’Brien & Bierman, 1988).
Social cognition refers to the ability to make sense of These self-reported accounts of the importance of peer
the world through processing signals generated by acceptance are supported by the results of a behavioral
members of the same species (Frith, 2008). Social cogni- study investigating the effects of social exclusion in the
tion includes basic perceptual processes such as face lab. After being excluded by other players in a labora-
processing (Farroni et al., 2005), biological motion tory ball-throwing game called “Cyberball,” young and
detection (Pelphrey & Carter, 2008), and joint attention mid-adolescents (11–15 years) reported lowered overall
(Carpenter, Nagell, & Tomasello, 1998)—all of which mood and young adolescents (11–13 years) reported
are developing rapidly from birth (see chapter 1, this higher state anxiety compared with adults (Sebastian
volume). Other social cognitive processes are relatively et al., 2010). Thus, it appears that complex social
developmental trajectories of the social brain network. et al., 2012), and thinking about the preferences and
Gray matter volume and cortical thickness in medial dispositions of oneself or a fictitious story character
Brodmann Area 10 (a proxy for dmPFC), TPJ, and (Pfeifer et al., 2009). In some studies, higher activity in
pSTS showed continued reductions from childhood more posterior regions, such as the pSTS/TPJ (Blake-
into the early twenties, whereas the ATC continued to more et al., 2007) and the ATC (Burnett et al., 2009),
increase in gray matter volume until adolescence and was observed in adults as compared to adolescents.
in cortical thickness until early adulthood. This pro- These changes in functional recruitment are hypothe-
tracted development demonstrates that areas of the sized to reflect changes in neurocognitive strategy and/
brain involved in deciphering the mental states of or neuroanatomy (Blakemore, 2008).
others are still maturing in terms of their structure In an adapted version of the Director Task (Apperly
from late childhood to early adulthood. et al., 2010; Dumontheil et al., 2010), areas of the social
brain network were engaged when participants had to
Functional Development of the Social Brain use social cues to select an appropriate action in a com-
Network There have been a number of functional municative context (Dumontheil, Hillebrandt, Apperly,
MRI (fMRI) studies that show functional changes across & Blakemore, 2012). While both adults and adolescents
adolescence in the brain networks associated with social recruited the dmPFC when the social cues were
cognition, including mentalizing, social emotion, peer needed to accurately perform the task, adolescents also
evaluation, and peer influence. We discuss these studies recruited the dmPFC when social cues were not needed.
below. The authors suggest that this engagement of the dmPFC
in social conditions, even when social signals are irrel-
Mentalizing Several fMRI studies on mentalizing report evant, may reflect the use of brain regions involved in
decreases in dmPFC recruitment between adolescence mentalizing even when they are not necessary during
and adulthood (Blakemore, 2008, 2012). These studies adolescence.
have used a variety of tasks that require mental state Adolescents also show developmental changes in sen-
attribution, such as understanding irony (Wang, Lee, sitivity to the perspectives of others. In an fMRI study,
Sigman, & Dapretto, 2006), thinking about social emo- young adolescents (12–14 years), older adolescents
tions like guilt (Burnett et al., 2009), understanding (15–17 years), and emerging adults (18–22 years) com-
intentions (Blakemore et al., 2007), understanding pleted a social exchange game in which participants
emotions from photographs of eyes (Gunther Moor were the “second players” in an investment game (van
ABSTRACT Three of the most distinctive attributes of Homo the cultural accumulation of complexity. Our facility
sapiens are language, complex technology, and our degree with technology thus depends in part on our capacity
of reliance on social learning. It is an open question
to learn from others through observation and imita-
whether these human faculties reflect a proliferation of dis-
tinct cognitive adaptations or the diverse behavioral expres- tion. Imitation involves reproducing both the end result
sion of a smaller number of key underlying capacities. By of an observed action as well as the specific movements
comparing modern human brains with the brains of other used to achieve it (Hecht et al., 2012; Visalberghi &
living primate species, we can glean insights into the neuro- Fragazy, 2002). Although chimpanzees are capable of
biological changes that evolved to support these abilities. In imitation, they more commonly emulate, reproducing
particular, comparison with our closest living primate rela-
tive, the chimpanzee, is crucial for reaching conclusions the end result of observed actions but not the specific
about human brain evolution. The recent advent of nonin- movements used to achieve them (Tennie, Call, &
vasive neuroimaging techniques has opened new possibili- Tomasello, 2009). Humans are exceptional imitators
ties for such comparative studies. These and other studies and will even over-imitate by reproducing movements
collectively suggest that language, technology, and aspects of in an observed action that do not contribute to reach-
social learning draw on overlapping regions of association
cortex in prefrontal, temporal, and parietal regions that
ing its end result (Whiten, McGuigan, Marshall-Pescini,
expanded and became functionally specialized during & Hopper, 2009). This expertise in social learning is
human evolution. essential for the social transmission of knowledge upon
which human culture is based (Boyd, Richerson, &
Like all primate species, humans have species-specific Henrich, 2011). It is an open question to what degree
cognitive and behavioral attributes. Three of the most human faculties for language, technology, and social
distinctive attributes of Homo sapiens are language, learning reflect a proliferation of distinct cognitive
complex technology, and our facility with and propen- adaptations as opposed to the diverse behavioral expres-
sity for social learning. While other species communi- sion of a smaller number of key underlying capacities
cate, none does so by combining thousands of symbols (Stout & Chaminade, 2012).
according to a defined set of rules to generate phrases It is thus natural to ask how the human brain was
with a nearly infinite variety of meanings (Pinker, 2000). modified throughout evolution to enable these unique
Humans are also uniquely technological. Although faculties. The fossil record shows that brain size approx-
nonhuman primates are capable of simple tool use, imately tripled over the last 2.5 million years of human
using objects to implement motor to mechanical trans- evolution (Holloway, Sherwood, Rilling, & Hof, 2008),
formations that amplify movements of the upper limbs, and this may be part of the explanation. However, there
humans are far superior at complex tool use, in which may also have been important evolutionary changes in
objects are used to convert movements of the hands the internal organization of the brain, changes that
into mechanical actions qualitatively different from would be difficult to identify in fossils (Holloway, 1968).
manual actions (Frey, 2007). Humans also excel at tool To investigate internal changes, we must turn to the
making. Not even the most highly enculturated modern comparative study of the brains of living primate species.
great apes approach the stone tool-making abilities of If we can identify a characteristic of the human brain
our ancestors 2.5 million years ago (Schick et al., 1999). that is not found in the brain of any closely related
Only in humans are complex tool use and tool making primate species, then we can infer that the trait evolved
elaborated into truly “technological” systems involving in the hominin lineage after we diverged from our
socially coordinated action, intentional teaching, and common ancestor with chimpanzees some 5 to 7 million
on the homologue of Wernicke’s area in the posterior implicating the arcuate fasciculus as the key substrate
superior temporal gyrus. Humans, however, also possess for human language evolution. Furthermore, the
a massive projection of the arcuate into the middle and human arcuate fasciculus is leftwardly asymmetric,
inferior temporal gyri, ventral to classic Wernicke’s area while the extreme capsule is not (Rilling, Glasser,
(Rilling et al., 2008). Jbabdi, Andersson, & Preuss, 2011).
These projections lie within a region of temporal In summary, specializations of the human brain that
association cortex that seems to have expanded in may be relevant to explaining our capacity for language
human evolution, displacing the nearby extrastriate include (1) wider cortical minicolumns in both Broca’s
visual cortex in the process. For example, the middle and Wernicke’s areas, (2) leftward asymmetries in
temporal visual motion area, or MT, lies within the Broca’s area volume and PT minicolumn width, and 3)
superior temporal sulcus (STS) in chimpanzees and arcuate fasciculus projections beyond Wernicke’s area
rhesus macaques. In humans, however, MT is located to a region of expanded association cortex in the middle
posterior to the STS, which instead contains associa- and inferior temporal cortex involved in processing
tion cortex (Glasser et al., 2013). The posterior limit of word meaning (figure 5.1).
temporal-lobe arcuate fasciculus terminations in the
human brain coincides very closely with the anterior Technology
limit of visual motion area MT, consistent with dis-
placement of MT by the highly expanded arcuate Many species make and use tools, but humans are dis-
pathway. tinguished by the extent and complexity of their tech-
The region of expanded cortex that receives arcuate nological behavior. In fact, the term “technology”
projections has been dubbed an “epicenter for lexical- properly refers to a complex assemblage of artifacts,
semantic processing,” based on lesion, functional MRI institutions, social relations, knowledge, and skills that
(fMRI), and structural and functional connectivity data has no strict counterpart in nonhumans. Comparative
(Turken & Dronkers, 2011). Thus, this portion of the studies are critical to unraveling the evolutionary origins
arcuate fasciculus may carry lexical-sematic information of this uniquely human mode of behavior. Technologi-
to Broca’s area. Although some have postulated that cal behavior may be heuristically divided into simple
evolution recruited the extreme capsule pathway into tool use, complex tool use, and tool making.
the language system and that this pathway has played a
key role in language evolution, comparative DTI data Simple Tool Use In humans and macaques, grasping
suggest that the evolutionary expansion of the arcuate and manipulation depend on circuits that connect the
fasciculus outstripped that of the extreme capsule, intraparietal sulcus with the inferior frontal cortex via
the third branch of the superior longitudinal fasciculus semantic knowledge about tool form and function.
(SLF III). As macaques learn to use rakes to extend These streams converge in the inferior parietal lobule
their reach, the receptive fields of visuotactile neurons (IPL) to integrate the action and semantic knowledge
in the intraparietal sulcus increase over time (Iriki, required for complex tool use. This information is com-
2006). It has been argued that this reflects an extension municated via SLF III to the inferior frontal cortex,
of the body schema to incorporate the handheld tool, which supports hierarchically structured action sequenc-
and that simple tool use may be accounted for by ing during production and perception of object-
straightforward extension of the sensorimotor mecha- directed actions (Fadiga, Craighero, & D’Ausilio, 2009).
nisms involved in prehension. Below, we present evidence that these tool-relevant
regions of temporal, parietal, and frontal cortex are
Complex Tool Use Body schema extension is insuf- relatively larger and more extensively interconnected in
ficient to explain the use of complex tools that alter the humans than in other primates (figure 5.2).
functional properties of the hand (e.g., knives, hammers, Evidence for temporal lobe expansion is discussed
potholders), which require an additional causal under- above in the context of language. Human IPL is simi-
standing of tool properties as distinct from the hand larly expanded relative to macaques (Orban et al.,
(Frey, 2007). Complex tool use is understood with refer- 2006) and chimpanzees (Glasser et al., 2013). Finally,
ence to the “two-streams” account of visual perception human Broca’s area displays wider minicolumns than
(Milner & Goodale, 1995). A “dorsal stream” from its ape homologue (Schenker et al., 2008), is one of the
extrastriate visual cortices to the posterior parietal lobe most greatly expanded cortical regions in humans com-
supports visuospatial-motor transformations for action, pared to apes, particularly in the left hemisphere
whereas a “ventral stream” from occipital to posterior (Schenker et al., 2010), and displays greater bilateral
temporal cortices maps visual percepts to stored connectivity with IPL via SLF III (Hecht et al., 2012).
cortex (via the middle/inferior longitudinal fascicu- temporal cortex; however, skill is represented in infe-
lus), and between inferior parietal cortex and inferior rior parietal cortex, which is functionally specialized for
frontal gyrus (via SLF III; figure 5.3). If IFG processes specific aspects of tool use. Both the inferior parietal
the goals of actions, then the ventral connections that cortex and the pathway linking it with ventral lateral
dominate macaque and chimpanzee brains would be temporal cortex, which presumably integrates these
involved mainly with linking the observed action with dorsal and ventral streams of information about tools,
its goal or intention. These pathways may primarily have also expanded disproportionately in human evolu-
support emulation. Inferior parietal cortex is involved tion. Whereas tool use seems to be lateralized to left
in the spatial mapping of movement and may be useful hemisphere parietofrontal systems, tool making is more
for extracting more kinematic details from observed reliant on the right hemisphere homologues of these
actions. This pathway might support reproducing the areas. Finally, imitation may depend on connections
specific actions that lead to the goal, and its greater between the superior temporal sulcus and the inferior
development in humans may explain our penchant for frontal gyrus that run via the inferior parietal lobe, a
imitation (Hecht et al., 2012). Thus, the connectivity of pathway that is better developed in humans than non-
the inferior parietal cortex, another region of associa- human primates. More fine-grained studies are needed
tion cortex that appears to have expanded in human to determine if the cortical substrates supporting these
evolution (Bruner, 2004; Glasser et al., 2013; Orban et different human specializations are indeed the same, or
al., 2006), emerges as potentially crucial for the human merely in close proximity to one another.
capacity for learning through imitation.
Conclusion
REFERENCES
The distinctive attributes of human language, complex Aitken, P. G. (1981). Cortical control of conditioned and
technology, and reliance on social learning draw on spontaneous vocal behavior in rhesus monkeys. Brain Lang,
overlapping regions of association cortex in prefrontal, 13(1), 171–184.
Boyd, R., Richerson, P. J., & Henrich, J. (2011). The cul-
temporal, and parietal regions that have expanded and tural niche: Why social learning is essential for human
become functionally specialized during human evolu- adaptation. Proc Natl Acad Sci USA, 108(Suppl 2),
tion. Specializations supporting human language 10918–10925.
include microstructural specializations of Wernicke’s Bruner, E. (2004). Geometric morphometrics and paleoneu-
and Broca’s areas, as well as expansion of ventral lateral rology: Brain shape evolution in the genus Homo. J Hum
Evol, 47(5), 279–303.
temporal cortex involved in lexical-semantic processing
Buxhoeveden, D. P., Switala, A. E., Litaker, M., Roy, E., &
and its connection with Broca’s area via the arcuate Casanova, M. F. (2001). Lateralization of minicolumns in
fasciculus. Semantic knowledge of tools seems also to human planum temporale is absent in nonhuman primate
be represented within the expanded ventral lateral cortex. Brain Behav Evol, 57(6), 349–358.
ABSTRACT In The Neurobiology of the Prefrontal Cortex, Passing- (Rose, 2006), and their visual advances probably pro-
ham and Wise build on the work of Preuss and Goldman- vided improved depth perception and object detection,
Rakic, who found that new prefrontal areas emerged during
as well as vision below and in front of their snout
primate evolution. We propose that the new prefrontal areas
of early primates perform search and valuation functions, (Allman, 2000; Barton, 1998, 2004; Changizi, 2009).
and that later, in anthropoid primates, additional prefrontal Among their motor specializations, early primates
areas evolved to perform related functions. The new anthro- used their hindlimbs more in locomotion than other
poid areas augment ancestral reinforcement-learning mecha- mammals and generated less force to move from place
nisms, which use feedback events to update the state of to place (Larson, 1998; Schmitt, 2010). The latter trait
memories through the cumulative adjustment of associa-
tions, averaged over many events. In contrast, the anthropoid minimized predator-attracting movements of flimsy
prefrontal areas store the memory of single events and use branches, and the former enabled forelimb specializa-
event representations to reduce the number of unproduc- tions for reaching and grasping. Like many modern
tive, risky, or costly foraging choices. They do so by using the prosimians, early primates probably used a hand-to-
memory of events to speed learning and by using event rep- mouth feeding technique (MacNeilage, Studdert-
resentations to generate goals from abstract rules and strate-
gies, thus bringing previous solutions to bear on novel
Kennedy, & Lindblom, 1987), one that required
problems. coordination among head, mouth, and hand move-
ments (Wise, 2007). Primates also reach in a unique way
(Shadmehr & Wise, 2005), which depends on a suite of
Passingham and Wise (2012) present a proposal on the new cortical areas (Preuss, 2007a, 2007b), a topic taken
fundamental function of the primate prefrontal cortex up later.
(PF), which this chapter presents in summary form.
Figure 6.1 illustrates the terminology used to explain Anthropoid evolution
our idea. Primates include strepsirrhines and haplo-
rhines; haplorhines consist of tarsiers and anthropoids; Early anthropoids engaged in diurnal foraging, as indi-
anthropoids include platyrrhines (New World monkeys) cated by their relatively small eyes (Fleagle, 1999; Heesy
and catarrhines; and catarrhines comprise Old World & Ross, 2004). Daylight foraging increased the risk of
monkeys, humans, and apes (figure 6.1B). The term predation, as well as competition from other diurnal
“prosimian” refers to tarsiers and strepsirrhines. Figure animals, such as frugivorous birds. Anthropoids inher-
6.1A recognizes seven subdivisions of PF: caudal (PFc), ited the primate fovea, which emerged in haplorhines
orbital (PFo), ventral (PFv), dorsolateral PF (PFdl), (Ross, 2004), and amplified several traits of early pri-
dorsal (PFd), medial (PFm), and polar (PFp). mates, such as large brains, long lives, and frontally
directed eyes. Later anthropoids evolved various forms
Early primate evolution of trichromatic vision, with catarrhines developing the
“routine” form that depends on three distinct photopig-
Most primatologists agree that early primates adapted ment genes (Williams, Kay, & Kirk, 2010).
to life in the fine branches of angiosperm trees and that Early anthropoids were small animals, approximately
their innovations included grasping hands and feet, 100 grams, and tooth-wear patterns suggest that they
nails instead of claws, and frontally directed eyes foraged mostly on fruits and insects (Fleagle, 1999;
(Fleagle, 1999; Rose, 2006). These adaptations proba- Rose, 2006). An increase in size began around 34
bly supported visually guided leaping (Martin, 1990) million years ago, after the divergence of platyrrhines
and foraging (Cartmill, 1972), and fossil evidence sug- and catarrhines (Williams et al., 2010). The first of
gests that the grasping specializations arose first (Bloch these larger anthropoids were probably arboreal quad-
& Boyer, 2002). Early primates foraged nocturnally rupeds (Fleagle, 1999; Kay, Williams, Ross, Takai, &
Anthropoids
PFp PFm PFdl, PFd, PFv New World (platyrrhine) monkeys
granular PFm
CC (?) PFp
PFd Old World (catarrhine) monkeys
PFc
pre
SMA SMA gibbons
haplorhines
catarrhines
orangutans
Primates fovea
gorillas
PMd PFc 17%
M1 ‘routine’
granular PFo
PMd, PMv trichromacy chimpanzees & bonobos
PFc
PFd preSMA, SMA
PFdl
PMv IT humans
PP
PFp PFv
tarsiers
prosimians
strepsirrhines
bush babies
lorises
lemurs
Figure 6.1 (A) PF regions in macaques. (B) Primate clado- supplementary motor area; c, caudal; d, dorsal; dl, dorsolat-
gram showing shared derived traits (black bars). The gray eral; o, orbital; p, polar; m, medial; v, ventral. Lines in
lettering and bracket in (B) indicate a paraphyletic group. (A) reproduced from Petrides and Pandya (2007) by permis-
Circle diameter proportional to percentage of cortex com- sion of the Society for Neuroscience. (A) and (B) reproduced
prising granular PF. Abbreviations: IT, inferotemporal from Passingham and Wise (2012) by permission of Oxford
areas; PP, posterior parietal areas; PF, prefrontal cortex; University Press.
PM, premotor cortex; M1, primary motor cortex; SMA,
Shigehara, 2004), a trait that many modern anthro- premotor and posterior parietal areas, along with a
poids have retained (Schmitt, 2010). This kind of loco- dozen or more new visual and auditory areas (Kaas,
motion requires considerable energy, especially with 2012). In addition, a new part of the PF cortex emerged,
frequent changes in elevation. the main topic of this chapter. As Preuss (1995) pointed
As anthropoids increased in size, they began to forage out, this part of the PF cortex has a prominent internal
over a larger home range, an energy-intensive activity granular layer, and so can be called the granular PF
(Martin, 1981). To meet their needs, they probably cortex. This name, however, reflects just one of its
obtained much of their nutrition from ripe, high-energy diagnostic traits; as reviewed elsewhere, extensive addi-
fruits and tender, easily digestible leaves. Like modern tional evidence confirms that the granular PF cortex
anthropoids, these animals foraged among thousands occurs uniquely in primates (Passingham & Wise, 2012;
of angiosperm trees, few of which had ripe fruit on any Preuss, 1995, 2007a, 2007b; Wise, 2008).
given day. Among their many foraging challenges, trees Preuss and Goldman-Rakic (1991a, 1991b) provided
of a given species generated fruit at various times and the key support for this idea. They concluded that
frequencies, with substantial annual variation and rip- agranular PF areas have homologues in all mammals
ening rates that varied by location (Janmaat, Byrne, & and that PFc and PFo emerged as new granular areas
Zuberbühler, 2006). When shortfalls in preferred nutri- in early primates. First, they identified the granular PF
ents occurred, competition intensified, as did the risk cortex in the bush baby, a strepsirrhine primate. As in
of predation. anthropoids, these granular areas are situated rostral
to the premotor cortex (area 6), a group of agranular
Cortical evolution in primates areas. Second, they compared the connections, cytoar-
chitectonics, and myeloarchitectonics of the frontal
Early Primates As early primates adapted to a noc- cortex in bush babies with anthropoid monkeys
turnal, fine-branch niche, they developed several new (macaques) and other mammals. They concluded that
60
Figure 6.2 (A) Brain versus body size in modern primates points for anthropoids and strepsirrhines, respectively.
and selected fossils: A, Aegyptopithecus; C, Chilecebus; P, Para- (B) Granular PF as percentage of frontal cortex. Data for (A)
pithecus; T, tarsiers. Shadings and dashed-line bound data from Bush et al. (2004); data for (B) from Elston et al. (2006).
durations, distances, speeds, and locations of visual errors in novel situations: the former by solving a
events (Passingham & Wise, 2012). These areas also problem quickly, the latter by transferring solutions
contribute to the control of attention, in part by ori- from similar problems to novel ones. A reduction of
enting the fovea (Paré & Dorris, 2011). errors results in more productive foraging choices, less
In support of the idea that granular PF cortex wasted energy, and lower risk of predation.
expanded in anthropoids, Preuss and Goldman-Rakic Our proposal relies partly on neuroimaging and neu-
(1991b) made two key observations in addition to those rophysiological findings, but the most direct evidence
cited earlier: many of the granular PF areas in macaque comes from the effects of selective brain lesions, as sum-
monkeys have a paucity of myelin, and bush babies lack marized in the next section.
large myelin-poor regions within their granular PF.
Chimpanzees and humans, like anthropoid monkeys, Using events to reduce foraging errors
also have myelin-poor regions within their PF (Glasser,
Goyal, Preuss, Raichle, & Van Essen, 2014). The myelin- Credit Assignment As noted earlier, PFo functions
poor parts of granular PF were probably, therefore, in updating the mappings between objects and the out-
anthropoid innovations. comes that follow the appearance or choice of those
objects. PFo receives high-level, conjunctive visual rep-
Proposal resentations of objects and outcomes from inferotem-
poral cortex (IT); olfactory, gustatory, and visceral
Anthropoid innovations include several parts of granu- information about outcomes from agranular parts of
lar PF: PFv, PFdl, PFd, the rostral, granular parts of orbital and insular cortex; and signals that allow it to
PFm, and probably PFp. Passingham and Wise (2012) update the value of these outcomes from the amygdala
suggest specialized roles for each of these areas, but (Izquierdo et al., 2004).
their most important contribution comes from the inte- Walton et al. (2010) provided evidence that monkeys
grated function of granular PF as a whole (Wilson, use their memory of single events to choose a foraging
Gaffan, Browning, & Baxter, 2010). Specifically, we goal. They studied the choices that monkeys made
propose that the primate PF cortex generates the goals among three objects that produced rewards at a vari-
appropriate to a current context and to current biologi- able rate. Normal monkeys can generate appropriate
cal needs, and that it can sometimes do so based on a choices based on a memory of which previous choice
single event. seemed to cause a particular outcome. In contrast,
As a result, the new granular PF areas of anthropoids monkeys with lesions of the granular PFo were more
reduce foraging errors, which they accomplish in at affected by the mappings between objects and out-
least two ways: through fast learning and by applying comes that accumulate across trials, as an average of
abstract rules and strategies. Both capacities reduce several events. The choices of lesioned monkeys thus
Percent error
Percent error
30 30
20 20
10 10
0 0
Pr
PF
Pr
PF
C
PF
PF
on
on
e-
-o
po
x
op
tro
tro
IT
IT
IT
st
l
-o
po
p
st
-o
p
C D E
Conditional motor learning Abstract strategies Temporal extended events
80 0.8 40 ‘Binding’ Blank
chance
Strategy application
Errors to criterion
stimulus screen
Percent error
60 0.6 30
(perfect = 1)
40 0.4 20
10
20 0.2
0 0 0
PF
PF
Pre-op PFv+o Pre-op PFv+o
on
on
x
x
tro
tro
post-op
IT
IT
post-op
l
Figure 6.3 (A) Learning set, performance on trial 4. et al. (2007); for (B), from Browning et al. (2005); for (C) and
(B) Object-in-place scenes task, performance on trial 4. (D), from Bussey et al. (2001); for (E), adapted from Wilson
(C) Conditional motor learning, mean over 48 trials. (D) Appli- et al., 2010, with permission from Elsevier; reproduced
cation of the change-shift strategy, first eight change trials. from Passingham and Wise (2012), by permission of Oxford
(E) Temporally extended events. Data for (A) from Browning University Press.
resemble those of nonprimate mammals and other ver- concurrent discrimination task. Monkeys learn much
tebrates, which learn associations between responses faster and make fewer errors on the serial version of the
and outcomes through cumulative experience. The task (figure 6.3A, gray). In general, their high error rate
capacity for learning the outcome caused by a single on the concurrent task resembles that of naive monkeys
event provides an advantage in generating future forag- on the serial task. Murray and Gaffan concluded that
ing choices. when experienced monkeys face the serial discrimina-
tion task, they choose their next goal based on events
Discrimination and Reversal Learning Set With from the previous trial. If the choice of an object pro-
experience, macaque monkeys can make the correct duced a reward, then the monkey will choose that
choice between two objects on the second opportunity object as the goal for the next trial; otherwise, it chooses
to make that choice, a capability called “learning set.” the alternative object. Monkeys can then maintain the
These experiments usually involve the serial discrimina- goal in short-term memory until the opportunity comes
tion task, in which the same two stimuli appear trial to make the next choice, a process called prospective
after trial. In the concurrent discrimination task, by coding. The concurrent discrimination task precludes
contrast, monkeys choose between two objects and then this strategy by imposing intervening objects.
many trials with different objects intervene before The development of a strong learning set depends
monkeys face the same choice again. on granular PF and its interactions with IT (Browning,
Murray and Gaffan (2006) found that monkeys fail Easton, & Gaffan, 2007). Disconnection lesions that
to develop a strong learning set when tested on the block interactions between PF and IT slow learning and
ABSTRACT Despite advances in in vivo human studies, neuro- The purpose of this essay is to critique the MOP. We
scientists still depend on studies of nonhuman species for emphasize that we are not arguing against animal
information that cannot be obtained without invasive tech-
research. Rather, we identify limitations and liabilities
niques. This research is concentrated in a very few “model”
species, mostly macaque monkeys, rats, and mice. Although of animal research conducted within the current frame-
it is well documented that brain organization varies markedly work of the MOP, and suggest ways to modify science
across mammals, species differences are perceived as threat- policy so as to enhance our understanding of humans
ening to model-organism research and so are often ignored through the study of other animals.
or, if acknowledged, are interpreted in the context of a
simple-to-complex phylogenetic progression that is at odds
with the modern, branching-tree conception of evolution. What model organisms do for us
We argue that the failure of the model-organism paradigm
to deal forthrightly with phyletic variation can compromise The advancement of knowledge in the biological and
the integrity of research by slanting the generation, report- behavioral sciences has been pursued primarily with
ing, and interpretation of results, and by placing the enter- research on a relatively small number of model organ-
prise outside the bounds of evolutionary plausibility. We
propose that many of the deficiencies of the model-organism
isms. Model organisms are physical representations of
paradigm could be remedied, and the quality of inferences phenomena of interest, and those representations are
about humans improved, by increasing the number and phy- of several kinds (Bolker, 2009; Preuss, 2001): a species
logenetic diversity of core experimental species and embed- might serve as a disease model—that is, as a proxy or
ding model research in a broader comparative framework. in vivo replica of a disease, symptom, or disease process
(e.g., the rat model of Parkinson’s disease), or as an
Research in cognitive neuroscience primarily involves exhibitor of some widespread phenomenon (e.g., a
studying humans with one of several noninvasive model for studying neurogenesis, a model of prefrontal
imaging techniques. Nevertheless, the field is still very cortex), or a general-purpose stand-in for other species.
dependent on experimental studies of nonhuman In any case, the goal is to advance research by standard-
species for obtaining certain kinds of information, espe- izing experimental methods and materials (Robert,
cially small-scale features of brain organization, and 2008a).
information that can only be obtained with invasive Current model organisms were established for
techniques that are considered ethically unacceptable primarily pragmatic reasons: they are lab-friendly
for use in humans. Given this dependence, what is the (relatively cheap, small, and generally susceptible to
best strategy for studying animals in order to under- husbandry), experimentally tractable (relatively easy
stand human beings? For most scientists, the answer is to observe and manipulate), and standard (the ones
obvious: focus on one of the widely used “model” that everyone uses). Model organisms canalize entire
species. In the neurosciences, those would be rats and domains of research because they create research effi-
mice, mainly, but also macaque monkeys and other ciencies, including beneficial economies of scale and
nonhuman primates (Manger et al., 2008). This reliable replication of results (Robert, 2004b, 2008a).
approach, which we will refer to as the “model-organism The “model organism” honorific is most properly
paradigm” (Preuss, 2000), or MOP, is the dominant applied to those animals sanctioned by the National
animal-research paradigm not only in the neurosci- Institutes of Health (NIH) and other funders, and
ences, but in the biomedical sciences generally, as around which a model “system” has grown (including
reflected in its support by funding organizations such research networks, literatures, databases, and so on).
as the National Institutes of Health (cf. Robert, 2008a, Thus, model organisms are more than just animals: they
on “model organism-ism”). are the foci of human institutions (e.g., Logan, 2005b;
preuss and robert: animal models of the human brain: repairing the paradigm 59
Rader, 2004). When scientists achieve prominence principle” by Hans Krebs (1975), although Krogh would
through their research on a particular model, they probably not have endorsed Krebs’s view that species
attract students who typically work on that model. differences are “minor modifications” of a standard
Funding agencies take notice and direct resources to animal plan (Logan, 2002). It is the MOP’s default
research on the model. Research know-how on the assumption of generality that justifies the concentration
model accumulates and supporting infrastructures of effort on internal validity and repeatability rather
develop. The community develops an ideology about than on external validity (see also van der Staay, Arndt,
the best way to do research and how it applies to & Nordquist, 2009).
humans. Members of the model-organism community— This ideology promotes a kind of species neglect. Since
researchers, administrators, students—develop vested species are thought not to matter in any fundamental
interests in the political as well as scientific success of way, titles of neuroscience papers commonly do not
their model, interests that, we argue, influence their name them—as one can easily confirm by scanning the
view of biological reality. contents of the Journal of Neuroscience—and species are
Typically, organisms destined to become models sometimes omitted from abstracts as well. Instead,
come to the attention of scientists owing to some scien- papers focus on the structure or function of particular
tifically interesting or methodologically convenient biological entities, such as brain areas and genes. These
characteristic. For example, Norway rats came to the entities are thus presented as independent of the species
attention of researchers because they mature quickly that harbor them, although it is well known that the
and were perceived as being hypersexual, at a time phenotypic effects of a particular gene or protein, or
when researchers were beginning to study the physiol- the functional properties of a particular brain structure,
ogy of sex (Logan, 2001). However, as Logan has can differ dramatically between even closely related
documented, when rats became domesticated through species (see, e.g., Krubitzer, Campi, & Cooke, 2011;
generations of breeding, and researchers became McNamara, Namgung, & Routtenberg, 1996; Rekart,
increasingly reliant on their use, they underwent a Sandoval, & Routtenberg, 2007; and, generally, Robert,
remarkable transformation in the minds of scientists, 2004a, 2008b). The reality of species differences is
their rodent- and species-specific characteristics receiv- further highlighted when models come into competi-
ing progressively less attention and their status as “typical tion with each other, leading to sometimes rancorous
mammals” promoted (Logan, 1999, 2002, 2005a). With disputes about the best model for a particular purpose,
this change, and the availability of pure strains, the as for example in the debate over primate versus rodent
status of rats changed from biological species to labora- models in Parkinson’s disease research (Robert, 2008a).
tory reagent. This is an accurate depiction of model Species neglect in the neurosciences is reinforced by
organisms more generally: they are standardized, explicit doctrine: the claim that there is a “basic unifor-
quality-controlled laboratory materials useful for mity” of neocortical organization, an idea most fully
making experiments work. The purification of these articulated by Rockel, Hiorns, and Powell (1980). “Basic
reagents probably increases the internal validity and uniformity” holds that the numbers of cells and propor-
repeatability of experimental results, but unfortunately tions of cell types in a cortical column are the same
does not assure their external validity—that is, their across species and cortical areas (except the primary
generalizability to other species—and may, in fact, com- visual area of primates), and that the local circuitry of
promise it. the cortex is likewise invariant; functional differences
between areas result from differences in extrinsic con-
What model organisms do to us nectivity. Taken seriously, basic uniformity implies that
for most research purposes, any mammal will do. But
While the benefits of model-organism research are basic uniformity cannot be sustained, as modern
widely recognized, it also has liabilities, especially in the research has revealed numerous differences in intrinsic
neurosciences, where research is concentrated in so few as well as large-scale cortical organization between areas
species. Here, we consider how scientists’ commitment and species (Hof & Sherwood, 2007; Preuss, 2010). Nev-
to models affects how they understand the place of their ertheless, basic uniformity remains an influential concept
models in the phylogenetic scheme, how they view their in neuroscience.
work in the broader context of science, and how they The MOP’s distortions of biology extend into the
represent their findings. realm of evolution (Preuss, 1995a, 2009; Robert, 2004a).
At the heart of the MOP is the assumption that find- Modern evolutionary biology views every species as a
ings obtained in model species will hold for many or mosaic of features, some unique or specific to the
most other species. This was dubbed the “August Krogh species, some shared with close relatives, and some
shared with more distantly related species. The totemic references in the use of terms such as “infrahuman”
organism of modern evolutionary biology is the branch- primates and “lower” mammals, remain common in the
ing tree, representing both the unity and diversity of neuroscientific literature. Even NIH’s animal care and
life. The rise of “tree thinking” in the latter part of the use guidelines recommend using animals “lower on the
twentieth century marked a profound change in the phylogenetic scale” whenever possible (Striedter, 2007).
way evolutionary biologists understand evolution. Trees The fact that the phylogenetic scale remains the
drawn in the nineteenth century and first half of the common-sense, folk understanding of evolution in our
twentieth century looked as much like scales as like culture may explain some of its continuing appeal
trees (figure 7.1) and, indeed, early evolutionists within the MOP. Yet what the MOP has done is to
(including Darwin) understood evolution as progres- promote a way of thinking about the relationship
sion from undifferentiated “lower” forms to the most between humans and other animals incompatible with
complex and perfect “higher” forms: humans, that is basic principles of modern biology—while still passing
(see reviews by Fleagle & Jungers, 1980; Hodos & Camp- for modern biology.
bell, 1969; Preuss, 2009; Striedter, 2007). The phyloge- For scientists working within the MOP, commitment
netic scale compresses the diversity of life onto a single, to a specific model can shape the ways data are inter-
vertical axis of complexity and perfection. The bushier preted and basic concepts are framed. A case in point
trees drawn by modern evolutionists (figure 7.1C) is prefrontal cortex. Primates possess a region of pre-
reflect the rejection of a linear, human-centered view of frontal cortex (PFC) located mainly on the dorsolateral
evolution, emphasizing instead its diversifying charac- aspect of the frontal lobe that has a well-formed granu-
ter, each branch of the tree possessing unique features. lar layer IV: the granular PFC. This region, which is
The MOP is much more compatible with the old view involved in some very high-level cognitive functions
of life than with tree thinking. Explicit references to such as conscious reasoning and planning, comprises
the defunct phylogenetic scale, as well as implicit numerous subdivisions that are nodes in networks
preuss and robert: animal models of the human brain: repairing the paradigm 61
Figure 7.2 Alternative interpretations of frontal-lobe homologies in macaques and rats. Modified from Preuss (2007).
linking granular PFC with higher-order parietal and granular PFC, the medial frontal cortex in rodents, as
temporal areas (reviewed by Preuss, 1995b, and Passing- in primates, is notable for its strong connections with
ham & Wise, 2012). Granular PFC presents a problem subcortical limbic and autonomic structures. Rebrand-
for rodent models, because obvious homologues are ing rostral cingulate cortex as “prefrontal” may enhance
found only in primates (Brodmann, 1909; Preuss, the stature of rodent models by providing them with an
1995b). One proposed solution is to focus on other enlarged region of PFC, but it amounts to little more
features of anatomy that are shared by rodents and than an accounting change, and one that obscures
primates. Since primate granular PFC receives projec- important differences among cortical regions and
tions from the lateral part of the mediodorsal (MD) among species without advancing our understanding
thalamic nucleus, the part of rodent cortex that receives of brain structure and function.
inputs from lateral MD would be a candidate for homol- Not only can models shape conceptual space, they
ogy. In rodents, however, lateral MD projects not to can police access to it, influencing what data are
dorsolateral frontal cortex, but rather to the medial accepted as legitimate and significant. Consider the
frontal cortex, including areas 24, 32, and 25, areas macaque model of the visual system, which has domi-
originally classified as “cingulate.” This has prompted nated our understanding of the neurobiology of human
the claim that rodent areas had been misidentified, and vision. This dominance is no accident: a series of studies
that they actually correspond to granular PFC, or made a point of emphasizing the value of macaques as
perhaps to some amalgam of granular PFC and cingu- proxies for humans, creating the impression that there
late cortex (see, for example, Krettek & Price, 1977; are few differences of consequence between the central
Leonard, 1969; Uylings, Groenewegen, & Kolb, 2003; visual systems of humans and macaques (e.g., de Valois,
and figures 7.2A and 7.2B). Current evidence, however, Morgan, Polson, Mead, & Hull, 1974; Harwerth &
clearly indicates that the medial frontal areas of rodents Smith, 1985; Merigan, 1980). So strong is the influence
resemble the rostral cingulate areas of primates in virtu- of the macaque model that one is hard-pressed to find
ally every feature that can be compared: in both, major references to human-macaque visual-system differences
inputs arise from nucleus MD, both have an agranular in the secondary literature or in textbooks. Moreover,
histology, and lesions in both produce similar behav- its influence has made it seem reasonable to dismiss
ioral effects—whereas these areas bear no diagnostic evidence of human specializations out of hand, on the
similarity to primate granular PFC (Passingham & Wise, grounds that features present in humans but not
2012; Preuss, 1995b; figure 7.2C). In contrast to macaques must be artifacts of poor methods or
materials because similar characteristics had never paradigm. As we have seen, adopting the MOP can bias
been reported in macaques (as an anonymous reviewer the generation of results, because so few species are
explained in critiquing the findings ultimately pub- studied, as well as the reporting and interpretation of
lished by Preuss, Qi, and Kaas, 1999). Yet, while there results. In this way, the MOP challenges the integrity
is no question that macaque and human visual systems of individual researchers and of the larger research
have much in common, the primary literature also con- enterprise, ultimately undermining the evidence base
tains abundant evidence of differences between humans of science. Moreover, the fundamental assumption
and macaques, and attests that these differences are of the MOP—that species differences are trivial—is
found at every level of the visual system, from the retina inconsistent with the modern understanding of evolu-
to association cortex (figure 7.3). Why aren’t these dif- tion, and thus challenges the intellectual integrity of
ferences better known? We suggest that acknowledging biology.
human-macaque differences is perceived as threaten- Additionally, animal research undertaken within the
ing to the monkey model. The lesson in this case, as in MOP can undermine the credibility of animal research
the example of prefrontal cortex, is that when the more broadly, and in so doing abet those who would
model comes into conflict with the thing modeled— limit or eliminate animal research altogether. Not all
humans, that is—the model wins. Alternatively, if some- animal research is equally well justified; poorly justified
thing doesn’t exist in the model, it doesn’t exist. MOP research lacking external validity threatens the
moral acceptability of more epistemically legitimate
Ethical dimensions of the MOP research with nonhuman species. Scientists are obli-
gated to make the best use of nonhuman species in
We briefly note here the ethical hazards resulting from research, and the species neglect endemic to the MOP
the reality-distorting effects of the model-organism works against this goal.
preuss and robert: animal models of the human brain: repairing the paradigm 63
Reforming Research As we see it, the liabilities behavioral ends: for example, certain avian species
and contradictions of the model-organism paradigm appear to have evolved cognitive capacities similar to
stem from the way findings in model species are thought those of primates, despite marked differences in brain
to relate to other species, including humans— structure (e.g., Avian Brain Nomenclature Consortium,
specifically, from the default assumption that any 2005; Emery & Clayton, 2004). How did birds achieve
given finding in a model will generalize to other animals that? Perhaps they share with us emergent, higher-
until proven otherwise. If evolution took the form order features of brain organization, features relevant
of a phylogenetic scale, and all the diversity of life to cognition that we do not yet appreciate even in our
could be collapsed onto a single axis of complexity, own species. Comparative studies can also identify
the default assumption would make sense, and the extreme cases that illustrate processes or design fea-
distinction between internal and external validity tures relevant beyond specific phylogenetic groups. For
would be relatively unimportant. But evolution is like a example, our appreciation of the capacity of cerebral
tree and not a scale, and this fact requires we acknowl- cortex to compartmentalize processing would be dimin-
edge that internal validity is no guarantee of external ished if we did not know that compartmental, modular
validity. Demonstrating the generalizability of results organization evolved repeatedly in mammalian history,
requires that neuroscientists adopt a more comparative star-nosed moles providing a remarkable example of
approach, extending their research broadly and deeply this (Catania, 2006).
to include species outside of the current stable of model
animals. Increase the number of core species In generating an appro-
Let us make clear what we are not proposing. We do priately comparative context for research, we recom-
not advocate eliminating any of the currently favored mend a strategic and systematic approach rather than
model animals, because we acknowledge that there is, a haphazard and idiosyncratic one. Determining which
indeed, value in working with animals that have estab- species are especially well suited for addressing particu-
lished husbandry practices and research systems, or are lar biological problems will and should depend on the
particularly tractable for one purpose or another (for problem itself, not strictly on historical antecedents and
example, making transgenic animals, in the case of canalized research programs. We cannot, of course,
mice). Nevertheless, we believe the concentration of expect to study every species in great detail, or even in
resources in a few highly entrenched models has passed modest detail, but we would benefit from adding species
the point of diminishing returns and that the enter- for concentrated research in a way that increases the
prise would be better served by distributing resources breadth and depth of phyletic coverage. This would
over a larger number of species. We advocate the fol- entail adding more representatives of the rodent and
lowing reforms based on both epistemological and primate orders, adding species from other mammalian
ethical considerations: orders, and adding selected taxa from other animal
groups, especially those meeting the criteria discussed
Embed research in a broader comparative framework Without in the previous paragraph. Species to be favored might
broad comparative studies, it is not possible to deter- also include those for which genome sequences are
mine how particular features of brain organization and available. Access to primates is a particular concern,
function are distributed across the evolutionary tree, or given the reduction of primate species available at
within the human part of it. For a program of research NIH’s National Primate Research Centers over the past
designed to improve our understanding of humans, a two decades, as this represents a serious obstacle to
reasonable approach would, at a minimum, involve obtaining a deeper understanding of primate (includ-
identifying features of brain organization characteristic ing human) biology.
of ancestral mammals, of ancestral eutherian mammals,
of the ancestors of the modern eutherian orders Foster multispecies research and training For the sake of
(including of course Rodentia and Primates), as well as improving external validation, funding institutions
the specialized features of the genera and species from should promote the study of specific processes and phe-
which our model species are drawn. As well as identify- nomena in multiple species (models or otherwise).
ing widely shared features, this approach would help to Also, studies of phenomena in a particular animal group
identify additional species that possess features similar should be favored, not disfavored, if most previous
to those present in humans that resulted from conver- research has been concentrated in some other species.
gent evolution: adult pair bonding in voles comes to In addition, just as we accept that it is usually better
mind (Young, Wang, & Insel, 1998). Comparative for students to work with several mentors and at
studies can also highlight alternative means to similar different institutions during their training, we should
preuss and robert: animal models of the human brain: repairing the paradigm 65
Donaldson, Adolf Meyer, and “the” albino rat. Hist Psychol, Preuss, T. M. (2007). Evolutionary specializations of primate
2(1), 3. brain systems. In M. J. Ravosa & M. Dagasto (Eds.), Primate
Logan, C. A. (2001). “[A]re Norway Rats … Things?”: origins: Adaptations and evolution (pp. 625–675). New York,
Diversity versus generality in the use of albino rats in experi- NY: Springer.
ments on development and sexuality. J Hist Biol, 34(2), Preuss, T. M. (2009). The cognitive neuroscience of human
287–314. uniqueness. In M. S. Gazzaniga (Ed.), The cognitive neurosci-
Logan, C. A. (2002). Before there were standards: The role ences, 4th ed. (pp. 49–64). Cambridge, MA: MIT Press.
of test animals in the production of empirical generality in Preuss, T. M. (2010). Reinventing primate neuroscience for
physiology. J Hist Biol, 35(2), 329–363. the twenty-first century. In M. L. Platt & A. A. Ghazanfar
Logan, C. A. (2005a). The legacy of Adolf Meyer’s compara- (Eds.), Primate neuroethology (Vol. 1, pp. 422–454). Oxford,
tive approach: Worcester rats and the strange birth of the UK: Oxford University Press.
animal model. Integr Physiol Behav Sci, 40(4), 169–181. Preuss, T. M. (2011). The human brain: Rewired and running
Logan, C. A. (2005b). Making mice: Standardizing animals hot. Ann NY Acad Sci, 1225(Suppl. 1), E182–E191.
for American biomedical research, 1900–1955 (book Preuss, T. M., Qi, H., & Kaas, J. H. (1999). Distinctive com-
review). J Hist Behav Sci, 41(3), 293–294. partmental organization of human primary visual cortex.
Manger, P. R., Cort, J., Ebrahim, N., Goodman, A., Henning, Proc Natl Acad Sci USA, 96(20), 11601–11606.
J., Karolia, M., … Strkalj, G. (2008). Is 21st century Rader, K. A. (2004). Making mice: Standardizing animals for
neuroscience too focussed on the rat/mouse model of American biomedical research, 1900–1955. Princeton, NJ:
brain function and dysfunction? Front Neuroanat, 2, 5. Princeton University Press.
McNamara, R. K., Namgung, U., & Routtenberg, A. (1996). Rekart, J. L., Sandoval, C. J., & Routtenberg, A. (2007).
Distinctions between hippocampus of mouse and rat: Learning-induced axonal remodeling: Evolutionary diver-
Protein F1/GAP-43 gene expression, promoter activity, and gence and conservation of two components of the mossy
spatial memory. Brain Res Mol Brain Res, 40(2), 177–187. fiber system within Rodentia. Neurobiol Learn Mem, 87(2),
Merigan, W. H. (1980). Temporal modulation sensitivity of 225–235.
macaque monkeys. Vis Res, 20(11), 953–959. Rilling, J. K. (2008). Neuroscientific approaches and appli-
Murphy, W. J., Eizirik, E., O’Brien, S. J., Madsen, O., Scally, cations within anthropology. Am J Phys Anthropol, Suppl. 47,
M., Douady, C. J., … Springer, M. S. (2001). Resolution 2–32.
of the early placental mammal radiation using Bayesian Robert, J. S. (2004a). Embryology, epigenesis and evolution:
phylogenetics. Science, 294(5550), 2348–2351. Taking development seriously. Cambridge, UK: Cambridge
Passingham, R. E., & Wise, S. P. (2012). The neurobiology of the University Press.
prefrontal cortex: Anatomy, evolution, and the origin of insight. Robert, J. S. (2004b). Model systems in stem cell biology.
Oxford, UK: Oxford University Press. BioEssays, 26(9), 1005–1012.
Peeters, R., Simone, L., Nelissen, K., Fabbri-Destro, M., Robert, J. S. (2008a). The comparative biology of human
Vanduffel, W., Rizzolatti, G., & Orban, G. A. (2009). nature. Phil Psych, 21(3), 425–436.
The representation of tool use in humans and monkeys: Robert, J. S. (2008b). Taking old ideas seriously: Evolution,
Common and uniquely human features. J Neurosci, 29(37), development, and human behavior. New Ideas Psychol, 26(3),
11523–11539. 387–404.
Preuss, T. M. (1995a). The argument from animals to Rockel, A. J., Hiorns, R. W., & Powell, T. P. (1980). The
humans in cognitive neuroscience. In M. S. Gazzaniga basic uniformity in structure of the neocortex. Brain,
(Ed.), The cognitive neurosciences (pp. 1227–1241). Cam- 103(2), 221–244.
bridge, MA: MIT Press. Sherwood, C. C., Subiaul, F., & Zawidzki, T. W. (2008). A
Preuss, T. M. (1995b). Do rats have prefrontal cortex? The natural history of the human mind: Tracing evolutionary
Rose-Woolsey-Akert program reconsidered. J Cogn Neurosci, changes in brain and cognition. J Anat, 212(4), 426–454.
7(1), 1–24. Striedter, G. F. (2007). A history of ideas in evolutionary
Preuss, T. M. (2000). Taking the measure of diversity: Com- neuroscience. In J. H. Kaas (Ed.), Evolution of nervous systems:
parative alternatives to the model-animal paradigm in corti- A comprehensive reference (Vol. 1, pp. 1–5). Amsterdam: Aca-
cal neuroscience. Brain Behav Evol, 55(6), 287–299. demic Press.
Preuss, T. M. (2001). The discovery of cerebral diversity: An Uylings, H. B. M., Groenewegen, H. J., & Kolb, B. (2003).
unwelcome scientific revolution. In D. Falk & K. R Gibson Do rats have a prefrontal cortex? Behav Brain Res, 146(1–2),
(Eds.), Evolutionary anatomy of the primate cerebral cortex (pp. 3–17.
138–164). Cambridge, UK: Cambridge University Press. van der Staay, F. J., Arndt, S. S., & Nordquist, R. E. (2009).
Preuss, T. M. (2004). Specializations of the human visual Evaluation of animal models of neurobehavioral disorders.
system: The monkey model meets human reality. In J. H. Behav Brain Funct, 5(1), 11.
Kaas & C. E. Collins (Eds.), The primate visual system Young, L. J., Wang, Z., & Insel, T. R. (1998). Neuroendocrine
(pp. 231–259). Boca Raton, FL: CRC Press. bases of monogamy. Trends Neurosci, 21(2), 71–75.
ABSTRACT The immense complexity of the human brain is Cellular mechanisms of human brain development
reflected in its cellular organization and the vast behavioral
and cognitive repertoire that it can generate. The human Human brain development involves many cellular and
brain develops through a dynamic and prolonged process
during which a myriad of cell types are generated and assem-
molecular processes that unfold over the course of
bled into intricate synaptic circuitry. Deviations from this almost two decades (Kang et al., 2011; Kostović & Judas,
normal course of development can lead to a variety of pathol- 2002; Sidman & Rakic, 1973; see figure 8.1). One of the
ogies, including disorders, such as autism and schizophrenia, most remarkable aspects of human development is that,
that affect some of the most distinctly human aspects of cogni- by the time of birth, the general architecture of the
tion and behavior. While humans share many features with
brain has been assembled and the majority of neurons
other mammals, in particular with other primates, organiza-
tional and developmental differences have allowed for the have migrated to their final positions. The organization
elaboration of human-specific cognition and behavior. Analyz- of human neurodevelopment can be divided into three
ing molecular and cellular processes involved in human brain main sequences of events: generation of neuronal and
development, along with parallel studies in nonhuman pri- glial cells types, migration of newly born cells to their
mates, is necessary for defining both ancestral and uniquely
final destination, and their differentiation into mature
human features, but this is often difficult to do in a systematic
and comprehensive manner. In this review, we summarize and properly functioning cells within neural circuits.
current knowledge about molecular and cellular processes
underlying human brain development and evolution. Particu- Genesis of Neuronal and Glial Cells The ven-
lar emphasis is given to studies of the cerebral cortex because tricular and subventricular zones (VZ and SVZ, respec-
of its importance in higher cognition and because it has been tively) comprise the germinal zones of the developing
the focus of many comparative and developmental studies.
telencephalon and give rise to neurons and macroglia
The human brain is composed of over eighty billion (astrocytes and oligodendrocytes) of the cerebral cortex
neurons and at least an equal number of glial cells (figure 8.2; Caviness, Takahashi, & Nowakowski, 1995;
(Azevedo et al., 2009). Neurons are connected with Fishell & Kriegstein, 2003; Sidman & Rakic, 1973). VZ
approximately 150,000 to 180,000 km of myelinated is the first germinal zone to form and is composed of
axons, and within the neocortex alone, there are about elongated polarized neuroepithelial cells that undergo
0.15 quadrillion synaptic contacts (Pakkenberg et al., interkinetic nuclear migration. Early neuroepithelial
2003). These basic facts illustrate the organizational progenitor cells each produce two daughter cells that
complexity of the human brain and highlight some dif- re-enter the cell cycle. This symmetrical division doubles
ficulties we face when trying to understand the molecu- the number of progenitor cells each time and exponen-
lar and cellular mechanisms of its development and tially expands the pool of progenitor cells (Caviness et
evolution. In this chapter, we will first review the al., 1995; Fishell & Kriegstein, 2003; Rakic, 1995). Early
sequences of cellular events in the developing human in neurogenesis, neuroepithelial progenitor cells trans-
brain, with a focus on the cerebral neocortex, and then form into radial glial cells that elongate along apico-
highlight advances in understanding the molecular pro- basal axis and begin dividing asymmetrically to generate
cesses associated with its development and evolution. a new progenitor and a postmitotic neuron or glial cell.
The generation of glial cells follows neurogenesis and
peaks around birth in humans (Sidman & Rakic, 1973).
There are important species differences in the orga-
*These authors contributed equally to this work. nization of neural progenitor cells and the generation
5 cm
B Conception Birth
Embryonic Fetal development Infancy Childhood Adolescence Adulthood
Period 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
C Neocortex
Neurogenesis
SP, L5/6 proj. neurons
L2-4 projection neurons
Neuronal migration
Axonogenesis
Dendritogenesis
Synaptogenesis
Myelination
Figure 8.1 Timeline of major cellular events and gross mor- in post-conception days (PCD). (C) Summaries of the occur-
phological changes in human brain development. (A) Sche- rence and progression of major cellular events in the human
matic images of developing brains from Gustaf Retzius’s 1896 neocortex. Black indicates the developmental age when the
atlas and an adult brain generated by magnetic resonance defined event reaches its peak or is indistinguishable from
imaging. (B) Periods of development and adulthood as previ- adult periods.
ously defined (Kang et al., 2011). Age is represented
of neural cell types. One of the most prominent is the & Anton, 2012; Leone, Srinivasan, Chen, Alcamo, &
prolonged period of neuronal and glial production in McConnell, 2008; Molyneaux, Arlotta, Menezes, &
humans compared to other primates and mammals Macklis, 2007). In contrast, the inhibitory neurons or
(see figure 8.2), which has been postulated to play a interneurons are GABAergic, form local circuit connec-
critical role in regulating the size of the brain and matu- tions, and account for 15–25% of all cortical neurons
ration of neural circuits (Caviness et al., 1995; Rakic, (DeFelipe et al., 2013). A number of studies have shown
1995). Increases in neocortical size, particularly in that cortical interneurons share a common origin with
primates and humans, have been linked to the expan- striatal neurons, arising from progenitors within the
sion of progenitor cells in the outer subventricular zone ganglionic eminences of the ventral forebrain and
(oSVZ) during development (Fietz et al., 2010; Hansen, migrating tangentially into the cortex (Marín & Ruben-
Lui, Parker, & Kriegstein, 2010; Smart, Dehay, Giroud, stein, 2003). Intriguingly, studies in humans and non-
Berland, & Kennedy, 2002). human primates (NHPs) have reported that certain
There are also potential differences in the origin and cortical interneurons arise from dorsal, instead of
migrational routes of cortical neurons, particularly ventral, pallial progenitors (Jakovcevski, Mayer, &
between primates and rodents. The neurons of the cere- Zecevic, 2011; Letinic, Zoncu, & Rakic, 2002; Petanjek,
bral cortex can be roughly classified into two distinct Kostović, & Esclapez, 2009), suggesting that the origin
groups: excitatory and inhibitory. The excitatory and migration of primate cortical interneurons are evo-
neurons utilize the excitatory neurotransmitter gluta- lutionarily divergent. However, the extent of these
mate. The great majority of them have characteristic species differences is unclear. A study of human holo-
pyramidal-shaped cell bodies and a long apical dendrite prosencephaly brains, which exhibit severe ventral
covered with spines, and project long axons to other forebrain hypoplasia and lack a subgroup of ventral
regions of the central nervous system (Kwan, Sestan, progenitors that generate distinct types of interneurons
in rodents, revealed that these patients also lack the synaptic contacts with ingrowing cortical afferents.
same subtypes of cortical interneurons (Fertuzinhos et These neurons also play a key role in establishing corti-
al., 2009). Further support for the importance of the cal lamination and patterning of cortical connections
ventral forebrain in generating human cortical inter- (Allendoerfer & Shatz, 1994; Kostović & Judas, 2002;
neurons comes from a recent study (Hansen et al., 2010) Rice & Curran, 2001).
demonstrating that cultured human neocortical pro- Neurons use different modes of migration to reach
genitors do not generate interneurons. their final destination. For example, newly born pyra-
midal neurons either undergo somal translocation
Neuronal Migration Upon leaving the cell cycle, (Morest & Silver, 2003) or migrate while attached to
neurons migrate toward their final destination (figure radial glial fibers (Rakic, 1971) to their final position in
8.2). The first generated neurons settle immediately the cortical plate, following a precise inside-first,
above the VZ, forming the early marginal zone MZ, outside-last order. Hence, radial glia serves dual roles
which is also referred to as the primordial plexiform as a neural progenitor and a transient scaffold for neu-
layer or the preplate (Marin-Padilla, 1978). The subse- ronal migration. Cortical interneurons, on the other
quent generation of neurons destined for the cortical hand, migrate tangentially from the ventral forebrain
plate (future layers 2–6) are thought to split the pre- or within the growing dorsal cerebral wall (Marín &
plate into a superficial marginal zone (future layer 1) Rubenstein, 2003).
and a deep subplate zone. Neurons in the marginal and In all mammals, including humans, at least two types
subplate zones are the first to be generated and achieve of spatial information must be imprinted onto neurons
functional maturity, as well as the first to establish of the developing cerebral cortex: (1) their position in
10 SPORNS 91
14 RECANZONE 143
krubitzer: introduction 77
heritability, and therefore only the genetic component which it is composed. For example, a social system has
of the phenotype. What I grew to appreciate was that properties that cannot be easily explained by the behav-
the important question was how complex phenotypes ior of a single individual, or brain, or neural firing
emerge. This deals with both genetic, heritable traits as pattern.
well as traits that are context-dependent. Every given On the other end of the spectrum, brains can be
species is a combination of both, and any individual decomposed into cortical networks, made up of nodes
within a species could have a multitude (or at least a or cortical areas, which are composed of local neural
broad distribution) of phenotypes of brain organization circuits. These circuits are formed by neurons, which
and behavior that are expressed in varying environmen- are held together by the cytoskeleton and plasma mem-
tal contexts. Thus, evolution and plasticity are also branes and connected to each other by synapses. These
intertwined. in turn include smaller elements such as synaptic recep-
The focus of this section, “Plasticity and Learning,” is tors, which are composed of molecules encoded by
the neocortex, the portion of the brain associated with genes, and the expression and transcription of genes
volitional motor control, perception, and higher cogni- can be controlled by environmental context. While
tive processing. Collectively, this section spans many genes are the heritable part of this complex hierarchy,
levels of organization. When considering how the neo- it is the larger, often context-dependent components of
cortex works and how it changes within the lifetime of organization, such as behavior, that are the targets of
an individual and across species over time, it is impor- natural selection. To further complicate matters, the
tant to recognize that the neocortex is only one com- relationship between the targets of selection and genes
ponent of the entire nervous system. It should be is often correlative and generally indirect.
obvious, but it is valuable to remember that the nervous Keeping these different levels of organization in
system is embedded in a body that has sensory receptor mind, the authors of this section were charged with
arrays and a particular morphology that is often highly addressing several important questions. What is chang-
derived. For example, a bat interacts with the world in ing in the neocortex, and does it directly or indirectly
a very different way than a monkey. Each have different co-vary with the target of selection, which in this case is
forelimb morphologies, a wing versus a hand, and each the ability to learn and respond appropriately in a
have enhanced sensory receptor arrays that allow for complex and dynamic multisensory context? At what
extraordinary abilities associated with the auditory level should we be looking for these alterations in the
(sonar) versus the visual (form vision) system, respec- neocortex? Will there be changes in the microcircuitry
tively. The cortical areas that process inputs from these (Rodney J. Douglas and Kevan A. C. Martin) and/or
specialized effector arrays have expanded and are speci- the macrociruitry (Olaf Sporns)? How does enhanced
fied by their unique inputs, and additional fields have or degraded input from our sensory receptor arrays
been added to the portions of the neocortex that ulti- affect the functional organization of the brain and our
mately receive inputs from these sensory systems. perceptions of a sensory experience ( Jon H. Kaas and
Further, cortical networks necessary for sensory motor Charnese Bowes)? Do particular types of neocortical
integration for flight versus manual dexterity, and for organizations promote plasticity (Man Chen and
interfacing the specialized effectors (ears/pinna versus Michael W. Deem)? What is the impact of groups of
eyes) with the object to be explored are also uniquely brains (social systems) on the development of brain
wired within the brain. Finally, each body faces a differ- structure and function (Courtney Stevens and Helen
ent and changing physical environment, areal versus Neville), and how does the brain change throughout
terrestrial-arboreal, that constrains movement and the entire life of an individual (Gregg H. Recanzone)?
perception. What are the underlying mechanisms that generate
This body/brain configuration of a given individual these phenotypic alterations at all levels of organization
within a species interacts with other bodies and brains, (Tania Roth), from cellular to systems levels, and how
chasing, courting, mating, attacking, fleeing, communi- do these changes result in variant behaviors? The objec-
cating, and forming social systems. These individuals tive of addressing these questions is to begin to under-
interact with other species and other social systems in stand the boundary conditions of each level of
a variable physical environment composed of living and organization; how higher levels of organization emerge
nonliving elements. This group of organisms and their from and interrelate with lower levels; how the brain
environment generates a complex and highly dynamic dynamically generates highly adaptive, context-depen-
collective biomass that has emergent properties that dent behaviors throughout a lifetime; and how human
differ from, and often exceed, the individual parts of social and cultural evolution impact brain evolution.
ABSTRACT Every percept, every concept, every memory, every version of this same idea in their “canonical circuit” for
decision, and every action we make arise from the activity of the neocortex, based on intracellular recordings and
neurons in our brains. Of all brain structures, the neocortex, computer simulations of cat area 17. While there is
which forms over 80% of the volume of the human brain, is
arguably the most critical structure that makes us what we are broad agreement that neocortex is built of common
and allows us to create the societies in which we live. This is components linking in a stereotyped combination, it is
a paradox, because the local circuits of the neocortex in all worth emphasizing from the start that, however beguil-
mammals, from mouse to man, are thought to be very similar ing this concept, it is still a working hypothesis 100 years
and determined by the laminar distribution of relatively few after the Vogts proposed it.
types of excitatory and inhibitory neurons organized accord-
ing to common principles of connectivity. Here we trace how
all these different elements contribute to the dense thickets Composition
of cortical circuits, and how evolution has organized these
circuits across the two-dimensional surface of the cortical A quick look at the composition of the neocortex is
sheet to produce a map of behavior. instructive. In each cubic millimeter of cortical gray
matter we find up to 100,000 neurons, about 100 million
synapses, and 4 km of axon (Braitenberg & Schüz,
The most astonishing hypothesis about the human neo- 1991). By contrast the white matter, which is generally
cortex is not that it is the seat of consciousness, but that thought to contain most of the wiring of the brain,
it achieves its enormous range of competence through contains only 9 m of axon per cubic millimeter. These
local circuits that may bear a close similarity to those bald numbers suggest that the processing strategy of the
found in the brain of a mouse. Of course, the human neocortex is to exploit direct broadband transmission
neocortex forms a far larger percentage of the brain by for local circuits in the gray matter and to use the white
volume than that of the mouse (35% vs. 80%), but the matter only for long-distance, low-bandwidth transmis-
basic local circuits, areal organization, and interconnec- sion. The implication of this is that most of the synapses
tions with subcortical structures are qualitatively similar made in any area originate from neurons within that
in man and mouse. The hypothesis that the neocortex area, not from outside sources, like the thalamus or
of all mammals is built of a common circuit is long- other cortical areas. Thus we should first look to the
standing: in the early twentieth century, Cecile and local circuits if we are to understand how the neocortex
Oscar Vogt (1919) referred to neocortex as “isocortex,” performs its multifarious tasks of perception, memory,
because of its relatively uniform six-layered appearance. cognition, and action. Of course, the long-distance
Pioneers of cortical functional architecture, like Mount- external inputs to these local circuits are not incidental
castle (1997) and Hubel and Wiesel (1974), were con- players, since some of them connect cortex to the
vinced that their favorite areas were built of repeated peripheral structures, like the spinal cord, while others
units, “like a crystal” (Hubel & Wiesel, 1974). In 1980, connect cortex to itself and to other essential machin-
Rockel, Hiorns, and Powell published their controver- ery like the basal ganglia, claustrum, brain stem, and
sial classic entitled “The basic uniformity of structure of cerebellar nuclei. Unfortunately, the multiple roles of
the neocortex,” in which they claimed that all cortical these external pathways are rather poorly documented.
areas, with the exception of primate area 17, had the We have nothing like a comprehensive description of
same number of neurons lying under each unit of their synaptic connectivity and physiology, despite such
surface area (see Carlo & Stevens, 2013, for a confirma- iconic representations as the summary circuit diagram
tion of Rockel et al.’s findings and a rebuttal of the from Felleman and Van Essen’s (1991) meta-study of
counterclaims). By the end of the same decade, Douglas, the hierarchy of connections among visual cortical
Martin, and Whitteridge (1989) expressed a functional areas in the macaque monkey.
DB
LBC
LBC
P
CH
4 NG
5
M
6
Figure 9.1 Pyramidal cells and their principal inputs. Most target different portions of the dendritic tree—only the best
excitatory (glutamatergic) synapses are formed on spines and studied are shown here. P, pyramidal cell; CH, chandelier cell;
arise from other pyramidal cells within the same area. Inhibi- DB, double bouquet cell; NG, neurogliaform cell; LBC, large
tory (GABAergic) synapses are provided by neurons that basket cell; CR, Cajal-Retzius cell; M, Martinotti cell.
All textbooks will tell you that about 80% of the monkeys. Interestingly, they are not found in some
neurons that occupy the six layers of the neocortex are primary sensory areas, like the auditory cortex of the
pyramidal cells. These neurons are spiny, have a promi- mouse. Spiny stellate cells are usually lumped together
nent apical dendrite, use glutamate as their neurotrans- with the pyramidal cells, since both are spiny and excit-
mitter, and are excitatory in function (figure 9.1). The atory. Spiny stellate neurons are not found throughout
remaining 20% are smooth neurons, often misleadingly the neocortex, but only in the middle layers of certain
called “interneurons” without further qualification. sensory cortical areas like rodent barrel cortex and the
These neurons do not have densely spiny dendrites primary visual cortex of cats and monkeys (Lund, Lund,
(hence “smooth”), use gamma amino-butyric acid Hendrickson, Bunt & Fuchs, 1975).
(GABA) as their neurotransmitter, and are inhibitory The 4 : 1 ratio of the two basic morphological and
in function. The confusion about cortical “interneu- physiological types of neurons is standard in most
rons” arises because they were traditionally defined as “biologically realistic” models of neocortical functions,
cells whose entire axon arborizes in the region around but in two-dimensional (2D) models of cortex their
its parent cell body; that is, in contrast to “projection relative spatial distributions are usually tweaked to
neurons,” their axon does not project out of their own ensure that monosynaptic inhibitory connections
cortical area. While this is largely true of the smooth spread further than the excitatory connections. This
neurons, is it also true of some pyramidal cells and of center-surround pattern (also called a DOG [Differ-
most members of the second type of excitatory neuron, ence of Gaussians] or “Mexican hat”) is widespread,
the “spiny stellate,” which is spiny, lacks an apical den- as, for example, in the structure of the receptive fields
drite, and has a rather symmetric dendritic arbor, which of retinal ganglion cells and thalamic relay cells. In
gives it its name. Spiny stellate neurons are not found models of cortex the pattern of center excitation–
throughout the neocortex, but only in the middle layers surround inhibition is also applied to ensure stability
of certain sensory cortical areas like rodent barrel of the network. However, in reality, the smooth
cortex and the primary visual cortex of cats and neurons typically have much more compact axon
Nonetheless, the unifying hypothesis that the neocor- evolve, develop, and function. Importantly, it is on this
tex is essentially a computational sheet, which assem- cortical sheet that the sensory, perceptual, emotive, cog-
bles itself using common components and rules of nitive, and motor aspects of behavior are systematically
connection, is not only extremely attractive, it offers mapped and form their connections with the rest of the
new insights into the way in which the neocortex might brain and spinal cord.
ABSTRACT The architecture of brain networks has become a In formal terms, networks are collections of nodes
strong focus of recent empirical and theoretical studies, build- (network elements) and edges (interconnections). Net-
ing on the availability of comprehensive data sets on anatomi-
works can be represented as connection matrices where
cal brain connectivity in multiple species, including humans,
as well as on the development of novel analytic techniques each entry in the matrix records the presence or
coming from the emerging field of network science. Numer- absence, weight, sign, and direction of connections
ous studies have documented a strong tendency of brain net- between pairs of elements. The full set of network
works to minimize network cost, expressed as the physical nodes and edges can be described as a graph, and the
length or volume of connections or the energy budget devoted set of pairwise relations constitutes the graph’s topol-
to neural signaling. Network studies also show that cost mini-
mization cannot fully explain all empirically observed features ogy. Importantly, graph topology defines proximity or
of network architecture. The prevalence of high-cost aspects distance among elements solely in terms of the configu-
of network organization, including long-distance projections ration of the link structure, without explicitly taking
and highly connected brain regions, suggest that these aspects into account the spatial or geometric embedding of the
confer advantages in terms of efficiency and integrative graph within the real-world system from which it is
network function that outweigh their cost. This chapter
explores aspects of network cost and network efficiency and
derived. Thus, network topology captures the configu-
suggests that the topology of brain networks represents a ration of network nodes and edges in the absence of
trade-off between these opposing driving forces. any spatial context, while spatial embedding explicitly
considers the spatial relations among network elements.
Nervous systems are complex networks of structurally This distinction of network topology and spatial embed-
connected and dynamically interacting neural ele- ding is important for the main subject of this chapter,
ments. Informally used, the term “network” has a long which focuses on the relationship between these two
history in neuroscience. In recent years, an increasing ways of capturing brain network architecture. Another
focus on recording comprehensive data on brain important distinction is that between structural net-
connectivity has introduced a new formal and quantita- works and functional networks. Structural networks
tive network perspective to cognitive and systems are constructed from data on connectional anatomy,
neuroscience. The important innovation lies in the derived from microscopy, histology, or neuroimaging.
application of computational and analytic tools and Functional networks are based on time series data from
methods for mapping and interpreting patterns of neural recordings, capturing various aspects of correla-
connections and interactions among neurons and brain tion or dynamic coupling. Unless otherwise noted, the
regions (Bullmore & Sporns, 2009; Sporns, 2014). current chapter focuses on properties of structural
Empirical studies of brain networks at all scales, from brain networks and the role of these properties in brain
synaptic circuits in the mammalian retina (Briggman, function.
Helmstaedter, & Denk, 2011) to large-scale connectivity A deeper understanding of the relationship between
in the monkey (Markov et al., 2011) and human cere- network structure and network function is one of the
bral cortex (Hagmann et al., 2008), continue to reveal central challenges for cognitive and systems neurosci-
network patterns in ever-greater detail. These patterns ence (Bassett & Gazzaniga, 2011). This chapter attempts
are characterized by distinct nonrandom structural fea- to approach this challenge by exploring key factors that
tures (Sporns, 2011a), many of which are shared across can account for and may have shaped structural brain
brain networks in different species. Since brain net- networks, with an emphasis on “highly evolved” or
works underpin dynamic interactions among neural complex brains (including humans). To motivate the
populations, their structural organization is thought to approach taken in this chapter, let us consider the space
be essential for efficient and adaptive brain function of all possible brain network architectures. This space
(Sporns, 2011b). is truly astronomical, as it spans all of the ways in which
organized as a perfect lattice or random network. A Network Modules and Hubs Many real-world net-
statistical overabundance of some motifs is likely driven works can be partitioned into a set of modules or
by the presence of high clustering, which tends to network communities wherein each module corre-
promote fully connected and recurrent motif configu- sponds to a set of nodes that are densely interconnected
rations. The strong tendency of cortical networks to among one another, while connections between
exhibit local clustering is partly due to wiring cost mini- modules are relatively sparse (figure 10.1). This defini-
mization. An abundance of short-distance connections tion can be formalized in the form of a modularity
naturally leads to high local clustering as seen in a metric that expresses the degree to which a given
regular lattice. Counteracting this tendency are network network exhibits modular organization, given a module
features that break regularity and drive brain organiza- partition (Newman & Girvan, 2004). An optimal parti-
tion in the direction of “randomness,” in the process tion can then be identified using an optimization algo-
incurring higher wiring cost. rithm (Blondel, Guillaume, Lambiotte, & Lefebvre,
The features of network organization discussed so far, 2008). The optimal partition assigns each network node
while present in many examples of structural brain net- to exactly one module, thus decomposing the network
works, are also broadly, perhaps universally, shared into nonoverlapping network communities. Modularity
across many other social, technological, or biological detection is an active research area in its own right (for
networks (Boccaletti et al., 2006; Watts & Strogatz, a review, see Fortunato, 2010), and recent methodologi-
1998). Hence, their presence in brain networks is cal developments include new search algorithms, refine-
not unique, which begs the question whether there ments of modularity measures (Fortunato & Barthelemy,
are additional aspects of network organization that 2007), and reformulations of modularity in terms of
are compatible with these universal features but link communities (Ahn, Bagrow, & Lehmann, 2010).
more directly indicative of specific aspects of neuronal Modularity is ubiquitous in brain networks. Modules
information processing. In this context, topological fea- of densely interconnected neurons have been identi-
tures pointing to the existence of network communities fied in C. elegans (Sohn, Choi, Ahn, Lee, & Jeong,
or modules that are interconnected by important hub 2011), and modules of brain regions have been found
nodes are of special importance. in whole-brain analyses across multiple mammalian
ABSTRACT After damage to peripheral nerves and other parts to another in rats. Such observations suggested that the
of sensory and motor systems, adult humans often experience mature nervous system has a functional organization
sensory or motor impairments that diminish over weeks to
that is highly fixed, as functional adaptations to altered
months after the injury. Studies on monkeys, rats, and other
animals have helped us understand how these recoveries sensory inputs did not seem to occur. Second, the early
occur and suggest ways to promote them. While valuable experiments that demonstrated alterations in the orga-
information has come from a wide range of studies, those on nization and functions of the visual system after sensory
nonhuman primates are especially informative because their deprivation in cats and monkeys found that the suscep-
nervous systems more closely resemble our own. Studies have tibility to such changes was largely restricted to a short
focused on recoveries after various types of restricted damage
to sensory or motor systems that produce sensory or motor period early in postnatal development, the so-called
deficits by deactivating parts of sensory or motor systems. sensitive or critical period, and the same treatments
These studies indicate that the growth of new connections in have little or no effect on the mature visual system
the central nervous system as a consequence of deactivations (Daw, 1995). Third, it seems logical to conclude that a
can be extensive and lead to reactivations that promote behav- fully functional nervous system should no longer be
ioral recovery. Importantly, several types of treatment appear
to enhance adaptive growth of new nervous system connec-
plastic at maturity, as almost any change would impair
tions and behavioral recovery. The mature nervous system is function. Of course, early investigators recognized that
more plastic than previously thought. at least systems for learning remain plastic throughout
life, perhaps at a reduced level for language in adults,
“In the general’s entourage, the discomfort José Maria and that patients often recover some or all lost func-
Carreño experienced in the stump of his arm was reason for
tions after some brain damage. However, evidence that
cordial teasing. He felt the movements of his hand, the sense
of touch in his fingers, the pain bad weather caused in bones the mature nervous system remains quite plastic has
he did not have.” accumulated, and it is useful to consider this evidence
—Gabriel Garciá Márquez, The General in His Labyrinth and understand the nature of the nervous system
changes that occur. Thus, we want to know how such
The developing nervous system has long been known changes emerge, how they relate to the recovery of
to be quite plastic. This means that the developing abilities, and why they sometimes lead to errors in per-
nervous system has the potential to develop in different ception. We also need to evaluate treatments that have
ways depending on sensory experience, other environ- the potential to promote functional recoveries.
mental factors, and nervous system damage (e.g., Kaas, This review focuses on plasticity studies of sensory
Merzenich, & Killackey, 1983; Wiesel, 1982). As the and motor systems in monkeys and other primates.
developing nervous system is always changing, it seems While much has been learned about the plasticity of the
reasonable that the course of development can be human brain from clinical patients, studies in animals
altered, sometimes in ways that compensate for nervous allow the repetition and control of variables and provide
system damage or allow an adaptation to an abnormal evidence that is more easily interpreted. Because
environment. In contrast, early neuroscientists were monkeys and our other primate relatives have nervous
often skeptical about the possibility of mature nervous systems that more closely resemble our own than those
systems being plastic, and even recently the concept of of cats, rats, and mice, most of the experimental evi-
visual cortex in mature monkeys being plastic has been dence presented here comes from studies on monkeys.
seriously challenged. However, evidence from other mammals is important
There are several reasons for this skepticism. First, and relevant, and some of it is included here. In addi-
early investigators such as Sperry (1959) noted that tion, the emphasis in this review is on the plasticity of
behavioral compensations did not occur after eye rota- the somatosensory system, since the organization of this
tion in frogs or after a nerve was crossed from one leg system in monkeys is well understood and the system
kaas and bowes: plasticity of sensory and motor systems after injury 103
offers several technical advantages for studies of plastic- Anterior parietal
Post parietal
ity, including the ability for the regeneration of sensory M1
Frontal CS 5 +7
1 2
nerves. We also consider the plasticity of visual and audi-
tory systems, as well as the motor system, for compari- Cortex
son. Other systems—for example, those devoted to
3b
cognitive functions—might be even more plastic than
4
sensory or motor systems. However, the structural and
S2
functional organizations of sensory and motor systems 3a
are much better understood, and thus changes in these VPS PV
systems as a result of experimental manipulation are VPL
more likely to be detected. Finally, the types of plasticity VPM
VP
that are due to experience and learning in sensory Face Hand Foot
systems (e.g., perceptual learning) and motor systems
are considered elsewhere (Buomomano & Merzenich, Thalamus
VPI
1998; Diamond, Armstrong-James, & Ebner, 1993; Li,
Piech, & Gilbert, 2004; Nudo, Milliken, Jenkins, &
Merzenich, 1996; Recanzone, Schreiner, & Merzenich,
1993). External
Cuneate N. Gracile N. RA
Muscle +
Cuneate N.
Plasticity of the somatosensory system after sensory spindle SA
T rigeminal N. inputs
receptor
loss or lesions inputs Cervical spinal cord
Medulla
Dorsal
The structural organization of the somatosensory system column Spinothalamic tract
of primates is well known (Kaas, 2011). Thus, altera- Dorsal
pathway
tions from the normal organization as a result of plastic- root
ganglion
ity can be detected (figure 11.1). Sensory afferents from
Receptors Spinal cord
the receptor sheet, the skin, largely preserve their
peripheral neighborhood relationships as they termi-
nate in the ipsilateral spinal cord and brainstem nuclei.
Left Right
Topographic (or somatotopic) patterns of connections
are further preserved in projections to the contralateral Figure 11.1 The somatosensory system of primates. Note
ventroposterior nucleus of the contralateral somatosen- that cutting the dorsal column pathway removes the direct
sory thalamus, and then to primary somatosensory somatosensory input from low-threshold, rapidly adapting
cortex, S1 (area 3b of Brodmann, 1909). Early studies (RA) and slowly adapting (SA) cutaneous peripheral nerve
afferents from the lower body, while cutting dorsal roots of
of the somatotopic organization of somatosensory peripheral nerves removes all sensory inputs. Cutting all
cortex in primates did not have the resolution to distin- dorsal column branches of RA and SA afferents from the
guish the four systematic representations of the contra- forearm deactivates neurons in the cuneate nucleus and the
lateral body in the four architectonic fields of Brodmann hand territories of the contralateral ventroposterior (VP)
areas 3a, 3b, 1, and 2. The unfortunate consequence nucleus of the thalamus and cortical areas 3b (primary
somatosensory cortex, S1) and area 1. Other areas with direct
has been that the term S1 is often used to refer to all
or indirect inputs from areas 3b and 1 are also deprived (e.g.,
four fields combined. However, only area 3b is homolo- area 2; the second somatosensory area, S2; the parietal ventral
gous to S1 of other mammals, and it is important to area, PV; posterior parietal cortex, areas 5 & 7; area 3a; and
distinguish area 3b from these other fields (Kaas, 1983). primary motor cortex, M1 or area 4.) VPM, the medial divi-
Area 1 forms a mirror image representation of touch sion of VP represents the face and mouth, while VPL, the
lateral division, represents the forelimb, trunk, and hind limb.
receptors just caudal to area 3b. Parallel somatotopic
representations in areas 3a and 2 are merged with pro-
prioceptive inputs from the ventroposterior superior Thus, we focus on changes in area 3b, although there
nucleus of the thalamus. Additional representations may be further changes that are mediated in these addi-
with inputs from areas 3b, 3a, 1, and 2 include the tional processing areas.
second somatosensory area, S2, and the parietal ventral Many studies of plasticity of the somatosensory system
area, PV. As information about touch is distributed have demonstrated changes in the somatotopy of
from area 3b to all of these areas, any plastic change in cortex or other parts of the system after a loss of some
area 3b would be relayed to these higher-order areas. of the sensory inputs from the skin. Alterations or
kaas and bowes: plasticity of sensory and motor systems after injury 105
contralateral ventroposterior nucleus (Garraghty & Lesions of Peripheral Nerve Afferents as They
Kaas, 1991b). Thus, with a lack of competition from the Travel in the Dorsal Columns of the Spinal
afferents of the median nerve, the sparser inputs from Cord A third way of producing and studying sensory
the radial nerve of the back of the hand likely sprout loss has been to cut the ascending branches of periph-
over the very short distances in the cuneate nucleus to eral nerve afferents as they course in the dorsal columns
activate deprived neurons (Florence & Kaas, 2000). of the spinal cord to the dorsal column nuclei. The
Alternatively, subthreshold connections to somatotopi- major advantage of this approach is that the sensory loss
cally mismatched locations in the cuneate nucleus may is very specific, and very limited. The large axons con-
already exist, and they become stronger by providing ducting touch information from the skin of the hand
more synapses or because of other homeostatic mecha- and arm branch as they enter the spinal cord—with one
nisms (Garraghty, LaChica, & Kaas, 1991; Turrigiano, branch entering the dorsal column of axons that project
1999). As the time of regeneration of peripheral nerves to the cuneate nucleus representing the forearm, and
depends on distance, reactivations in the central the other branch terminating on dorsal horn neurons
nervous system that depend on longer axon growth of the spinal cord that preserve this touch information
should take longer times. Of course, some of the corti- for use in local reflexes and motor control—and send
cal reactivation that follows a nerve cut may depend on this information to other locations, including the
axon growth at thalamic and cortical levels, perhaps cuneate nucleus. All other afferents, including those
occurring over longer times. that mediate crude touch, pain, and temperature, ter-
minate normally in the dorsal horn, as they do not
branch into the dorsal columns (Kaas, 2011). Thus,
Reactivations After Damage to Dorsal Roots of monkeys with dorsal column lesions appear quite
Peripheral Nerves as They Enter the Spinal Cord normal in most behavior, including running and climb-
Another type of sensory loss occurs when the sensory ing, but do have impaired hand use for a period of 2
inputs to the spinal cord are reduced or eliminated by to 4 weeks (Qi, Gharbawie, Wynne, & Kaas, 2013). They
cutting the dorsal sensory roots of peripheral nerves as may drop grasped objects, and even look in their hand
they enter the spinal cord, leaving the ventral motor to see if the object is present. However, they rapidly
root outputs to muscles intact. Years after this approach recover, and normal hand use returns. Immediately
was used to evaluate arm use after a complete sensory after such dorsal column lesions, the hand region of
loss from one arm, it became possible to study cortical contralateral area 3b is largely or completely unrespon-
responsiveness in some of these monkeys (Pons et al., sive to touch on the hand (figure 11.2), and this level
1991). The surprising result was that the complete arm of unresponsiveness persists for 1 to 2 weeks, after
representation in area 3b was activated by touch on the which the cortex is progressively reactivated by any of
face, as well as the arm representations in areas 3a, 1, inputs in the dorsal columns from the hand that were
and 2. These results showed that major reactivation of preserved (Chen, Qi, & Kaas, 2012; Jain, Catania, &
deprived cortex is possible, but there was no informa- Kaas, 1997; Qi, Chen, & Kaas, 2011). Spinal cord
tion about the time course of the massive reactivation neurons that receive the branched dorsal horn inputs
or mechanisms of this reactivation. from peripheral nerve axons and project to the cuneate
More recently, this approach has been used to only nucleus (Rustioni, 1976; Witham & Baker, 2011) also
partially denervate the arm and hand by cutting only a likely contribute to the reactivation. Finally, with lesions
few dorsal roots, thus removing most, but not all, of the that are extensive, and when all of the hand cortex is
inputs from some of the digits. As a result, only a few not reactivated by the hand, then parts of the hand
preserved inputs from the digits, ones that were too few representations may become activated by touch on the
to activate cortex immediately after surgery, became face (Jain et al., 1997; Jain, Qi, Collins, & Kaas, 2008).
effective in activating most of the deprived cortical ter- The activation by inputs from the face appears to
ritories for those digits, with a recovery of hand use over depend on the growth of a few axons from the face past
a period of weeks (Darian-Smith & Ciferri, 2005, 2006). their normal targets in brainstem trigeminal nuclei to
These experiments demonstrated that even sparse reach the cuneate nucleus for the forearm (Jain, Flor-
remaining inputs can reactivate their former cortical ence, Qi, & Kaas, 2000). Much of the hand subnucleus
territories over short periods of time. Furthermore, the of the contralateral ventroposterior nucleus also
cortical reactivations and behavioral recoveries were becomes responsive to the face (Jain et al., 2008). The
associated with sprouting of preserved axons in the reactivation of hand cortex by inputs from the face
cuneate nucleus and spinal cord (Darian-Smith & takes 6 to 8 months to emerge, a longer time perhaps
Ciferri, 2006), as well as in cortex. related to the longer distance the face axons need to
Palm
D4 growth of new connections at thalamic and cortical
D4 Unresponsive
Palm D3 levels.
D3
D2 D2
D1 D1 Reorganization After Amputation A fourth way of
Face Face Face studying sensory loss has been after injuries to a limb
S1-3b Face Face that require amputation in monkeys or humans. Such
Face
injuries are not common, but occasionally, monkeys in
Figure 11.2 Effects of dorsal column (DC) section at a high social groups are injured to the extent that a therapeu-
cervical level on the somatotopic organization of primary tic amputation is necessary. The somatosensory systems
somatosensory cortex, S1 (area 3b). Only the face to forearm of a few of these monkeys have been studied years after
region is shown. (A) The normal organization of area 3b in
monkeys. The digits 1–5 are represented in order in the injury. After a loss of the lower forearm, neurons
rostral two-thirds of area 3b, just medial to the representation throughout the forearm portion of area 3b and adjoin-
of the face. The palm is represented caudal to the digits. ing areas 1 respond to touch on the stump of the
(B) For days to weeks after a complete or nearly complete DC amputated arm and on the lower face (Florence &
cut, neurons throughout the hand, wrist, and forearm region Kaas, 1995). In addition, neurons in the region of the
of area 3b are unresponsive to tactile stimulation of the fore-
limb. Neurons in the face region remain normally responsive.
hand representation in the ventroposterior nucleus of
(C) After 4–6 post-lesion weeks, much of the deactivated hand the somatosensory thalamus became responsive to
and forelimb region becomes responsive to touch in the touch on the face and on the stump of the amputated
digits, palm, and forearm in a roughly normal somatotopic arm (Florence & Kaas, 2000). As with long-standing
pattern. However, the reactivation is variable across cases, and lesions of the dorsal column afferents, tactile inputs
digit territories are usually smaller than normal, but some-
from the face came to activate portions of the hand
times larger or discontinuous. In other regions, the neurons
are unresponsive to touch. After 6–8 months of recovery, cortex after a major sensory loss from the entire lower
some regions of hand cortex may become responsive to touch arm.
on the face. We know the perceptual consequences of such major
reactivations and reorganizations of the somatosensory
system because similar reactivations occur in humans
grow to reach the cuneate nucleus. A rapid and exten- with long-standing amputations of an arm. Noninvasive
sive recovery of hand use after dorsal column lesions imaging studies (fMRI) of the brain activity in humans
appears to depend on the amount of cortical reactiva- with arm amputations show that much of the hand and
tion, as well as the somatotopy of the reactivation. If arm region of contralateral somatosensory cortex (areas
most of the fingers are represented, especially digits 1 3b, 1, and 2) responds to touch on the face (Flor et al.,
and 2, and in their normal cortical territories, recovery 1995; Karl, Birbaumer, Lutzenberger, Cohen, & Flor,
is faster and better. 2001; Moore et al., 2000). Other evidence comes from
In summary, after a complete or nearly complete a patient who was being treated for pain in the phantom
lesion of the sensory afferents in the dorsal columns of limb after a long-standing amputation (Davis et al.,
the upper spinal cord, contralateral somatosensory 1998). As in many other patients with arm amputations,
cortex is unresponsive to touch on the hand, and hand this patient felt that the arm was still present, a feeling
use is impaired. Cortical reactivation by touch on the called a “phantom limb.” In this patient, as in some
hand is rapid, over a period of weeks, and hand use others, pain was felt in the phantom, and recordings
recovers. This reactivation and hand use is apparently with an electrode were necessary to locate the region
mediated by the potentiation of the few preserved producing the sensation of pain in the phantom.
axons in the dorsal column, and by the potentiation of Recording from neurons in the hand portion of the
second-order axons that travel outside the lesion to ventroposterior nucleus contralateral to the amputa-
reach the cuneate nucleus. These second-order neurons tion revealed receptive fields on the stump of the arm.
normally provide subthreshold modulating inputs. Yet when these same neurons were electrically stimu-
Finally, large lesions that remove more of the inputs to lated with the same electrode, the patient reported
the cuneate nucleus leave parts of the hand representa- feeling touch on the missing hand; that is, the phantom
tion unresponsive for a period of 6 to 8 months, after hand. This result indicates that even though the reacti-
which inputs from the face may activate the cortex via vated thalamic neurons responded to the stump, they
kaas and bowes: plasticity of sensory and motor systems after injury 107
signaled touch on the hand, as if the hand were still more extensive growth of new connections (Jain et al.,
present. This result is consistent with the reported sen- 2000; Xu & Wall, 1997).
sations of patients with amputations of the arm. Such Given the limited plasticity that was observed in the
patients may feel touch on the missing hand (the lateral geniculate nucleus after retinal lesions, a sur-
phantom) when touched on the face, or on the stump prising level of reactivation was found after in primary
(Ramachandran, Rogers-Ramachandran, & Stewart, visual cortex. After months of recovery from small,
1992). bilateral 100 or less lesions of the retina, neurons
In summary, all types of sensory loss produce reactiva- throughout the deprived region of visual cortex
tions of deprived portions of the somatosensory system. responded to visual stimulation with receptive fields
The reactivation can be rapid, over days to weeks, with based on parts of the retina just outside the lesion. As
a limited sensory loss, and the reactivation from pre- this fringe zone of retina activated neurons in both
served afferents appears to lead to considerable func- their normal locations in V1, and in neurons across
tional recovery. When the sensory loss is extensive, the deprived zone, the normal retinotopic organiza-
reactivations are incomplete for periods of months, fol- tion of V1 was disrupted, as neurons in the fringe
lowed by reactivations based on inputs that do not con- and in the deprived zone had overlapping receptive
tribute to any recovery of function, but instead lead to fields. This overlap was called receptive field “pile-up.”
perceptual errors. This happens when inputs from the Cortical reorganization after retinal lesions was first
face activate cortex formerly devoted to the hand. All reported for visual cortex of cats (Kaas et al., 1990),
reactivations appear to depend on the formation of new and soon thereafter for V1 of monkeys (Darian-Smith
connections in the somatosensory system, perhaps at & Gilbert, 1995; Gilbert & Wiesel, 1992; Heinen &
several levels of processing. Skavenski, 1991). With much larger cortical deac-
tivation produced by bilateral lesions of the foveal
region of the retina in monkeys, the reactivation was
Reactivations of visual cortex after lesions of incomplete, revealing a cortical core of unresponsive
the retina cortex (Heinen & Skavenski, 1991). In a more recent
study with microelectrode recording and optical
The capacity of the mature visual system to reorganize imaging, the authors focused on the unresponsive
has been studied by placing small, visuotopically match- core (Smirnakis et al., 2005). However, microelectrode
ing or overlapping lesions of the retina of each eye. This recordings from neurons in cortex bordering the unre-
procedure produced a region in primary visual cortex, sponsive core revealed receptive field “pile-up”—clear
V1, which no longer received its normal source of acti- evidence of plasticity. The reactivation of the deprived
vating input from the lateral geniculate nucleus of the zone of cortex, called the lesion projection zone, con-
visual thalamus. Well before the cortical effects of such tinued to increase, and neuron responses to visual
lesions were known, Eysel, Gonzalez-Aguillar, and Mayer stimuli improved for at least one year (Giannikopoulos
(1980, 1981) discovered that small lesions of the retina & Eysel, 2006). Overall, the evidence for the reactiva-
in cats produced a permanent core of unresponsive tion of visual cortex is compelling. Considerable corti-
neurons in the deprived portions of the lateral genicu- cal plasticity does occur after retinal lesions, but when
late nucleus that were directly denervated by the lesions the region of deactivated primary visual cortex is large,
of the retina. However, the core of unresponsive neurons complete reactivation may not occur.
was bordered by a narrow zone of deprived neurons Because of the limited reactivation of deprived
that had become activated by inputs from parts of the neurons in the lateral geniculate nucleus, the reactiva-
retina around the lesion. Thus, there was plasticity as tion of cortical neurons appears to be largely the result
the neurons along the margins of the deprived zones of the sprouting of cortical axons intrinsic to V1 into
acquired new sources of activation. However, the forma- the deprived zone, as the extents of new axon growth
tion of new connections from the retina in the lateral match the sizes of the reactivated zones (Darian-Smith
geniculate nucleus was very limited. The effects of & Gilbert, 1994; Yamahachi, Mavik, McManus, Denk, &
retinal lesions on the lateral geniculate nucleus of Gilbert, 2009). In addition, lesions of V1 along the
monkeys were similar. Restricted retinal lesions pro- margin of the reactivated cortex block the responsive-
duced a core of unresponsive neurons in the deprived ness of neurons in the lesion-projection zone (Calford,
zones of lateral geniculate nucleus layers, and plasticity Wright, Metha, & Taglianetti, 2003). The response
was extremely limited (Darian-Smith & Gilbert, 1995). properties of the reactivated cortical neurons to visual
In contrast, the neurons in the cuneate nucleus with stimuli appear to be nearly normal, although response
direct inputs from the hand and arm are reactivated by thresholds may be elevated and response magnitudes
kaas and bowes: plasticity of sensory and motor systems after injury 109
cortex (Kaas, Gharbawie, & Stepniewska, 2011; Dan- cortex lend to reactivation of deprived neurons in
cause et al., 2006). But much of the recovery of func- higher-order visual areas, either by a potentiation of
tion likely depends on new termination patterns of other sources of activation, as proposed for the media-
corticospinal projections of intact parts of M1 to tion of blindsight (Cowey, 2010; Jenkins & Merzenich,
occupy and activate deprived motor and premotor 1987; Stoerig & Cowey, 1997; Weiskrantz, 1990; Weisk-
neurons that mediate muscle activity and organize rantz et al., 1974), or by the potentiation of intrinsic,
movements. lateral connections from activated to deprived cortex
Other types of nervous system damage also alter the within the higher-order visual area (Collins, Lyon, &
organization of M1. Amputation of limbs not only Kaas, 2003, 2005). Similarly, a small lesion of primary
deprives motor cortex of its normal sensory input, but somatosensory cortex, area 3b, is followed by a local
also of its normal motor functions. After months to reorganization of that cortex (Jenkins & Merzenich,
years of recovery after therapeutic amputation of the 1987), likely induced by a sprouting of thalamocortical
forelimb in monkeys, electrical stimulation of the inputs, or the potentiation of subthreshold connections
deprived forelimb portion of M1 produces shoulder that existed before the lesions, but new connections are
and arm stump movements (Qi et al., 2000; Schieber & also likely formed in area 1, as well as in other areas
Deuel, 1997; Wu & Kaas, 1999). In part, these changes with a loss of direct inputs from area 3b. Lesions of
in cortical organization were the result of the sprouting primary somatosensory cortex, and adjoining areas 1
of cut peripheral-nerve motor axons that had inner- and 2, resulted in a reactivation of deprived parts of
vated the muscles of the missing limb to newly innervate the second somatosensory area, S2 (Pons, Garraghty, &
muscles of the stump (Wu & Kaas, 2000). Mishkin, 1988). Furthermore, lesions of the hand rep-
The functional reorganization of motor cortex is resentation in M1 lead to an increase in size of the hand
also altered by the sprouting of cut corticospinal axons representation in the ventral premotor area and an
to innervate new spinal cord motor and premotor increase in somatosensory connections (Dancause et
neurons. Lesions of the corticospinal tract at the level al., 2005). In addition, small lesions of cortical regions
of the upper cervical spinal cord are followed by in posterior parietal cortex that are involved in a spe-
impaired hand use, followed by a period of recovery. cific behavior, such as reaching, are followed by a rapid
The corticospinal projections arise predominantly, but recovery of the impaired behavior (Padberg et al.,
not exclusively, from M1 (Galea & Darian-Smith, 1994). 2010). Thus, we propose that wherever neurons are
Most of these projections from M1 are to the contralat- deprived of their normal sources of activation in cortex,
eral spinal cord, but a significant number are to the and likely at other levels, they are partially or totally
ipsilateral spinal cord. Experimental results so far indi- reactivated by the potentiation of other inputs, by the
cate that there is little spontaneous sprouting of cut growth of axons to form new connections, or both.
corticospinal axons past the barrier of the glial scar of
the lesion site (“axons do not penetrate the lesion, Treatments that promote or refine plasticity
cross it, or grow around it … in significant numbers”;
Steward et al., 2008, p. 6836). Instead, much of the Repairing the Injured Central Nervous System: A
behavioral recovery may be due to the considerable Focus on Spinal Cord Injury Although the mature
sprouting of the spared ipsilateral corticospinal axons nervous system retains considerable ability to form new
to cross the spinal cord midline below the spinal cord connections and recover from damage, various treat-
lesion (Rosenzweig et al., 2010). ments have the potential of increasing useful plasticity
In summary, the functional reorganization in motor and promoting functional recovery. Recent studies,
cortex may depend in part on rearrangements of intrin- especially in the spinal cord injury field, have made
sic connections and cortical inputs, but the terminal great progress in elucidating conditions under which it
arbors of corticospinal axons are plastic and capable of can be coaxed to increase its malleability. These com-
forming new contacts. In addition, the synaptic contacts prise not only alterations in synaptic strength, short-
of spinal motor neurons on muscles are plastic and can range rewiring or terminal sprouting (Darian-Smith &
change to innervate different muscles. Ciferri, 2006; Florence & Kaas, 1995; Jain et al., 2000;
Kaas et al., 2008), but also the extension of large
All cortical areas in adult mammals are plastic numbers of axons over remarkable distances (Lu et al.,
2012b). It is worth a reminder that increased plasticity
All cortical areas in adult primates and other mammals within the central nervous system (CNS) doesn’t neces-
are likely plastic. The evidence is not comprehensive, sarily translate to improved functional recovery (Lu et
but it is compelling. Partial lesions of primary visual al., 2012a), and indeed may actually induce mechanical
kaas and bowes: plasticity of sensory and motor systems after injury 111
most other studies had focused on the resulting changes (PTEN) in the sensorimotor cortex of adult mice. Liu
at the spinal cord level, the reactivation of primary and colleagues (2010) used Cre-expressing adeno-asso-
somatosensory cortex to which dorsal column afferents ciated virus (AAV-Cre) to delete the gene encoding
are relayed was evaluated. By intentionally sparing PTEN in homozygous conditional PTEN mutant mice.
dorsal column afferents from digit 1 (thumb), it was After lesions of the corticospinal tract in adults that
demonstrated that months after the dorsal column had received neonatal AAV-Cre injections, PTEN dele-
lesion and cuneate nucleus treatment, the cortical ter- tion elicited considerable sprouting of the intact ipsi-
ritory in area 3b of somatosensory cortex activated by lateral corticospinal tract into the contralateral cord.
touch on D1 was larger in chABC-treated than in control Further studies by these authors (Sun et al., 2011)
monkeys. showed an increased capacity for axonal regeneration
after co-deletion of PTEN and suppressor of cytokine
Gene therapy signaling 3 (SOCS3) in an optic nerve crush model,
but similar observations have not yet been demon-
Growth Factors Given the discovery that secretion strated in the spinal cord. Importantly, it has been
of exogenous growth factors has the ability to not only shown that conditionally deleted SOCS3 expression in
promote the regrowth of axons after injury, but to also nestin-expressing cells have increased STAT3 activa-
serve in chemotactic guidance toward appropriate tion, and that this limits the inflammatory expression
targets (Massey et al., 2008), researchers have sought after injury, thereby reducing cell death (Okada et al.,
methods to deliver these neurotrophic factors constitu- 2006). Alternatively, CNS deletion of STAT3 delays
tively after injury. Additionally, growth-factor delivery formation of the glial scar, leading to greater inflam-
has been shown to be effective not only when delivered mation, cell death, and functional impairment (Her-
acutely, but also after chronic injuries (Kadoya et al., rmann et al., 2008). Therefore, SOCS3 reduction may
2009). However, widespread delivery of these neuro- indeed be a good combinatorial component for reduc-
trophic factors can lead to adverse effects, and there- ing post-SCI deficits.
fore local and targeted administration is required. In 2009, a group of researchers (Moore et al., 2009)
Extended neurotrophic delivery has been achieved via screened developmentally regulated genes in retinal
infusion, ex vivo methods wherein grafts of tissue genet- ganglion cells and identified a family of Kruppel-like
ically modified to secrete brain-derived neurotropic transcription factors (KLF) that regulated intrinsic
factor, neurotrophin -4/5, or glial-cell-line–derived axon regeneration ability. Of these, KLF7 was later
neurotropic factor have been implanted into the lesion found to have a large impact on axon growth after its
site after SCI, or via viral vector gene delivery (Blits, developmental down-regulation was countered with
Dijkhuizen, Boer, & Verhaagen, 2000; Piantino, Burdick, overexpression in in vitro cortical neurons (Blackmore
Goldberg, Langer, & Benowitz, 2006; Ruitenberg, et al., 2012). This held true in vivo when overexpression
Vukovic, Sarich, Busfield, & Plant, 2006; Schnell, Sch- of a KLF7 chimera with a VP16 transactivation domain
neider, Kolbeck, Barde, & Schwab, 1994; Zhou & Shine, in adult mice produced axonal regrowth in the CST
2003). These grafts were usually successful in inducing tract.
axonal elongation, but only to a limited extent. Typically,
greater elongation of sensory axons occurred (Bradbury Cell Transplantation The allure of cell transplan-
et al., 1999; Taylor, Jones, Tuszynski, & Blesch, 2006), tation is quite understandable given the many potential
but corticospinal tract axons remained resistant to benefits that could be achieved with an effective trans-
manipulation. plant. Cell transplantation has the capacity to physically
bridge a lesion site, replace lost neurons or supporting
Regeneration-Associated Genes As the central cells, secrete growth factors that can enhance regenera-
nervous system matures, a down-regulation of mTOR tion or sprouting of damaged axons, and even aid in
activity occurs. Age and injury-dependent down-regu- chemotactic signaling for guidance toward appropriate
lation of neuronal mTOR activity was identified as a targets (Biernaskie et al., 2007; Grill, Blesch, & Tuszyn-
major cause of the lack of regeneration of optic nerve ski, 1997; Guest, Rao, Olson, Bunge, & Bunge, 1997;
axons after injury, and genetic activation of mTOR was Haas & Fischer, 2013; Kajikawa, de la Mothe, Blumell,
observed to facilitate axon regrowth (Park et al., & Hackett, 2003; Kobayashi et al., 2012; Martin et al.,
2008). Subsequent efforts (Liu et al., 2010) revealed 1996; Ruitenberg et al., 2006; Tuszynski, Grill, Jones,
that considerable sprouting of the corticospinal tract McKay, & Blesch, 2002; X. M. Xu, Zhang, Li, Aebischer,
also occurred after the conditional deletion of the & Bunge, 1999). Regenerating axons will grow into a
gene for the phosphate tensor homologue protein cell graft, but the greatest difficulty still lies in coaxing
kaas and bowes: plasticity of sensory and motor systems after injury 113
after a cervical dorsal column lesion. Proc Natl Acad Sci USA, Darian-Smith, C., & Ciferri, M. M. (2006). Cuneate nucleus
109, 2595–2600. reorganization following cervical dorsal rhizotomy in the
Bradbury, E. J., Khemani, S., Von, R., King, V. R., Priestley, macaque monkey: Its role in the recovery of manual dexter-
J. V., & McMahon, S. B. (1999). NT-3 promotes growth of ity. J Comp Neurol, 498, 552–565.
lesioned adult rat sensory axons ascending in the dorsal Darian-Smith, C., & Gilbert, C. D. (1994). Axonal sprout-
columns of the spinal cord. Euro J Neurosci, 11, 3873–3883. ing accompanies functional reorganization in adult cat
Bradbury, E. J., Moon, L. D., Popat, R. J., King, V. R., striate cortex. Nature, 368, 737–740.
Bennett, G. S., Patel, P. N., … McMahon, S. B. (2002). Darian-Smith, C., & Gilbert, C. D. (1995). Topographic
Chondroitinase ABC promotes functional recovery after reorganization in the striate cortex of the adult cat
spinal cord injury. Nature, 416, 636–640. and monkey is cortically mediated. J Neurosci, 15, 1631–
Brodmann, K. (1909). Vergleichende Lokalisationslehre der Gross- 1647.
hirnrhinde. Barth, Leipzig. David, S., Lopez-Vales, R., & Wee Yong, V. (2012). Harmful
Buomomano, D. V., & Merzenich, M. M. (1998). Cortical and beneficial effects of inflammation after spinal cord
plasticity, from synapses to maps. Ann Rev Neurosci, 21, injury; potential therapeutic implications. Handb clin neurol,
149–196. 109, 485–502.
Calford, M. B., Wright, L. I., Metha, A. B., & Taglianetti, Davis, K. D., Kiss, Z. H. T., Luo, L., Tasker, R. R., Lozano,
V. (2003). Topographic plasticity in primary visual cortex is A. M., & Dostrovsky, J. O. (1998). Phantom sensations
mediated by local corticocortical connections. J Neurosci, 23, generated by thalamic microstimulation. Nature, 391,
6434–6442. 385–387.
Caroni, P., Savio, T., & Schwab, M. E. (1988). Central Daw, N. W. (1995). Visual development. New York, NY: Plenum
nervous system regeneration, oligodendrocytes and myelin Press.
as non-permissive substrates for neurite growth. Prog Brain Deng, L. X., Deng, P., Ruan, Y., Xu, Z. C., Liu, N. K., Wen,
Res, 78, 363–370. X., … Xu, X. M. (2013). A novel growth-promoting pathway
Caroni, P., & Schwab, M. E. (1988). Antibody against myelin- formed by GDNF-overexpressing Schwann cells promotes
associated inhibitor of neurite growth neutralizes nonper- propriospinal axonal regeneration, synapse formation, and
missive substrate properties of CNS white matter. Neuron, 1, partial recovery of function after spinal cord injury. J. Neu-
85–96. rosci, 33, 5655–5667.
Casha, S., Zygun, D., McGowan, M. D., Bains, I., Yong, Detloff, M. R., Fisher, L. C., McGaughy, V., Longbrake,
V. W., & Hurlbert, R. J. (2012). Results of a phase II placebo- E. E., Popovich, P. G., & Basso, D. M. (2008). Remote
controlled randomized trial of minocycline in acute spinal activation of microglia and pro-inflammatory cytokines
cord injury. Brain, 135(PT 4), 1224–1236. predict the onset and severity of below-level neuropathic
Chen, L. M., Qi, H-X., & Kaas, J. H. (2012). Dynamic reorgan- pain after spinal cord injury in rats. Exp Neurol, 212,
ization of digit representations in somatosensory cortex of 337–347.
nonhuman primates after spinal cord injury. J Neurosci, 32, Diamond, M. E., Armstrong-James, M., & Ebner, F. F.
14649–14663. (1993). Experience-dependent plasticity in adult rat barrel
Chino, Y. M., Smith, E. L., 3rd, Kaas, J. H., Sasaki, Y., & cortex. Proc Natl Acad Sci USA, 90(5), 2082–2086.
Cheng, H. (1995). Receptive-field properties of deafferen- Donnelly, D. J., & Popovich, P. G. (2008). Inflammation and
tated visual cortical neurons after topographic map reorga- its role in neuroprotection, axonal regeneration and func-
nization in adult cats. J Neurosci, 15, 2417–2433. tional recovery after spinal cord injury. Exp Neurol, 209,
Collins, C. E., Lyon, D. C., & Kaas, J. H. (2003). Responses 378–388.
of neurons in the middle temporal visual area after long- Eysel, U. T., Gonzalez-Aguillar, F., & Mayer, U. (1980). A
standing lesions of the primary visual cortex in adult New functional sign of reorganization in the visual system of
World monkeys. J Neurosci, 23, 2251–2264. adult cats: Lateral geniculate neurons with displaced recep-
Collins, C. E., Lyon, D. C., & Kaas, J. H. (2005). Distribution tive fields after lesions of the nasal retina. Brain Res, 191,
across cortical areas of neurons projecting to the superior 285–300.
colliculus in New World monkeys. Anat Rec, 285A, Eysel, U. T., Gonzalez-Aguillar, F., & Mayer, U. (1981).
619–627. Time-dependent decrease in the extent of visual deafferen-
Cowey, A. (2010). The blindsight saga. Exp Brain Res, 200, tation in the lateral geniculate nucleus of adult cats with
3–24. small retinal lesions. Exp Brain Res, 41, 256–263.
Dancause, N., Barbay, S., Frost, S. B, Plautz, E. J., Chen, Fawcett, J. W., Schwab, M. E., Montani, L., Brazda, N., &
D., Zoubina, E. V., … Nudo, R. J. (2005). Extensive cortical Muller, H. W. (2012). Defeating inhibition of regenera-
rewiring after brain injury. J Neurosci, 25, 10167–10179. tion by scar and myelin components. Handb Clin Neurol,
Dancause, N., Barbay, S., Frost, S. B, Zoubina, E. V., 109, 503–522.
Plautz, E. J., Mahnken, J. D., & Nudo, R. J. (2006). Effects Flor, H., Elbert, T., Knecht, S., Wienbruch, C., Pantev,
of small ischemic lesions in the primary motor cortex on C., Birbaumer, N., … Taub, E. (1995). Phantom-limb pain
neurophysiological organization in ventral premotor as a perceptual correlate of cortical reorganization follow-
cortex. J Neurophysiol, 96, 3506–3511. ing arm amputation. Nature, 375, 482–484.
Dancause, N., & Nudo, R. J. (2011). Shaping plasticity to Florence, S. L., Garraghty, P. E., Wall, J. T., & Kaas, J. H.
enhance recovery after injury. Prog Brain Res, 192, (1994). Sensory afferent projections and area 3b somato-
273–291. topy following median nerve cut and repair in macaque
Darian-Smith, C., & Ciferri, M. M. (2005). Loss and recov- monkeys. Cereb Cortex, 4, 391–407.
ery of voluntary hand movements in the macaque following Florence, S. L., & Kaas, J. H. (1995). Large-scale reorganiza-
a cervical dorsal rhizotomy. J Comp Neurol, 491, 27–45. tion at multiple levels of the somatosensory pathway follows
kaas and bowes: plasticity of sensory and motor systems after injury 115
bridging axonal regeneration one year after spinal cord Merzenich, M. M., Kaas, J. H., Wall, J. T., Sur, M., Nelson,
injury. Neuron, 64, 165–172. R. J., & Felleman, D. J. (1983b). Progression of change
Kajikawa, Y., de la Mothe, L., Blumell, S., & Hackett, following median nerve section in the cortical representa-
T. A. (2003). Response properties of neurons in medial belt tion of the hand in areas 3b and 1 in adult owl and squirrel
auditory cortex of marmoset monkeys. Assoc Res Otolaryngol monkeys. Neuroscience, 10, 639–665.
Abstr, 26, 204. Moon, L. D., Asher, R. A., Rhodes, K. E., & Fawcett, J. W.
Karl, A., Birbaumer, N., Lutzenberger, W., Cohen, L. G., (2001). Regeneration of CNS axons back to their target
& Flor, H. (2001). Reorganization of motor and somato- following treatment of adult rat brain with chondroitinase
sensory cortex in upper extremity amputees with phanton ABC. Nat Neurosci, 4, 465–466.
limb pain. J Neurosci, 21, 3609–3618. Moore, C. I., Stern, C. E., Dunbar, C., Kostyk, S. K., Gehi,
Kobayashi, Y., Okada, Y., Itakura, G., Iwai, H., Nishimura, A., & Corkin, S. (2000). Referred phantom sensation and
S., Yasuda, A., … Okano, H. (2012). Pre-evaluated safe cortical reorganization after spinal cord injury in humans.
human iPSC-derived neural stem cells promote functional Proc Natl Acad Sci USA, 97, 14703–14705.
recovery after spinal cord injury in common marmoset Moore, D. L., Blackmore, M. G., Hu, Y., Kaestner, K. H.,
without tumorigenicity. PLoS ONE, 7, e52787. Bixby, J. L., Lemmon, V. P., & Goldberg, J. L. (2009). KLF
Lee, Y.-S., Lin, C.-Y., Jiang, H.-H., DePaul, M., Lin, V. W., & family members regulate intrinsic axon regeneration ability.
Silver, J. (2013). Nerve regeneration restores supraspinal Science, 326, 298–301.
control of bladder function after complete spinal cord Nudo, R. L., & Milliken, G. W. (1996). Reorganization of
injury. J Neurosci, 33, 10591–10606. movement representations in primary motor cortex follow-
Lemons, M. L., Howland, D. R., & Anderson, D. K. (1999). ing focal ischemic infarcts in adult squirrel monkeys. J Neu-
Chondroitin sulfate proteoglycan immunoreactivity in- rophysiol, 75, 2144–2149.
creases following spinal cord injury and transplantation. Nudo, R. L., Milliken, G. W., Jenkins, W. M., & Merzenich, M.
Exp Neurol, 160(1), 51–65. M. (1996). Use dependent alterations of movement repre-
Li, W., Piech, V., & Gilbert, C. D. (2004). Perceptual learn- sentations in primary motor cortex of adult squirrel
ing and top-down influences in primary visual cortex. Nat monkeys. J Neurosci, 96, 785–807.
Neurosci, 7, 651–657. Okada, S., Nakamura, M., Katoh, H., Miyao, T., Shimazaki,
Liebscher, T., Schnell, L., Schnell, D., Scholl, J., T., Ishii, K., … Okano, H. (2006). Conditional ablation of
Schneider, R., Gullo, M., … Schwab, M. E. (2005). Stat3 or Socs3 discloses a dual role for reactive astrocytes
Nogo-A antibody improves regeneration and locomotion of after spinal cord injury. Nat Med, 12, 829–834.
spinal cord-injured rats. Ann Neurol, 58, 706–719. Padberg, J., Recanzone, G., Engle, J., Cooke, D., Goldring,
Liu, K., Lu, Y., Lee, J. K., Samara, R., Willenberg, R., Sears- A., & Krubitzer, L. (2010). Lesions of posterior parietal
Kraxberger, I., … He, Z. (2010). PTEN deletion enhances area 5 in monkeys result in behavioral and cortical plastic-
the regenerative ability of adult corticospinal neurons. Nat ity. J. Neurosci, 30, 12918–12935.
Neurosci, 13, 1075–1081. Park, K. K., Liu, K., Hu, Y., Smith, P. D., Wang, C., Cai, B.,
Lord-Fontaine, S., Yang, F., Diep, Q., Dergham, P., Munzer, & He, Z. (2008). Promoting axon regeneration in the adult
S., Tremblay, P., & McKerracher, L. (2008). Local inhibi- CNS by modulation of the PTEN/mTOR pathway. Science,
tion of Rho signaling by cell-permeable recombinant 322, 963–966.
protein BA-210 prevents secondary damage and promotes Piantino, J., Burdick, J. A., Goldberg, D., Langer, R., &
functional recovery following acute spinal cord injury. Benowitz, L. I. (2006). An injectable, biodegradable
J Neurotrauma, 25(25), 1309–1322. hydrogel for trophic factor delivery enhances axonal rewir-
Lu, P., Blesch, A., Graham, L., Wang, Y., Samara, R., Banos, ing and improves performance after spinal cord injury. Exp
K., … Tuszynski, M. H. (2012a). Motor axonal regenera- Neurol, 201, 359–367.
tion after partial and complete spinal cord transection. Plant, G. W., Bates, M. L., & Bunge, M. B. (2001). Inhibitory
J Neurosci, 32, 8208–8218. proteoglycan immunoreactivity is higher at the caudal than
Lu, P., Wang, Y., Graham, L., McHale, K., Gao, M., Wu, D., the rostral Schwann cell graft-transected spinal cord inter-
… Tuszynski, M. H. (2012b). Long-distance growth and face. Mol Cell Neurosci, 17, 471–487.
connectivity of neural stem cells after severe spinal cord Pons, T. P., Garraghty, P. E., & Mishkin, M. (1988). Lesion-
injury. Cell, 150, 1264–1273. induced plasticity in the second somatosensory cortex of
Martin, D., Robe, P., Franzen, R., Delree, P., Schoenen, J., adult macaques. Proc Natl Acad Sci USA, 85, 5279–5281.
Stevenaert, A., & Moonen, G. (1996). Effects of Schwann Pons, T. P., Garraghty, P. E., Ommaya, A. K., Kaas, J. H.,
cell transplantation in a contusion model of rat spinal cord Taub, E., & Mishkin, M. (1991). Massive cortical reorgani-
injury. J Neurosci Res, 45, 588–597. zation after sensory deafferentation in adult macaques.
Massey, J. M., Amps, J., Viapiano, M. S., Matthews, R. T., Science, 252, 1857–1860.
Wagoner, M. R., Whitaker, C. M., … Onifer, S. M. (2008). Qi, H. X., Chen, L. M., & Kaas, J. H. (2011). Reorganization
Increased chondroitin sulfate proteoglycan expression in of somatosensory cortical areas 3b and 1 after unilateral
denervated brainstem targets following spinal cord injury section of dorsal columns of the spinal cord in squirrel
creates a barrier to axonal regeneration overcome by chon- monkeys. J Neurosci, 31, 13662–13675.
droitinase ABC and neurotrophin-3. Exper Neurol, 209, Qi, H. X., Gharbawie, O. A., Wynne, K. W., & Kaas, J. H.
426–445. (2013). Impairment and recovery of hand use after unilat-
Merzenich, M. M., Kaas, J. H., Wall, J., Nelson, R. J., Sur, eral section of the dorsal columns of the spinal cord in
M., & Felleman, D. (1983a). Topographic reorganization squirrel monkeys. Behav Brain Res, 252C, 363–376.
of somatosensory cortical areas 3b and 1 in adult monkeys Qi, H. X., Jain, N., Collins, C. E., Lyon, D. C., & Kaas, J. H.
following restricted deafferentation. Neurosci, 8, 33–55. (2010). Functional organization of motor cortex of adult
kaas and bowes: plasticity of sensory and motor systems after injury 117
Wiesel, T. N. (1982). Postnatal development of the visual into hemisected adult rat spinal cord. Euro J Neurosci, 11,
cortex and the influence of environment. Nature, 299, 1723–1740.
583–591. Yamahachi, H., Mavik, S. A., McManus, J. N., Denk, W., &
Willott, J. F., Aitkin, L. M., & McFadden, S. L. (1993). Gilbert, C. D. (2009). Rapid axonal sprouting and pruning
Plasticity of auditory cortex associated with sensorineural accompany functional reorganization in primary visual
hearing loss in adult C57BL/6J mice. J Comp Neurol, 329, cortex. Neuron, 64, 719–729.
402–411. Yang, L., Blumbergs, P. C., Jones, N. R., Manavis, J.,
Witham, C. L., & Baker, S. N. (2011). Modulation and trans- Sarvestani, G. T., & Ghabriel, M. N. (2004). Early expres-
mission of peripheral inputs in monkey cuneate and exter- sion and cellular localization of proinflammatory cytokines
nal cuneate nuclei. J Neurophysiol, 106, 2764–2775. interleukin-1beta, interleukin-6, and tumor necrosis factor-
Wu, C. W., & Kaas, J. H. (1999). Reorganization in primary alpha in human traumatic spinal cord injury. Spine, 29,
motor cortex of primates with long-standing therapeutic 966–971.
amputations. J Neurosci, 19, 7679–7697. Yang, L., Jones, N. R., Blumbergs, P. C., van den Heuvel,
Wu, C. W.-H., & Kaas, J. H. (2000). Spinal cord atrophy and C., Moore, E. J., Manavis, J., & Ghabriel, M. N. (2005).
reorganization of motor neuron connections following Severity-dependent expression of pro-inflammatory cyto-
long-standing limb loss in primates. Neuron, 28, 1–20. kines in traumatic spinal cord injury in the rat. J Clin Neu-
Xu, J., & Wall, J. T. (1997). Rapid changes in brainstem maps rosci, Australasia, 12(3), 276–284.
of adult primates after peripheral injury. Brain Res, 774, Zhou, L., & Shine, H. D. (2003). Neurotrophic factors
211–215. expressed in both cortex and spinal cord induce axonal
Xu, X. M., Zhang, S. X., Li, H., Aebischer, P., & Bunge, M. plasticity after spinal cord injury. J Neurosci Res, 74,
B. (1999). Regrowth of axons into the distal spinal cord 221–226.
through a Schwann-cell-seeded mini-channel implanted
ABSTRACT We discuss the relationship between modularity of Noor, & Alon, 2007). At short time, modularity improves
neural activity in the brain and cognitive ability and review the plasticity of a system, because the system more effi-
observations and theories relating modularity to plasticity of
ciently finds a solution in the smaller, modular sub-
brain neural activity. By analogy with evolutionary biology, we
hypothesize that selection for maximum plasticity of the space. At long time, the system has enough time to find
human brain occurs in young adulthood, which implies mod- the optimal organization in the space of all possibilities,
ularity should peak in young adults. We show that modularity which is unlikely to be in the modular subspace. There
of neural activity derived from functional MRI data rises from is, then, a tension between the increased ability to adapt
childhood, peaks in young adults, and declines in older that modularity confers upon a system at short time and
adults. We suggest experiments to measure the impact of
modularity on fast, low-level, automatic cognitive processes the constraint that modularity imposes upon a system
versus high-level, effortful, conscious cognitive processes. We at long time. The amount of modularity that arises in a
suggest modularity may be a useful biomarker for disease, system reflects a balance between these two competing
injury, or rehabilitation. effects, as a function of the time scale at which selection
occurs (Sun & Deem, 2007).
Biological systems are modular, and the organization of Modularity is an order parameter that characterizes
their genetic material reflects this modularity (Hart- the structure of correlations of neural activity in the
well, Hopfield, Leibler, & Murray, 1999; Rojas, 1996; brain. If a network of neural brain activity can change
Simon, 1962; Waddington, 1942). Genes are organized its architecture, it will change its modularity to a value
into exons, and expression of different exons allows that achieves the balance of plasticity and cognitive per-
one gene to produce multiple proteins. In many cases, formance demanded of it (Lorenz, Jeng, & Deem,
each exon confers a distinct function to the protein, 2011). In particular, a system under greater pressure to
and so an exon is a modular element. Similarly, while respond quickly will tend to develop a more modular
genes interact, each gene confers function or functions distribution of connections. A pressure to respond
to the organism, and so a gene is a modular element. quickly selects for an effective response function of the
Collections of related genes clustered together on the system, at the time scale of the environmental change
genome occasionally form a functional unit termed an (Sun & Deem, 2007). More modular systems have a
operon, which is also a module. more rapid response at short times. The rate of growth
In this chapter, we are interested in a different sort of modularity from an initially nonmodular state is
of modularity, that of spatially and temporally corre- roughly proportional to the variability in selection pres-
lated neural activity in the brain. A group of neurons sure (Sun & Deem, 2007). Similarly, if there is selection
that tends to be activated above threshold for a period pressure for a system to adapt rapidly to steady state
of time in response to a stimulus is termed a region of from an initially unfit state, modularity can arise
interest, which is a module (Schwarz, Gozzi, & Bifone, (Wagner & Altenberg, 1996). The coupling between
2008). Remarkably, these modules of neural activity are transport and a heterogeneous spatial environment
also generated spontaneously from purely subject- also leads to the emergence of modularity (Kashtan,
driven cognitive states (Shirer, Ryali, Rykhlevskaia, Parter, Dekel, Mayo, & Alon, 2009).
Menon, & Greicius, 2012). The remainder of this chapter is organized as
Organization of biology into modules simultaneously follows. In the following section, we introduce modu-
restricts the possibilities for function, because the larity and discuss a number of biological examples of
modular organization is a subset of all possible organi- modularity, culminating with a discussion of modular-
zations, and may lead to more rapid evolution, because ity in neural activity. We provide an intuitive and a
the evolution occurs in a vastly restricted modular sub- mathematical definition of modularity and discuss how
space of all possibilities (Sun & Deem, 2007; Kashtan, modularity may adapt in time. In the third section,
chen and deem: modularity, plasticity, and development of brain function 119
“Modularity and brain function,” we discuss how mod- clusters into these regions. The fMRI data measure
ularity in neural activity of the brain is correlated with brain activity with spatial and temporal resolution
learning rate. We also discuss how modularity is during a 20- to 30-minute imaging session.
lowered in patients with brain injury or disease. In The data are analyzed by extracting the activity as a
“Modularity and plasticity,” we discuss how modularity function of position within the brain and may be further
increases plasticity and how increased modularity of averaged within regions of interest—for example, all
neural activity in the brain is expected to increase per- the Brodmann’s areas. The correlations of neural activ-
formance at task switching. In “Modularity of neural ity among different regions of interest are then com-
activity increases then decreases with age,” we consider puted. A cutoff is typically applied so that only the
how modularity of neural activity of the brain develops largest of these correlations are considered. A sche-
from childhood to old age. We show that modularity matic of this process is shown in figure 12.1.
rises, peaks in young adults, and declines in old adults. Modularity is defined by rearranging and partition-
We also show that modularity is positively correlated ing the correlation matrix, with cutoff applied, so that
with raw IQ score. In the following section, we discuss it is as block-diagonal as possible (Fortunato, 2010).
some of the details of functional magnetic resonance Mathematically, modularity is the probability to have a
imaging (fMRI) analysis. We conclude in the final correlation within the modules, minus the probability
section. to have a correlation expected by random chance
(Newman, 2006). In other words,
Modularity 1 ⎛ a ia j ⎞
M= ∑
2e all modules
∑ ⎜⎝ A ij −
2e ⎠
⎟ (1)
Biological systems have long been recognized to be nodes ij within this module
modular. In 1942, Waddington presented his now classic where Aij is 1 if there is a link between node i and node
description of a canalized landscape for development, j and 0; otherwise, ai = Σj Aij is the degree of node i,
in which minor perturbations do not disrupt the func- 1
tion of developmental modules (Waddington, 1942). In and e = ∑ i a i the total number of links. The value of
2
1961, H. A. Simon described how biological systems are modularity depends on the definition of the modules,
more efficiently adapted and are more stable if they are that is, how the nodes are partitioned into modules in
modular (Simon, 1962). A seminal paper by Hartwell, Eq. 1. The clustering of the nodes into modules is
Hopfield, Leibler, and Murray (1999) firmly established defined by choosing the partitioning that maximizes
the concept of modularity in cell biology. Systems Eq. 1.
biology has since provided a wealth of examples of The modularity of biological systems changes over
modular cellular circuits, including metabolic circuits time. There are a number of demonstrations of the
(Ravasz, Somera, Mongru, Oltvai, & Barabási, 2002) evolution of modularity in biological systems. For
and modules on different scales; that is, modules of example, the modularity of the protein-protein interac-
modules (da Silva, Ma, & Zeng, 2008). Protein-protein tion network significantly increases when yeast are
interaction networks have been observed to be modular exposed to heat shock (Mihalik & Csermely, 2011), and
(Gavin et al., 2006; Spirin & Mirny, 2003; von Mering the modularity of the protein-protein networks in both
et al., 2003). yeast and E. coli appears to have increased over evolu-
Ecological food webs have been found to be modular tionary time (He, Sun, et al., 2009). Additionally, food
(Krause, Frank, Mason, Ulanowicz, & Taylor, 2003). webs in low-energy, stressful environments are more
The gene regulatory network of the developmental modular than those in plentiful environments (Lorenz
pathway exhibits modules (Raff & Raff, 2000; Wagner, et al., 2011), arid ecologies are more modular during
1996), and the developmental pathway is modular droughts (Rietkerk, Dekker, de Ruiter, & van de Koppel,
(Klingenberg, 2008). 2004), and foraging of sea otters is more modular when
Modules have also been found in spatial correlations food is limiting (Tinker, Bentall, & Estes, 2008). Other
of brain activity (Chavez et al., 2010; Ferrarnini et al., complex dynamical systems exhibit time-dependent
2009; He, Wang et al., 2009; Meunier, Achard, Morcom, modularity as well. The modularity of social networks
& Bullmore, 2009; Meunier, Lambiotte, & Bullmore, changes over time: stock-brokers’ instant messaging
2010; Schwarz et al., 2008). Modularity of brain activity networks are more modular under stressful market con-
is computed by analyzing correlations in fMRI data ditions (Saavedra, Hagerty, & Uzzi, 2011), and socioeco-
(Meunier et al., 2009; Schwarz et al., 2008). Intuitively, nomic community overlap decreases with increasing
modules are regions of correlated brain activity, and stress (Estrada & Hatano, 2010). Modularity of financial
modularity is a measure of how well the brain activity networks changes over time: the modularity of the world
C
D
Figure 12.1 Clustering of brain neural activity into modules modularity calculated from Eq. 1 is 0.6441, and the contribu-
and calculation of modularity. (A) The Brodmann’s areas, tions to M are 0.1585, 0.1310, 0.1592, and 0.1954 from the
onto which the measured fMRI neural activity data are pro- green, red, yellow, and orange modules, respectively. (D) The
jected (Dubuc). (B) Correlation matrix of the neural activity Brodmann’s areas grouped into the four modules. In this
between the different Brodmann’s areas, with only the largest slice, only two voxels of the second module are visible, with
elements retained (white). (C) Reordered correlation matrix, the majority of the voxels in a different region of the brain
showing the grouping of the areas into four modules. The not visible in this slice. From Chen & Deem (2014). (See color
Brodmann’s areas are grouped into modules by permuting plate 5.)
the rows and columns of the correlation matrix. The
trade network has decreased over the last 40 years, Features of modularity are correlated with learning
leading to increased susceptibility to recessionary shocks rates. In a study of subjects learning a simple motor
(He & Deem, 2010), and increased modularity has been skill, the flexibility of neural activity was positively cor-
suggested as a way to increase the robustness and adapt- related with learning ability (Bassett et al., 2011). Flex-
ability of the banking system (Haldane & May, 2011). ibility was defined as the probability that voxels of brain
neural activity participated in multiple modules. In
Modularity and brain function other words, the ability of the brain to reconfigure
neural activity into different modules was predictive of
Neural activity in the brain is modular (Fodor, 1983; learning rate.
Mountcastle, 1979). That is, neural activity in different In some cases, modularity of neural activity can dis-
regions of the brain is stimulated by different tasks. tinguish healthy subjects and patients with disease. For
Brain function of subjects activates different regions, example, an analysis of magnetoencephalographic
which can be observed in fMRI experiments (Shirer et signals from epileptic patients showed their neural
al., 2012). Patterns of neural activity that are activated activity to be less modular than that of normal subjects
during attention- and memory-demanding tasks are (Chavez et al., 2010). Patients with epilepsy had neural
correlated with structural connectivity in the brain activity networks with greater connectivity in addition
(Hermundstad et al., 2013). to lower modularity. These patients also had more
chen and deem: modularity, plasticity, and development of brain function 121
connections between nodes in different modules, which
by the definition of Newman’s modularity lowers the
modularity. Anecdotal evidence suggests patients with
traumatic brain injury have less modular neural activity
as well (Leclercq et al., 2000).
Fitness, f
M = 0.8
0.55
Modularity enhances plasticity, and modularity of
0.5 brain neural activity peaks in adulthood. This observa-
tion may be placed in mathematical form by consider-
0.45
ing the fitness of the brain to be a function of modularity.
0.4 For young children, this fitness function peaks at a
0.35
lower value of modularity than it does for adults. For
adults over 62, the fitness function again peaks at a
0.3 lower value of modularity. Figure 12.6 shows how the
200 300 400 500
fitness as a function of modularity may look for chil-
Number of edges
dren, young adults, and older adults. The modularities
Figure 12.3 Modularity of neural activity for children and measured for 400 edges are used to construct this esti-
adults. Modularity is greater for adults than for children.
mation of the fitness function.
Modularity is calculated from the correlation of neural activity
between Brodmann’s areas. The entries of the correlation
matrix are projected to unity for values above a cutoff and set Functional MRI data analysis
to zero otherwise. The number of correlation matrix entries
above the cutoff value—i.e., the number of white entries Here we present some details of the calculation of mod-
shown in figure 12.1—is denoted by edges. The trend of ularity from fMRI data. Such fMRI data provide an
increased modularity for adults is persistent for different cut-
offs—i.e., different edges—which quantify differing sparse- indirect measure of neural activity in the brain with
ness of the projected correlation matrix. The error bars are millimeter spatial and second temporal resolution. The
one standard error. (From Chen & Deem, 2014.) fMRI signal is an approximation of blood flow to each
chen and deem: modularity, plasticity, and development of brain function 123
0.55 0.5
0.5
Correlation between age and modularity
0.4
0.4
0.3
0.35
0.3
0.2
0.25
0.2
0.1
0.15
0.1 0
200 300 400 500 200 300 400 500
Number of edges Number of edges
Figure 12.4 For each child, the age is known, the raw KBIT-2 modularity and age. Right: The correlation between modular-
IQ score was measured (Cantlon & Li, 2013), and the modu- ity and raw KBIT-2 overall IQ score. Modularity is positively
larity of neural brain activity was calculated. The correlation correlated with both age and IQ. These positive correlations
between modularity and either age or IQ was calculated. Ages show that modularity increases with development. (From
were in the range 4–11. Left: The correlation between Chen & Deem, 2014.)
Modularity
0.44
Fitness
0.42
0.4
0.38
0 20 40 60 80
0.424 0.454 0.488 Age
Modularity
B
Kids, 4−11
Adults, 18−26
Adults, 62−76
figures 12.3 and 12.5. Data are from subjects with ages in the
solid part of the curve. (B) Schematic figure of the fitness of
brain activity as a function of modularity. Fitness is a measure
of the quality of neural activity, over all the tasks demanded of
the brain. To measure this average fitness, figure 12.2 is con-
structed for many typical challenges and typical response
times, and an average is taken. The peak of fitness is set at the
observed values of modularity for children, young adults, and
old adults. The width of the curve is set schematically to the
0.424 0.454 0.488
standard deviation of the observed subject modularity within
Modularity each group.
high-resolution anatomical data, typically by registering of interest is then calculated (Meunier et al., 2009;
one of the first few images of the fMRI data. Since Schwarz et al., 2008).
brains of different subjects are of different sizes, the Only those elements above a cutoff value are retained
imaging data are then typically scaled into standardized and set to unity, with all other elements set to zero. The
Talairach brain coordinates (Talairach & Tournoux, modularity of this cutoff correlation matrix is then cal-
1988). The fMRI data are then registered during the culated by Eq. 1.
time course to correct for frame-to-frame head motion
of the subject. The outcome of these processing steps Discussion and conclusion
gives neural activity as a function of time in fixed, stan-
dardized spatial coordinates. Modularity quantifies the degree to which neural activ-
For further analysis, the fMRI data may be pro- ity in the brain is modular. Modularity is a mathematical
jected to standardized regions of interest, for example concept, expressed in Eq. 1. Physical regions of the
the Brodmann’s areas; see figure 12.7. The average brain were defined by early anatomical studies, and
value of the processed fMRI data in each region may later studies identified functional regions by observing
be calculated as a function of time. A wavelet trans- how brain lesions from disease or trauma related to loss
form may optionally be applied to these data to of function. Functional MRI studies now provide
regress out smooth features of the time-dependent detailed maps of neural activity, allowing identification
data (Meunier et al., 2009), but is unnecessary. The of modules from correlations of neural activity (see
correlation of these data among the different regions figure 12.1).
chen and deem: modularity, plasticity, and development of brain function 125
an experiment could challenge subjects with a cognitive
task during the fMRI data collection. Among other
parameters, the ability of a subject to solve the task will
depend on the modularity of correlations of neural
activity. Quantifying this ability as a function of modu-
larity would provide the fitness function for this particu-
lar task. Perhaps different tasks will have fitness functions
that peak at different values of modularity. It has been
speculated that fast, low-level, automatic cognitive pro-
cesses are modular, whereas high-level, effortful, con-
scious cognitive processes are less modular (Meunier et
al., 2010). This speculation is related to figure 12.2: at
short times, modular networks are more efficient, but
at long times, nonmodular networks will find better
solutions. It is possible that fast, low-level cognitive pro-
cesses are to the left on this figure, and slow, high-level
tasks are to the right. Measuring fitness as a function of
modularity for low-level and high-level tasks would
resolve this issue.
Modularity may be used as a biomarker for brain
function. For example, the neural activity of epileptic
patients is less modular than that of normal subjects
(Chavez et al., 2010). Epileptic patients had more con-
nections and less modular structure in their correla-
tions of neural activity. An interesting application
Figure 12.7 The Brodmann areas of the human cerebral suggested by anecdotal evidence (Leclercq et al., 2000)
cortex, onto which fMRI data are projected for further analy- would be to quantify extent of traumatic brain injury
sis. (From Dow.) Each Brodmann area is shaded a different (TBI) and to quantify effectiveness of rehabilitation
color. (See color plate 6.) treatments. This diagnostic quantifies the reduction of
structure in background brain activity due to TBI. The
architecture of functional networks extracted from TBI
patients is expected to differ from that of healthy sub-
We hypothesize that brains of young people are likely jects. We expect that TBI, as a debilitating condition,
under more selective pressure to quickly solve challeng- will reduce the natural modularity of the healthy brain.
ing problems than are brains of older people. This We propose modularity as a metric for evaluating extent
pressure induces the emergence of modularity in of TBI. This metric can be used for quantification of
human brain activity as children develop into adults success of therapeutic strategies such as administration
(Kashtan et al., 2007; Sun & Deem, 2007). That is, this of ibuprofen and as a feedback to the success of reha-
hypothesis implies neural brain activity of young adults bilitation programs.
should be more modular than that of children or older
adults, as suggested schematically in figure 12.2. Figures ACKNOWLEDGMENTS We thank Jessica Cantlon for providing
the fMRI, age, and IQ data. This research was supported by
12.3, 12.4, and 12.5 show fMRI data that provide support
the US National Institutes of Health under grant 1 R01 GM
for this hypothesis. In other words, modularity rises 100468–01.
during childhood, peaks in young adults, and declines
in older adults.
The ability of the brain to solve a challenge in a given
amount of time is termed fitness. Fitness quantifies REFERENCES
brain function, and fitness depends on modularity. The Bassett, D. S., Wymbs, N. F., Porter, M. A., Mucha, P. J.,
dependence of fitness on modularity changes with age, Carlson, J. M., & Grafton, S. T. (2011). Dynamic recon-
shown schematically in figure 12.6. The highest values figuration of human brain networks during learning. Proc
Natl Acad Sci USA, 108, 7641–7646.
of modularity are selected for in young adults. Bogarad, L. D., & Deem, M. W. (1999). A hierarchical
It would be interesting to set up experiments to deter- approach to protein molecular evolution. Proc Natl Acad Sci
mine fitness as a function of modularity. For example, USA, 96, 2591–2595.
chen and deem: modularity, plasticity, and development of brain function 127
Shirer, W. R., Ryali, S., Rykhlevskaia, E., Menon, V., & Tzourio-Mazoyer, N., Landeau, B., Papathanassiou, D.,
Greicius, M. D. (2012). Decoding subject-driven cognitive Crivello, F., Etard, O., Delcroix, N., … Joliot, M.
states with whole-brain connectivity patterns. Cereb Cortex, 22, (2002). Automated anatomical labeling of activations in
158–165. spm using a macroscopic anatomical parcellation of the
Simon, H. A. (1962). The architecture of complexity. Proc MNI MRI single-subject brain. NeuroImage, 15, 273–289.
Amer Phil Soc, 106, 467–482. Von Mering, C., Zdobnov, E. M., Sophia Tsoka, S., Cicca-
Spirin, V., & Mirny, L. A. (2003). Protein complexes and relli, F. D., Pereira-Leal, J. B., Ouzounis, C. A., & Bork,
functional modules in molecular networks. Proc Natl Acad P. (2003). Genome evolution reveals biochemical networks
Sci USA, 100, 12123–12128. and functional modules. Proc Natl Acad Sci USA, 100(21)
Sun, J., & Deem, M. W. (2007). Spontaneous emergence of 15428–15433.
modularity in a model of evolving individuals. Phys Rev Lett, Waddington, C. H. (1942). Canalization of development and
99, 228107. the inheritance of acquired characters. Nature, 150,
Talairach, J., & Tournoux, P. (1988). Co-planar stereotaxic 563–565.
atlas of the human brain. New York, NY: Thieme. Wagner, G. P. (1996). Homologues, natural kinds and the
Tinker, M. T., Bentall, G., & Estes, J. A. (2008). Food evolution of modularity. Integr Comp Biol, 36, 36–43.
limitation leads to behavioral diversification and dietary Wagner, G. P., & Altenberg, L. (1996). Complex adapta-
specialization in sea otters. Proc Natl Acad Sci USA, 105, tions and the evolution of evolvability. Evolution, 50,
560–565. 967–976.
ABSTRACT Here, we report research on the neuroplasticity of versus deficits are observed. We also propose likely
different subsystems within vision, audition, sensory integra- mechanisms contributing to these different profiles of
tion, language, and attention. In each section, we note differ- plasticity. A final section describes our studies testing
ent profiles of plasticity observed in different subsystems
within the domain, situations in which enhancements versus the hypothesis, raised by this basic research on human
deficits are observed, and likely mechanisms contributing to neuroplasticity, that interventions that target the most
these different profiles of plasticity. A final section describes plastic, and thus potentially most vulnerable, neurocog-
our studies that test the hypothesis, raised by this basic nitive systems can protect and enhance children with,
research on human neuroplasticity, that interventions that or at risk for, developmental deficits.
target the most plastic, and thus potentially vulnerable, neu-
rocognitive systems can protect and enhance children with,
or at risk for, developmental deficits. Vision
The primate visual system contains multiple distinct
For several years we have employed psychophysics, elec-
visual areas, with a gross segregation of cortical visual
trophysiological, and magnetic resonance imaging
processing into dorsal and ventral streams that project
(MRI) techniques to study the development and neu-
respectively from V1 toward posterior parietal cortex
roplasticity of the human brain. We have studied deaf
and ventrally toward the temporal cortex (Grill-Spector
and blind individuals, people who learned their first or
& Malach, 2004; Ungerleider & Haxby, 1994). In our
second spoken or signed language at different ages,
research, we observe that aspects of vision and attention
and children of different ages and stages of cognitive
mediated by the dorsal pathway—including motion
development. Here, we review our research on the neu-
perception, peripheral vision, and selective attention—
roplasticity of different brain systems and subsystems.
are most modifiable with altered experience, showing
As detailed in the sections that follow, in each of the
both enhancements and deficits in special populations.
domains examined in this research we observe the fol-
In one line of research, we examined changes in
lowing characteristics:
visual processing in congenitally, genetically deaf adults.
• Different brain systems and subsystems and related
By virtue of an increased reliance on vision, we hypoth-
sensory and cognitive abilities display different degrees esized that deaf adults would exhibit enhancements in
and time periods (“profiles”) of neuroplasticity. visual processing. Indeed, our studies revealed improve-
• Neuroplasticity within a system acts as a double-
ments in some—but not all—aspects of vision in these
edged sword, conferring the possibility for either deaf adults. For example, congenitally deaf individuals
enhancement or deficit. have superior motion detection for peripheral visual
• Multiple mechanisms both support and constrain
stimuli than hearing individuals (Neville & Lawson,
the ability to modify different brain systems and subsys- 1987b; Neville, Schmidt, & Kutas, 1983; Stevens &
tems, with the most malleable systems generally display- Neville, 2006). These improvements are accompanied
ing longer developmental trajectories, higher levels of by increases in the amplitudes of early visual event-
redundant connectivity, and a greater concentration of related potentials (ERPs) for peripheral visual stimuli,
neurochemicals, including CAT301 and BDNF, which as well as increased functional magnetic resonance
are both important in neuroplasticity. imaging (fMRI) activation in motion-sensitive middle
In the sections that follow, we describe our research temporal (MT) and middle superior temporal (MST)
on neuroplasticity within visual, auditory, sensory inte- areas (Armstrong, Hillyard, Neville, & Mitchell, 2002;
gration, language, and attention systems. In each section, Bavelier et al., 2001; Bavelier et al., 2000; Neville &
we note different profiles of plasticity that are system- Lawson, 1987b; Neville et al., 1983). In contrast, we
and context-dependent and in which enhancements observe no differences between deaf and hearing adults
well, the neural response to known and unknown words of language shows a strong degree of biological invari-
and to syntactic anomalies is more strongly predicted ance, and several neural subsystems important in written
by a child’s language proficiency than by chronological and spoken language have also been observed when
age (Adamson, Mills, Appelbaum, & Neville, 1998; deaf and hearing native signers process American Sign
Adamson-Harris, Mills, & Neville, 2000; Mills, Coffey- Language (ASL), despite the fact that the eliciting
Corina, & Neville, 1993, 1997). These data suggest that sensory stimuli are very different (Capek, 2004; Capek
both age of acquisition and language proficiency affect et al., 2009; MacSweeney, Capek, Campbell, & Woll,
the neural systems used during language processing. 2009). However, it should be noted that although
To address whether altered neural organization in spoken and signed language processing share a number
late bilinguals is due to age of acquisition versus inter- of modality-independent neural substrates, there is also
ference of a first-learned language, we have studied a some degree of specialization based on language modal-
unique group of individuals: deaf signers who acquired ity. The processing of ASL, for example, is associated
sign language late in life. Many deaf children are born with additional and/or greater recruitment of right-
to hearing parents and, due to their limited access to hemisphere structures, perhaps owing to the use of
spoken language, do not have full access to a first lan- spatial location and motion in syntactic processing in
guage until exposed to a signed language, which often ASL (Capek et al., 2004, 2009; MacSweeney et al., 2009;
occurs very late in development. This provides an Neville et al., 1998). In support of this hypothesis, we
opportunity to study the neural systems underlying have shown that syntactic violations in ASL elicit a more
delayed first language acquisition, since the neurobiology bilateral anterior negativity for violations of spatial
P4
P4
2 µV –
Figure 13.4 Effects of selective attention on neural process- in the effects of attention on neural processing from pre- to
ing in children aged 3–5 years from lower socioeconomic post-training. Children in the PCMC-A program also made
status backgrounds, both before and after an eight-week train- significantly greater gains than either comparison group on
ing period. Only children in the family-based program with a standardized measures of receptive language and nonverbal
greater emphasis on parent training (“PCMC-A”) showed an IQ, as well as parent reports of child behavior. See main text
increase in the effects of attention on neural processing. Chil- for details. (Data from Neville et al., 2013.) (See color
dren in Head Start alone (“HS-alone”) or in a comparison plate 9.)
training program (“ABC”) did not show significant changes
transforms the results of basic research on the profiles potentials. Paper presented at the Cognitive Neuroscience
and mechanisms of neuroplasticity into the develop- Society.
ment of training and intervention programs. Our Armstrong, B., Hillyard, S. A., Neville, H. J., & Mitchell,
T. V. (2002). Auditory deprivation affects processing of
growing understanding of the limits and mechanisms motion, but not color. Brain Res Cogn Brain Res, 14(3),
of plasticity contributes to a basic understanding of 422–434.
human brain development and function and can also Astheimer, L., & Sanders, L. (2009). Listeners modulate
inform and guide efforts to harness neuroplasticity temporally selective attention during natural speech pro-
both to optimize and to protect the malleable and vul- cessing. Biol Psych, 80, 23–34.
Atkinson, J. (1991). Review of human visual development:
nerable aspects of human development. Crowding and dyslexia. In J. Cronly-Dillon & J. Stein (Eds.),
Vision and visual dysfunction (Vol. 13, pp. 44–57). Cam-
ACKNOWLEDGMENTS We thank our many collaborators in the
bridge, MA: MIT Press.
research reported here, supported by grants from NIH Atkinson, J. (1992). Early visual development: Differential
NIDCD (R01 DC000481, R01 DC000128) and Department functioning of parvocellular and magnocellular pathways.
of Education IES (R305B070018, R305A110397, and Eye, 6, 129–135.
R305A110398). Atkinson, J., King, J., Braddick, O., Nokes, L., Anker, S., &
Braddick, F. (1997). A specific deficit of dorsal stream
function in Williams’ syndrome. NeuroReport, 8(8),
REFERENCES
1919–1922.
Adamson, A., Mills, D., Appelbaum, G., & Neville, H. Auer, E. T., Jr, Bernstein, L., Sungkarat, W., & Singh, M.
(1998). Auditory sentence processing in adults and chil- (2007). Vibrotactile activation of the auditory cortices in
dren: Evidence from ERPs. Paper presented at the Cogni- deaf versus hearing adults. NeuroReport, 18, 645–648.
tive Neuroscience Society. Bakermans-Kranenberg, M., van Ijzendoorn, M. H.,
Adamson-Harris, A. M., Mills, D. L., & Neville, H. J. Pijlman, F. T. A., Mesman, J., & Femmie, J. (2008). Experi-
(2000). Children’s processing of grammatical and semantic mental evidence for differential susceptibility: Dopamine
information within sentences: Evidence from event-related D4 receptor polymorphism (DRD4 VNTR) moderates
ABSTRACT The auditory cortex is well known to show plasticity are over. Aging affects virtually all systems of the body,
in its functional organization in the adult as a consequence causing the central nervous system (which is also under-
of peripheral denervation, behavioral training, conditioning, going aging effects on its own) to adapt and reorga-
and other manipulations. It is also well established that
hearing deficits, both peripheral and central, are a common nize to accommodate these changes (e.g., Davis &
consequence of natural aging. What remains unclear is how Leathers, 1985; Gems & Partridge, 2013). For example,
these two phenomena are interrelated. This chapter will changes in the flexibility of joints and hypotrophy of
review the evidence for adult cortical plasticity in young adult the muscles will necessarily influence how cortical
animals and compare these functional changes that occur at motor circuits have to behave in order to produce the
younger ages with those seen at both the perceptual and
neural levels in a natural aging population.
same (or similar) motor functions (Ren, Wu, Chan, &
Yan, 2013). Nonneural effects of aging in sensory
systems will also dramatically affect central nervous
Adult mammals are continuously changing throughout system processing. A simple example includes presby-
their lifetime by adding new repertoires of behaviors opia, where aging causes the lens of the eye to become
while losing others. Nervous systems are also under a stiffer and less able to image near objects on the retina
constant state of change, which should be no surprise (Truscott, 2009). Similar affects can be noted for other
to those of us who attribute behaviors and perceptions sensory systems as well, all of which will require that the
to the nervous system. There are a myriad of studies on central nervous system in some way adapts to these
neural development, from the first formation of the changes to provide percepts that are as similar as pos-
neural tube to the final myelination of the frontal lobes. sible to those for the same stimuli at a younger stage in
Focus on the development of the cerebral cortex life.
includes neurogenesis, the development of polarity, Neural changes also occur in aging, the most popu-
synaptogenesis, migration, and critical periods of larly attributed to aging being Alzheimer’s and Parkin-
sensory perception (e.g., see Barnes & Polleux, 2009; son’s diseases. These are neurodegenerative diseases
Clowry, Molnar, & Rakic, 2010; Kanold & Luhmann, affecting the higher-order temporal and frontal cortical
2010; Levelt & Hubener, 2012). These studies have areas and the basal ganglion. However, there have been
given us a great perspective on how the nervous system far fewer studies of the effects of aging on unisensory
is made and what constraints are imposed during its areas, for example in primary sensory cortical areas A1,
development. Following development, adult cortical V1, and S1 (David-Jürgens, Churs, Berkefeld, Zepka, &
plasticity studies have shown that injury, training, or Dinse, 2008; David-Jürgens & Dinse, 2010; de Villers-
other means of altering peripheral input will reorganize Sidani et al., 2010; Engle & Recanzone, 2012; Juarez-
the functional representations of the sensory epithe- Salinas, Engle, Navarro, & Recanzone, 2010; Mendelson
lium in the cerebral cortex (e.g., see Feldman, 2009; & Ricketts, 2001; Recanzone, Engle, & Juarez-Salinas,
Irvine, Rajan, & McDermott, 2000; Pascual-Leone, 2011; Schmolesky, Wang, Pu, & Leventhal, 2000; Yang
Amedi, Fregni, & Merabet, 2005). These studies are et al., 2008; Yu, Wang, Li, Zhou, & Leventhal, 2006;
largely done in relatively young adult animals, and rep- Zhang et al., 2008). Any age-related effects seen at this
resent changes in the brain that occur over the course level must be at least partially due to the age-related
of weeks or months. As with developmental studies, differences in the periphery (hearing loss, cataracts,
these studies have also provided us with great insights etc.) but it remains a largely open question as to the
into the general function and constraints of the nervous extent to which the primary sensory areas are able to
system. compensate for these peripheral changes. What is also
One constant in the life of an animal, however, is that virtually unstudied is how natural aging influences the
it continuously ages once the developmental periods transfer of information throughout the cortical
low
X X
X X X X
X X X X
X X X X X X
high
Auditory cortex
1 2 3 4 5 1 2 3 4 5 1 2 3 4 5
Intensity
Figure 14.1 Cortical map changes after cochlear damage. there is an orderly and progressive shift of best frequency
(A) Schematic representation of the cochlea (top) with low across A1. (B) Results from lesions of the basal turn of the
frequencies represented at the apex (red) and progressively cochlea (high frequencies, denoted by X) and the subsequent
higher frequencies represented toward the basal end (blue). expansion of the representation of the frequencies at the
The middle panel shows a schematic of the primary auditory edge of the lesion, with little change in the representation of
cortex and the isofrequency bands where neurons have the other frequencies. (C) One possible outcome of high-
similar frequency tuning corresponding to the different frequency hearing loss as a consequence of natural aging,
cochlear locations. Black dots (numbered 1–5) denote record- where the representation of many frequencies are altered to
ing sites that correspond to the tuning functions of individual regain an orderly, topographic representation in A1. (See
neurons shown in the bottom row. In the normal animal, color plate 10.)
standard frequency increased, and this increase was well With natural aging, one of the common deficits
correlated with how well the individual animals per- noted is in “executive control” (Salthouse, 2012). Geri-
formed the perceptual task. An important finding was atric individuals show deficits in maintaining attention,
that the animals had to be attending to these stimuli, in allocating attention to one set of stimuli in the pres-
as attending to unrelated tactile stimulation resulted in ence of distractors, and in allocating attention as a func-
no such cortical plasticity (Recanzone et al., 1993). tion of the number of distractors (Li, Gratton, Fabiani,
Similarly in rats, basal forebrain stimulation, which & Knight, 2013; Fabiani, 2012). Each of these deficits
causes a release of acetylcholine in the auditory cortex would undoubtedly result in both a weakened percep-
in much the same way that attention does, also resulted tion of the relevant stimuli, but also in a decrease in the
in a broad increase in the representation of the paired potential for cortical plasticity mechanisms to enhance
frequency (Kilgard & Merzenich, 1998a). Subsequent the perception of those stimuli. Thus it may be difficult
studies showed that the cortical representation of the to disambiguate how much of a perceptual deficit is the
temporal aspects of acoustic stimuli were similarly result of bottom-up influences from the sensory periph-
plastic (Kilgard & Merzenich, 1998b; Kilgard et al., ery and top-down influence from decreased executive
2001). These microstimulation and operant condition- control. Again, both are likely to be in operation simul-
ing experiments indicate that attentional mechanisms, taneously, and given the unique location of the auditory
or “top-down” control, are critical for changes in per- cortex with multiple ascending auditory nuclei process-
ception and cortical plasticity, and suggest that atten- ing information in the brainstem, midbrain, and thala-
tional deficits will also strongly influence the magnitude mus, and the multiple feedback connections from the
and effectiveness of cortical plasticity mechanisms. frontal and parietal lobes, the primary auditory cortex
ABSTRACT A continuing challenge in neuroscience is elucidat- epigenetics then emerged as a leading candidate for a
ing complex relationships between nature and nurture. Both biological pathway linking gene-environment interac-
genetic polymorphisms and environmental stressors are rec- tions. Our current understanding is that epigenetic
ognized for their significant roles in producing behavioral
abnormalities and psychiatric disorders. The mechanisms by
alterations, acting either separately or in conjunction
which environmental factors may physically interact with gen- with genetic polymorphisms, serve as a risk factor
otype are likely varied, but increasing evidence argues for a responsible for long-term and even multigenerational
prominent role of DNA methylation. This chapter will focus trajectories in the development of behavior following
on the role that polymorphisms and epigenetic factors play stress and social adversity encountered early in develop-
in behavior.
ment. A central theme also explored here is that epi-
genetic regulation of genes is not a process exclusive to
For decades, psychologists and neuroscientists have developing organisms, but is an active process that
realized that early life experiences have far-reaching affects behavior throughout life into senescence. Such
effects on behavior. Stressful experiences and adverse observations are thus consistent with the notion that
social environments during sensitive periods of devel- epigenetics can provide a mechanism for gene-by-envi-
opment have been associated with a range of negative ronment interaction and behavior throughout the life
outcomes. These include structural changes in regions span.
such as the prefrontal cortex, hippocampus, amygdala,
and cerebellum and a high prevalence of anxiety and Contribution of genetic polymorphisms to behavior
depression (Cicchetti & Toth, 2005; Hanson, Chandra,
Wolfe, & Pollak, 2011; Tottenham et al., 2010). Nature- Common types of functional polymorphisms studied in
versus-nurture questions have long plagued scientists the context of behavior are single nucleotide polymor-
seeking to understand mechanisms responsible for phisms, or SNPs, in which a single nucleotide within the
behavioral development and psychiatric disorders. Is it DNA sequence is altered, and short tandem repeats, or
an individual’s genetic makeup, exposure to certain STRs, which are sections of DNA with repeating pat-
environmental factors, or a gene-by-environment inter- terns of short sequences (figure 15.1A). The presence
action that best predicts behavioral trajectories? of polymorphisms often produces changes in gene tran-
After completion of the Human Genome Project in scription, conferring either higher or lower levels of
the early 2000s, investigators realized that the human gene expression. A widely studied polymorphism that
genome did not possess enough genes to fully codes for the serotonin transporter 5-HTT is well rec-
explain the vast complexities of behavior. At the same ognized for its capacity to influence stress responsivity
time, studies were emerging showing that not all indi- and mood. Encoded by the gene SLC6A4 in humans,
viduals who encounter stressful events succumb to 5-HTT is responsible for removal and reuptake of sero-
depression or anxiety. Instead, data suggested that tonin. 5HTTLPR (see table 15.1 for abbreviations and
certain individuals may be more vulnerable to stress definitions commonly used in this chapter) is a poly-
and adversity because of so-called genetic predisposi- morphic region that has a variable number of tandem
tions. Such observations led us to the realization that repeats yielding either short or long alleles. The short
there is a gene-by-environment interaction influencing allele has reduced transcriptional efficiency compared
the development of behavior. With the birth of behav- with the long allele, ultimately leading to reduced sero-
ioral epigenetics research in the mid-to-late 2000s, tonin transporter in membranes.
Neuroimaging studies have revealed that healthy sub- Specifically, individuals with one or two copies of the
jects with at least one short allele have increased tem- short allele exhibited more depressive symptoms in
peramental anxiety and amygdala reactivity (Pezawas et relation to stressful life events, especially child maltreat-
al., 2005). Work with nonhuman primates and humans ment (Caspi et al., 2003).
provide evidence for an interaction between 5HTTLPR The gene encoding the brain-derived neurotrophic
genotype and lifetime stressors, especially early life factor protein (BDNF) has another commonly studied
stress, on behavioral and emotional outcomes. For polymorphism. A polymorphism in the Bdnf gene
example, rhesus monkeys with the short allele that were causes an amino acid substitution (driven by a G/A
deprived of maternal care showed deficiencies in sero- substitution) of valine to methionine at codon 66 (Val-
tonergic functioning, altered stress reactivity, and aber- 66Met), which ultimately leads to reduced secretion of
rant emotional behavior (Barr et al., 2004; Bennett et BDNF. Because BDNF is crucial for neural development
al., 2002). In humans, the interaction of early and central nervous system plasticity, alterations in its
life adversity and 5HTTLPR genotype was pioneered by secretion have been well linked to numerous cognitive
Caspi, who found that 5HTTLPR genotype moderated deficits and neuropsychiatric disorders (Bath & Lee,
the influence of stressful life events on depression. 2006). Independent of environmental factors, the Met
# %"
CREB1
Single nucleotide polymorphism (SNP) “Turn on”
MeCP2 CH3
gene Co- C G
activator
C A G T C
G C
T A
C G G T C
CH3 C G
G C
C G
HDAC
G C CH3 MeCP2 “Turn off”
Co-
Short tandem repeat (STR) repressor gene
A T
C T C T C T C T C T
G C
T A
C G
3′
5′
Figure 15.1 Schematic of common gene polymorphisms cytosine residues. MeCP2 binds to methylated DNA and can
and DNA methylation. (A) Single nucleotide polymorphisms either recruit HDACs and other co-repressors to suppress
(SNPs) consist of a single nucleotide variation between indi- gene transcription or can recruit cAMP response element-
viduals. Short tandem repeats (STRs) contain multiple repeats binding (CREB) protein and other transcriptional co-activa-
of a certain short sequence of nucleotides. (B) DNA methyla- tors to enhance gene expression. (See color plate 11.)
tion refers to the addition of methyl groups to specific
allele has been associated with poor episodic memory sex-specific manner in this task (Shalev et al., 2009).
and abnormal hippocampal activation (Egan et al., Specifically, Val/Met females showed greater increases in
2003). An important study from B. J. Casey’s lab showed salivary cortisol during the test than Val/Val homozy-
that the Val66Met polymorphism is biologically and gotes, while Val/Val males showed a higher cortisol
behaviorally relevant across species, since both humans response than heterozygotes.
and genetic knock-in mice showed abnormal neural Although 5HTTLPR and Val66Met polymorphisms
activity and significant impairments in fear memory are most commonly investigated, many other variations
extinction (Soliman et al., 2010). confer specific behavioral phenotypes either alone or
The Bdnf polymorphism also moderates the effect of in interaction with environmental factors. For example,
early or later-life environmental factors. In carriers of FKBP5 is a gene that controls HPA axis activity through
the Met allele, early life stress predicts higher levels of regulation of glucocorticoid receptors (GR), and
depression and anxiety that are linked to loss of pre- various FKBP5 SNPs have been recently linked to
frontal gray matter and reduction in hippocampal anxiety and depression following early life stress
volume (Gatt et al., 2009). The Trier Social Stress Test (Binder et al., 2008; Gillespie, Phifer, Bradley, &
(TSST), in which participants have to perform a speech Ressler, 2009). It is likely that having multiple polymor-
and do mathematical problems in front of a panel of phisms in combination with specific environmental
“interviewers,” is an effective psychosocial stress regimen events confers an even greater moderating conse-
commonly used in humans to explore the significance quence in terms of neural and behavioral outcomes.
of polymorphisms. The presence of the Bdnf Val/Met Indeed, such gene-by-gene interactions (having the
polymorphism has been shown to modulate hypotha- Met allele of the Bdnf gene and two short alleles for
lamic-pituitary-adrenal (HPA) axis reactivity in a 5HTTLPR) in conveying greater vulnerability to
ABSTRACT The nervous system is limited in its ability to process When your teacher suggested that you “pay atten-
stimuli and information. Attention is a term covering the many tion,” we might say that she was primarily addressing
ways in which the system can selectively process a useful subset your alerting mechanisms: those processes that allow
of all that could be processed. This chapter focuses on the
behavioral manifestations of attention, specifically in visual
you to be ready to sustain your sensitivity to incoming
processing. Subsequent chapters will describe the neural basis stimuli. Vigilance studies, in which an observer might
for some of these aspects of attention. be asked to simply push a key each time an infrequent
light appears, are measures of the decline or lapse of
In a text on cognitive neuroscience, there is going to alertness over time (Warm, Dember, & Hancock, 1996).
be a heavy emphasis on those aspects of attention that When your friend in that class suggested that you “look
are amenable to study with electrophysiology and neu- here,” he was orienting your attention: inducing you to
roimaging. In focusing on that subset of the field, one select a subset of the currently available sensory stimuli.
can lose track of the broader sweep of attentional phe- A vast body of research makes use of Posner’s cuing
nomena—the behaviors that got us interested in the paradigm to study orienting of attention. In its simplest
neural underpinnings of attention in the first place. form, observers are asked to detect a stimulus that can
The purpose of this chapter is to briefly review behav- appear at one of two locations. Prior to the stimulus, a
ioral manifestations of attention and some of the theo- cue directs attention to one or the other location. After
retical constructs that have been used to organize our a delay, the stimulus is presented at the cued or uncued
understanding of those behaviors.1 location. A benefit in speed, accuracy, or both at the
cued location or a penalty, relative to a baseline, at the
Posner’s taxonomy: Alerting, orienting, and uncued location is evidence for the selective orienting
executive function of attention. Simple cueing and probe-detection para-
digms are readily adapted to animal and neuroimaging
“Attention” refers to a wide range of mechanisms in work.
the mind/brain that help us deal with our inability to The executive function aspect of attention refers to
do everything at the same time. There are fundamen- those processes that serve to monitor our ongoing flow
tal bottlenecks in processing, and there are attentional of thoughts, reactions, and responses. These processes
mechanisms that control access and navigation for are of particular use in resolving conflicting demands.
many of these. At least since William James, it has been The tumbling kittens and babies at the top of a web
clear that attention is not A Single Thing. There have page are trying to summon your attention by virtue of
been many efforts to organize this broad category of their high perceptual salience (Yantis, 1993), but you
different processes and, indeed, one way to do this has wish to deploy top-down control in order to read the
been to look for anatomical or physiological networks turgid scientific prose elsewhere on the page. Executive
that seem to be devoted to one or the other class of processes mediate that conflict. Researchers have devel-
attentional process (see chapter 19, this volume). One oped a number of laboratory tasks to study these pro-
useful taxonomy grows out of the work of Michael cesses. For instance, in “stop” signal tasks, you must try
Posner (Posner & Cohen, 1984). He divides the terri- to countermand an order or plan to do something
tory into three networks: alerting, orienting, and exec- when a second command tells you to stop. For instance,
utive function. you might be asked to push a button when a light
appears but to stop or withhold that response if that
1
This chapter is lightly referenced because of space light turns red (Schall & Godlove, 2012). In anti-sac-
considerations. A more fulsomely referenced version can be cade tasks, a signal at one location tells you to make a
obtained from [email protected]. saccadic eye movement to a different location, again
Figure 16.2 Three central cues: Eyes and arrows behave more like exogenous cues than like endogenous, symbolic cues.
ABSTRACT Computational theories suggest that attention as well as occipital, temporal, parietal, and frontal
shapes the topographical landscape of priority maps in visual cortex. The activity level of neural representations that
areas extending from the lateral geniculate nucleus to pre- code each position in a given map can be modulated
frontal cortex. Here we review evidence suggesting that top-
down deployments of attention to spatial locations and to either via foreknowledge of an important spatial loca-
behaviorally relevant simple (e.g., orientations, colors) and tion (space-based attention) or by foreknowledge of the
complex (e.g., categorical object information) features can relevant visual features that define an object (e.g.,
jointly influence the topography of attentional priority maps. attend to yellow objects when searching for a NYC taxi
These topographic representations appear to integrate mul- cab, termed feature- and object-based attention). Here we
tiple sources of information to efficiently represent sensory
will review functional magnetic resonance (fMRI)
stimuli and to guide motor interactions with important objects
in the environment. studies that highlight the flexible and adaptive nature
of attentional modulation of neural activity across the
Selective attention is the mechanism by which an organ- visual hierarchy (see chapters 18 and 20 by Hillyard &
ism prioritizes the processing of relevant sensory inputs Luck and by Treue & Martinez-Trujillo, this volume, for
at the expense of irrelevant distractors. The need for related reviews that focus on monkey and human elec-
selective sensory processing arises for a variety of reasons trophysiology, respectively).
(Tsotsos, 1991). First, the neurons that encode environ-
mental stimuli are highly variable, such that the same The influence of spatial attention on attentional
physical stimulus will evoke a slightly different response priority maps
each time it is presented. In turn, this variability in
neural response patterns limits the amount of informa- Although the existence of visual maps in human cortex
tion that neurons can encode about basic stimulus was established nearly 100 years ago (Holmes, 1918), it
features (Pouget, Dayan, & Zemel, 2003; Seung & took until the advent of functional magnetic resonance
Sompolinsky, 1993). Second, multiple items in the imaging (fMRI) in the early 1990s for researchers to
visual field compete for representation, and this com- reliably define maps on a whole-brain basis. Mapping
petition must be resolved so that the most behaviorally the visual system also opened the door to the systematic
relevant sensory stimuli are represented robustly to exploration of the influence of attentional factors on
guide goal-directed behavior (Desimone & Duncan, the topographic representations maintained in each
1995). Collectively, these factors restrict the speed and visual area. In the earliest investigations, a key question
accuracy of perceptual decisions and place an upper was: How early in the visual system does attention have
limit on our ability to interact with objects in the an effect on spatially selective representations? In a pio-
environment. neering set of studies, several groups independently
Computational theories propose that attention mod- established that attention to the location of a moving
ulates the topographical landscape of spatial maps in stimulus (Gandhi, Heeger, & Boynton, 1999; Somers,
the visual system such that the location of the most Dale, Seiffert, & Tootell, 1999) and to a stationary stim-
important object is associated with the highest activa- ulus (Martinez et al., 1999) reliably modulated activa-
tion level, and the activation level of irrelevant objects tion levels in primary visual cortex, or area V1. This
declines in descending order of importance. Recent result, which has subsequently been replicated many
studies suggest that these attentional priority maps of times, provided the most compelling evidence to date
visual space are ubiquitous and span subcortical areas, that attention could operate on the earliest cortical
14s
0.5s
0.75
B * ***
Asymptotic accuracy
0.6
0.55
0.5
0.45
V1 V2v V3v V4v V3a MT IPS FEF
Visual Area
Figure 17.3 Feature- and object category–based attention by viewing intact vs. scrambled objects) in a group-average
effects. (A) Top panel: Sequence of events on a trial where the analysis at P < 0.005 (Talairach coordinates of peak: x = 35,
observer was attending to 45° motion in the right stimulus y = −41, z = −18; N = 10). The lower panel shows category
aperture. One-half of the dots in each stimulus aperture information as a function of category, task, and attention in
moved at 45°, and the other half moved at 135°, for the dura- individually defined object-selective cortex. Category informa-
tion of the 14 sec presentation period. Targets were defined tion was calculated by taking the difference between within-
as a brief slowing of the dots at the attended location that category comparisons and between-category comparisons,
moved in the currently attended direction (45° in this figure); and reflects the amount of category information in multi-
distractors were defined as a brief slowing of the dots at the voxel patterns of activation. Significant category information
attended location that moved in the unattended direction depended solely on task instruction and not on spatially
(135° in this figure). Bottom-left panel: Asymptotic multi-voxel attended location. Task-relevant information was processed to
classification accuracy for the attended direction of motion the categorical level, even when unattended, whereas task-
based on fMRI activation patterns in each visual region (error irrelevant information was not represented at that level, even
bars, ±SEM across observers). (Data adapted and reprinted when attended. Error bars indicate ± SEM. (Data adapted and
with permission from Serences and Boynton, 2007.) (B) Top reprinted with permission from Peelen et al., 2009.) (See
panel: Clusters of object-selective activations in ventral tempo- color plate 14.)
ral cortex (as determined by contrasting activations evoked
demonstrated that regions of IPS and PCC encode with the spatial attention system. Indeed, jointly direct-
information about the specific direction of an attended ing attention to locations and to simple and complex
motion stimulus, suggesting that these regions may play features—to the extent supported by top-down knowl-
a role in generating the biasing signals observed in early edge—is a highly efficient way to rapidly hone in on the
visual areas. Liu et al. (2011) extended these findings most relevant objects in the visual scene. The combined
by showing that areas of IPS and PCC also encode spe- information provided by foreknowledge of relevant
cific colors (as well as directions of motion). In addi- visual information is likely combined in higher-order
tion, Liu et al. (2011) demonstrated that partially topographic maps that are found in subregions of IPS
separable regions of IPS and PCC carry information and prefrontal cortex, where the landscape of topo-
about attended directions and colors, suggesting graded graphical representations is partially divorced from the
specialization or sensitivity in the putative priority maps exact task or motor plan that initially induced atten-
that are thought to be maintained in these regions. tional priority (Jerde & Curtis, 2013; Jerde et al., 2012).
In the future, a major challenge will be to form a
Conclusion more integrated understanding of the role that each
region in the visual system plays in building an increas-
Together, the rapid implementation of feature- and ingly selective and refined representation of relevant
object category–based attentional shifts and their mod- stimuli in the environment. For instance, neurons in
ulatory impact on the visual system raise many parallels each area of the visual system differentially encode
ABSTRACT Human vision unfolds rapidly—with dozens of dis- questions. First, ERPs provide a continuous, millisec-
tinct processes occurring within a half-second of the onset of ond-resolution measure of processing between a stimu-
a stimulus—and attention influences many of these processes. lus and a response (as well as prior to the stimulus and
The event-related potential (ERP) technique is particularly
well suited for measuring these rapidly varying attentional
after the response). This makes it possible to track shifts
processes because of its millisecond-level temporal resolution. of attention as they occur and to determine the stage
Recent ERP studies of attention have been used to study two at which attention influences the processing of incom-
major topics in attention, namely, the mechanisms by which ing sensory information. Second, ERPs make it possible
attention is directed to specific types of stimuli and the con- to “covertly” monitor the processing of a stimulus in the
sequences of attention on the processing of incoming sensory
absence of a behavioral response. This is especially valu-
information. In particular, recent research on attentional
control shows that salient stimuli are always detected, even if able in attention research, because it is difficult to
task-irrelevant, but can be prevented from capturing attention obtain a behavioral response to a stimulus that is truly
by top-down suppression mechanisms. Once attention has unattended.
been captured by a stimulus, these same suppression mecha- The temporal resolution of ERPs also makes it pos-
nisms may be used to terminate attention when the percep-
sible to answer questions that cannot be addressed by
tion of the attended object is complete. In addition, recent
studies of the effects of attention on sensory processing have techniques that are limited by the slow time course of
focused on a subset of ERPs called steady-state visual evoked the hemodynamic response, such as functional mag-
potentials (SSVEPs), which are oscillating potentials induced netic resonance imaging (fMRI). Many different atten-
by flickering stimuli. By tagging individual stimuli with differ- tion-related processes may occur within a given brain
ent flicker frequencies, SSVEPs make it possible to assess the area within the first half-second after the onset of a
effects of attention on sensory processing for multiple simul-
taneous stimuli. Recent SSVEP studies have shown that the stimulus, and it is difficult to pull these processes apart
“spotlight” of attention can be divided among multiple dis- with fMRI. In addition, fMRI typically mixes together
continuous locations and have delineated the speed at which brain activity elicited by attended and unattended
attention can be shifted. stimuli, making it difficult to assess the implementation
of selection with fMRI. In contrast, the ERPs elicited by
attended and unattended stimuli can easily be recorded
The role of event-related potentials in studying and measured separately, as long as these stimuli are
attention separated by a few hundred milliseconds.
A special class of ERPs elicited by repetitively flashing
Studies of visual selective attention are typically focused or flickering stimuli has proven to be particularly useful
on understanding two broad questions (see reviews by for studying the implementation of visual selection.
Luck & Gold, 2008, and Luck & Vecera, 2002). First, These so-called steady-state visual evoked potentials
how does the brain determine which sources of input (SSVEPs) are continuous oscillatory neural responses
should be attended? This is called the control of attention. elicited in the visual cortex at the same fundamental
Second, once attention has been focused on a given frequency as the flickering stimulus. If several stimuli
source of information, how does this change the pro- are presented simultaneously, but each is flickering at
cessing of the attended and unattended information? a different frequency, the brain activity for each stimu-
This is called the implementation of selection. lus can be separated by extracting oscillations at the
Event-related potentials (ERPs) have played a key specific frequencies of stimulation. This is called fre-
role in addressing both of these questions for many quency tagging the stimuli, and it makes it possible to
years (see reviews by Hillyard, Vogel, & Luck, 1998; separately measure sensory activity to attended and
Luck & Kappenman, 2012). ERPs have two main advan- ignored stimuli that are presented simultaneously
tages over behavioral measures for addressing these but flicker at different frequencies. Frequency-tagged
Green
N2pc
N1
LVF target LVF target Contralateral target
RVF target RVF target Ipsilateral target
Contra minus ipsi
+2 µV
Figure 18.1 Typical N2pc paradigm and results. Each stimu- gap in the target item is on the top or the bottom of the
lus array contains a distinctly colored item on each side, and square. The N2pc component is defined as the difference
one of these two colors is designated as the target color in the between the contralateral and ipsilateral waveforms (shown
instruction screen for each trial block. The locations of the as the shaded region), which can be isolated by constructing
items varied at random from trial to trial, except that the two a contralateral-minus-ipsilateral difference wave. (Adapted
pop-out colors are always on opposite sides. The subject is with permission from Luck, 2014.)
required to press one of two buttons to indicate whether the
A B C
Line is target Diamond is target
Diamond is distractor Line is distractor
PD
+2 µV +2 µV
N2pc
100 200 300 400 500 100 200 300 400 500
–2 µV –2 µV
Contralateral Ipsilateral
Contra minus ipsi
Figure 18.2 Example stimuli and results from the study of (N2pc) was elicited by the red item. When the green diamond
Hickey et al. (2009). Each array (A) contained a bright green was the target (C), the ERP was more positive contralateral to
square or diamond along with a short or long red line that the red distractor item than ipsilateral to this item. This con-
was isoluminant with the background. The red stimulus shows tralateral positivity is called the distractor positivity (PD).
almost no lateralized sensory activity, making it possible to Waveforms courtesy of John McDonald. (Adapted with per-
attribute any later lateralized activity to top-down processing. mission from Luck, 2012.)
When the red line was the target (B), a contralateral negativity
contralateral-minus-ipsilateral difference wave cannot N2pc onset latency can also be used to study indi-
deviate from zero microvolts until the brain has detected vidual and group differences in the speed of attentional
and localized the to-be-attended item and begun to control. For example, N2pc onset latency is increased
shift attention toward it. Thus, N2pc onset latency pro- in older individuals (Lorenzo-Lopez, Amenedo, &
vides a means of tracking the time course of attentional Cadaveira, 2008) and in patients with hepatic encepha-
control. Not surprisingly, N2pc onset latency varies with lopathy (Schiff et al., 2006). In contrast, no change in
the salience of the target and is earlier for targets that N2pc amplitude or latency has been observed in patients
are associated with large rewards (Kiss, Driver, & Eimer, with schizophrenia (Luck et al., 2006) or Parkinson’s
2009). disease (Praamstra & Plat, 2001).
P1
+1 µV
L standard Ignored
100 ms
–100
C1 100 200
Blank
100–300 ms –1 µV
Attended
R standard
N1
100 ms
Blank
100–300 ms
R standard
TI 100 ms
m
e
Blank
100–300 ms
L target
100 ms
Figure 18.4 Prototypical paradigm and results for using when it is attended and when it is ignored. The P1 and N1
ERPs to assess the effects of spatial attention on sensory pro- components are typically found to be increased in amplitude
cessing. Subjects attend to the left side of the display in some for attended-location stimuli. (Adapted with permission from
trial blocks and to the right side on others, making it possible Luck & Kappenman, 2012.)
to examine the ERP elicited by a stimulus in a given location
Figure 18.5 Schematic diagram of stimulus array and SSVEP of color changes. SSVEP waveforms shown were averaged in
waveforms from one subject in the study of Müller et al. the time domain using a moving window average over the
(1998). On each trial, subjects were cued to attend to the left interval 1–3 sec after cue onset. (Reprinted with permission
or right array of flickering LEDs and detect a target pattern from Müller, Teder-Sälejärvi, & Hillyard, 1998.)
in striate cortex, but this likely reflects late feedback within an array of flickering stimuli (Vialatte, Maurice,
signals rather than a feedforward modulation of incom- Dauwels, & Cichocki, 2010). SSVEP amplitude modula-
ing sensory signals (Martinez et al., 1999). tions have also been used to show that improvements
SSVEPs elicited by flickering stimuli at multiple loca- in visual attention seen in fast-action video game players
tions in the visual field are strongly modulated by the were associated with a more effective suppression of
allocation of spatial attention among those locations irrelevant competing inputs (Krishnan, Kang, Sperling,
(reviewed in Andersen et al., 2012). Typically, SSVEPs & Srinivasan, 2012; Mishra, Zinni, Bavelier, & Hillyard,
to stimuli within the spotlight of attention show 2011).
enhanced amplitudes in early visual-cortical areas rela-
tive to stimuli outside the spotlight (figure 18.5). When
flickering stimuli are presented at two spatially sepa- Feature-Based Attention In a typical ERP study of
rated locations that need to be attended simultane- feature-selective attention, a series of stimuli is pre-
ously, the spotlight can be effectively divided in two for sented at a single location, and the subject is asked to
periods of at least several seconds, as evidenced by attend only to the stimuli that possess a specific feature
SSVEPs being enhanced to attended-location stimuli (e.g., the color blue). Under these conditions, the P1
but not to intermediate, unattended stimuli (Müller and N1 waves are not any larger for stimuli containing
et al., 2003). Such a sustained splitting of the atten- the attended feature than for stimuli that lack this
tional spotlight is not readily achieved when stimuli are feature (Anllo-Vento & Hillyard, 1996). This suggests
flashed transiently at the attended locations, suggesting that selection based on nonspatial features occurs at a
that the human visual system can only divide attention later stage of processing than selection based on spatial
effectively between separated locations when stimuli are location. However, recent studies have shown that color-
continuously present (or flickering rapidly so that they based attention can influence P1 amplitude under
appear to be continuous), as is usually the case in the certain conditions, operating at the same stage as spatial
natural environment. The sensitivity of SSVEPs to attention (Zhang & Luck, 2009). In these studies, the
spatial attention has led to the development of brain- attended and ignored colors were present continuously
computer interfaces whereby the intentions of an and were spatially overlapping, which presumably maxi-
immobile subject can be conveyed via amplitude modu- mized the competition between them, and the ERPs
lations of the SSVEP produced by directing attention were elicited by transient probe events.
ABSTRACT Spatial neglect is a common syndrome after brain right-hemisphere lesions can cause neglect, only right-
injury, characterized by spatial and nonspatial attention, and hemisphere lesions cause severe and persistent deficits
perceptual and motor deficits. These deficits are due to the (Stone, Halligan, & Greenwood, 1993). This still-unex-
structural and functional damage of attention networks, and
their interaction with sensory and motor regions of the brain.
plained hemispheric asymmetry after nearly 70 years of
Accordingly, white matter damage is associated with more research is one of the primary reasons for the widely
severe neglect due to the disconnection of different interact- held view that the right hemisphere is dominant for
ing regions. The right hemisphere lateralization of neglect attention. Finally, spatial neglect is unique among the
is due to the hemispheric lateralization of arousal/vigilance. behavioral disorders resulting from focal lesions, since
Spatial neglect is an outstanding model to study how large-
its severity can be modulated by behavioral interven-
scale neural systems mediate behavior.
tions over very short timescales (e.g., seconds to
Spatial neglect is a common syndrome that follows focal minutes). Deficits in attending to and reporting objects
brain injury (stroke, tumor, etc.). It is defined as the in contralesional space can be lessened by (1) encour-
inability to attend and report stimuli on the side oppo- aging a patient to attend to the previously ignored
site the lesion (or contralesional) despite apparently stimuli using verbal cues; (2) presenting salient sensory
normal visual perception. Neglect is also frequently stimuli, such as noises (Robertson, Mattingley, Rorden,
associated with a reduction of arousal and vigilance, a & Driver, 1998); (3) asking the patient to perform hand
spatial bias for directing actions toward the hemi-space movements controlled by the injured hemisphere (Rob-
or hemi-body on the same side as the lesion (or ipsile- ertson & North, 1992); or (4) training patients to
sional), and several disorders of awareness, including a increase their alertness (Robertson et al., 1995). These
degree of obliviousness toward being ill and confabula- observations suggest that the neural mechanisms under-
tion about body ownership. lying the spatial deficit can be dynamically modulated
About 25–30% of all stroke patients suffer from this by signals from other parts of the brain reflecting
syndrome, an estimated 250,000–300,000 patients per endogenous or exogenous attention, movement, and
year. Clinically, spatial neglect negatively impacts motor arousal. Moreover, in a matter of days or weeks, most
recovery and outcome. Stroke patients with neglect are patients with spatial neglect recover from the more
less likely to go back to work, drive, and be independent obvious spatial impairments, which, however, continue
than stroke patients with a comparable degree of motor to negatively influence their ability to return to a pro-
disability but without spatial neglect. ductive life (Denes, Semenza, Stoppa, & Lis, 1982;
Spatial neglect has attracted tremendous interest as Paolucci, Antonucci, Grasso, & Pizzamiglio, 2001).
a model for understanding the neurological basis of To reconcile these observations, we have proposed
awareness, cerebral lateralization, spatial cognition, that neglect is mediated by the abnormal interaction
and recovery of function. Yet its neural bases remain between brain networks that control attention to the
poorly understood. environment in the healthy brain (Corbetta & Shulman,
There are several outstanding issues that are of great 2011; see Heilman, Watson, & Valenstein, 1985, and
interest for cognitive neuroscience. First, even though Mesulam, 1999, for other network formulations). We
strokes causing neglect occur in different locations in argue for a core set of spatial and nonspatial deficits
the brain, both cortically and subcortically, the syn- that match the physiological properties of these net-
drome is relatively stereotypical. This suggests that works. First, the core spatial deficit, a bias in spatial
some of the deficits are supported by a common patho- attention and salience mapped in an egocentric coor-
physiological mechanism despite variability in lesion dinate frame, is caused by the dysfunction of a dorsal
location. Second, although acutely both left- and frontal-parietal network that controls attention and eye
when they are task-irrelevant (Bays et al., 2010; Shom- at rest also suggest an indwelling imbalance in the
stein, Lee, & Behrmann, 2010; Snow & Mattingley, mechanism’s controlling gaze. These motor biases are
2006) or lead to repeated re-fixations during search likely associated with attentional biases because of the
tasks (Husain et al., 2001). functional relationship between the corresponding
Importantly, a spatially lateralized bias is observed neural systems (Corbetta et al., 1998; Rizzolatti, Riggio,
even in the absence of a stimulus. When neglect sub- Dascola, & Umiltá, 1987).
jects searched for a nonexistent object in complete
darkness, search patterns as measured by eye or head An Egocentric Frame of Reference Spatial deficits
position were strongly biased toward the ipsilesional can be separated based on the reference frame in which
field (Hornak, 1992). Moreover, gaze deviations are stimuli are coded. Neglect is often egocentric (viewer-
observed tonically at rest, similar to those observed centered), with left and right hemi-spaces based on
during task performance (figure 19.1C; Fruhmann the observer’s midline. Neglect can also be allocentric,
Berger, Johannsen, & Karnath, 2008). where the midline is defined from the central axis of
Spatial biases in the dark during target search could a stimulus, irrespective of its position in the environ-
reflect a reduced salience of contralesional spatial loca- ment (stimulus-centered) or of both its position and
tions during task performance, but the biases observed orientation (object-centered). (Chechlacz, Rotshtein,
& Humphreys, 2012; Marsh & Hillis, 2008). Egocentric & Luzzatti, 1978; Della Sala, Logie, Beschin, & Denis,
neglect is associated with regions classically damaged in 2004). In a famous experiment, patients with right-
spatial neglect, i.e., inferior parietal lobule (IPL), supe- hemisphere stroke and neglect were asked to report by
rior temporal gyrus (STG), and inferior frontal gyrus memory buildings to the left and right of the cathedral
(IFG), while allocentric neglect has been associated in the Piazza del Duomo in Milan while imagining they
with damage of inferior temporal regions (figure 19.2A; were facing the front of the cathedral. Patients with
but see Chechlacz et al., 2012). We suggest that egocen- neglect tended to ignore buildings on the left side of
tric neglect underlies a spatial disorder and is associated the square. However, when asked to repeat the same
with damage of systems involved in spatial processing task while imagining being rotated by 180 degrees, thus
(see below), while allocentric neglect reflects a problem looking toward the square with the shoulders to the
with object coding and is associated with damage to the cathedral, now they reported buildings on the right side
ventral visual system (Ungerleider & Mishkin, 1982). that had been previously ignored, while ignoring build-
ings on the left side that had been formerly reported.
What About Representational Neglect? An influ- This experiment clearly indicates the integrity of spatial
ential account proposes that neglect is a deficit in information in working memory, and the involvement
forming, storing, or manipulating the left side of mental of a “scanning” or “saliency” mechanism similarly to
images or information in VSTM, termed representa- what is observed during perceptual tasks. Furthermore,
tional neglect (i.e., representation within VSTM; Bisiach mechanisms for spatial attention, VSTM, and imagery
consistent with a direct effect of the activation of ventral Structural Damage Does Not Explain the Egocen-
right-hemisphere mechanisms on the dorsal attention tric Spatial Bias Spatial neglect is traditionally asso-
network. ciated with damage to parietal cortex (Critchley, 1953),
The interaction between mechanisms underlying especially IPL (Mort et al., 2003; Vallar & Perani, 1987).
nonspatial and spatial deficits is likely critical for the However, subsequent studies have also emphasized STG
pathogenesis and right-hemisphere lateralization of (Karnath, Ferber, & Himmelbach, 2001) and IFG
spatial neglect. We argue that this is due to the interac- (Husain & Kennard, 1996), and convincing evidence
tion of structural damage and secondary physiological exists that damage to other regions, including anterior
dysfunction of networks involved in the control of atten- insula and middle frontal gyrus, sometimes produces
tion (see below). neglect (figure 19.3A). Importantly, neglect patients,
especially in severe cases, have white matter fiber
Anatomy of spatial neglect damage, which can disconnect frontal, temporal, and
parietal cortex (Bartolomeo, Thiebaut de Schotten, &
Lesions causing spatial neglect are heterogeneous in Doricchi, 2007; Gaffan & Hornak, 1997; He et al., 2007;
location, since they can occur in frontal, temporopari- Thiebaut de Schotten et al., 2005; figure 19.3B–C).
etal, superior temporal, insular cortices, and even sub- White matter damage most commonly involves a dorsal
cortical structures (thalamus, basal ganglia). Often region lateral to the ventricle where arcuate and supe-
lesions involve the underlying white matter that con- rior longitudinal fasciculi (II and III) run parallel in an
nects anterior to posterior cortex. These observations anterior-to-posterior direction (Doricchi & Tomaiuolo,
raise the question: How does a relatively homogenous 2003; figure 19.3C). Finally, neglect can be also caused
behavioral syndrome arise from a heterogeneous set of by damage to subcortical nuclei (pulvinar, caudate,
structural lesions? putamen; Karnath et al., 2002; Vallar & Perani, 1987)
Importantly, arousal-related activations in the TPJ and primary spatial deficits in neglect are housed in dorsal
insula/frontal operculum overlap with regions that are regions, while the lesions that cause neglect occur in
damaged in spatial neglect and recruited during reori- ventral regions. The main insight comes from neuroim-
enting and target detection (Heilman, Bowers, Valen- aging studies in neglect patients showing that ventral
stein, & Watson, 1987; Malhotra, Coulthard, & Husain, lesions cause physiological abnormalities in structurally
2009). intact dorsal frontoparietal cortex corresponding to the
Therefore, a consistent set of right-hemisphere-dom- dorsal attention network. First, at 3 weeks post-stroke, a
inant regions is recruited during tasks that involve reori- widespread cortical hypo-activation during a spatial
enting, target detection, and arousal/vigilance. Damage attention task was observed more strongly in the right
to these regions acutely or chronically causes nonspatial than left hemispheres (figure 19.4A). The cortical
deficits similar to those described in patients with hypo-activation was associated with a large interhemi-
neglect. spheric imbalance of activity in dorsal parietal cortex
(figure 19.4B). The interhemispheric imbalance nor-
Pathogenesis of spatial neglect malized at the chronic stage (9 months post-stroke) in
parallel with an overall improvement of cortical activity
Physiological Correlates of the Egocentric and spatial neglect (figure 19.4B; Corbetta, Kincade,
Spatial Bias in Neglect The model we propose Lewis, Snyder, & Sapir, 2005). Interestingly, activity in
aims at reconciling the seeming contradiction that ipsilesional occipital visual cortex was also abnormal,
signals for saliency, attention, and eye movement; the showing reductions in magnitude and spatial selectivity
(Corbetta et al., 2005). These impairments in sensory- lateral rotation of the eyes, head, and body in neglect
evoked activity, possibly reflecting abnormal top-down patients at rest, and contribute to the observed abnor-
control (Bressler, Tang, Sylvester, Shulman, & Corbetta, malities in task-evoked responses. Overall, then, in
2008), may further lessen the saliency of contralesional structurally intact regions of the dorsal attention
stimuli. Secondly, neglect patients show anomalies in network, ventral lesions induce changes in blood-oxy-
the pattern of spontaneous activity fluctuations within gen-level-dependent (BOLD) resting functional con-
the dorsal attention network (figure 19.4C). Coherence nectivity and task-evoked activity that reflect abnormal
between left and right parietal regions was disrupted 3 interhemispheric interactions and response balances
weeks post-stroke and improved over time in parallel (Kinsbourne, 1987), which plausibly explain the ego-
with the improvement of neglect (He et al., 2007). centric spatial bias in neglect.
Both task-evoked activity and resting coherence are
functionally significant. Left parietal activity is stronger Right Hemisphere Lateralization of Spatial
in subjects with more severe neglect, as indexed by the Deficits Previous models have explained the right-
difference in response times to contralesional versus lateralization of neglect, and hence attention in the
ipsilesional visual targets (Corbetta et al., 2005). Stron- human brain, by proposing that the right hemisphere
ger left- than right-hemisphere activation of parietal contains a bilateral representation of space while the
and occipital regions may reflect a biased representa- left hemisphere contains predominantly a contralateral
tion of stimulus salience and the locus of spatial atten- space representation (Heilman, Bowers et al., 1987;
tion (Bisley & Goldberg, 2003; Sylvester, Shulman, Jack, Mesulam, 1999; figure 19.5A). Damage to the right
& Corbetta, 2007). This interpretation is consistent with hemisphere impairs attention to the left hemifield, while
transcranial magnetic studies, in which inactivation of damage to the left hemisphere can be compensated. A
left posterior parietal cortex reduced left-field neglect second, “opponent-process” theory proposes that each
(Brighina et al., 2003; Koch et al., 2008). Secondly, at hemisphere promotes orienting in a contralateral direc-
rest, significant correlations are found throughout the tion, but the strength of this bias is stronger in the left
dorsal attention network between reductions in inter- than right hemisphere (Kinsbourne, 1987; figure
hemispheric coherence and the magnitude of the 19.5B). Left hemisphere lesions cause only mild right
spatial bias; for example, the difference in reaction spatial neglect, because the unopposed orienting bias
times between contralesional and ipsilesional targets generated by the right hemisphere is relatively weak.
(figure 19.4D; Carter et al., 2010; He et al., 2007). The According to our model, right-hemisphere domi-
anomalies in spontaneous activity may underlie the nance of neglect reflects an asymmetrical interaction
ABSTRACT During the allocation of spatial attention, humans studies have revealed a basic organization of the
and animals selectively filter information that is relevant to primate visual system in which the image of an organ-
behavior, avoiding information-processing overload. A ism’s environment projected onto the retina is
change in receptive field profiles may be one neural corre-
late of such effects of attention. A neuron’s visual receptive
encoded by populations of neurons in a multitude of
field is the region of the visual field where the presentation areas in striate and extrastriate visual cortices, each
of a stimulus evokes a train of action potentials. Receptive containing retinotopically organized maps emerging
fields therefore are spatiotemporal filters representing the from the systematic tiling of the visual field by the RFs
spatial allocation of processing resources across the visual of neighboring neurons. Within a given cortical area,
field. Receptive fields have been seen as invariant, providing
the size of RFs increases with eccentricity. Similarly,
the visual system with a stable “labeled line” code for the
spatial position of visual stimuli. In recent years, this view RF sizes increase along the processing hierarchy, from
has changed; it has been proposed that under certain early striate to extrastriate visual cortices. In associa-
circumstances the spatial profile of receptive fields changes, tion areas of the frontal, parietal, and temporal lobes,
providing the visual system with flexibility for allocating RFs become so large that they cover most or all of the
processing resources in space. Here, we focus on the brain
visual field captured by the eyes.
system for encoding visual motion information and review
recent findings documenting interactions between spatial The spatial tuning embodied by a visual neuron’s RF
attention and receptive fields in the visual cortex of pri- is combined with a tuning for nonspatial features, that
mates. Such interactions create a fine balance between the is, a systematic selectivity for one or several nonspatial
benefits of receptive field invariance and those derived from stimulus properties. For example, neurons in area V1
the attentional modulation of receptive field profiles. change their firing rate when stimuli shown inside the
RFs vary their color, orientation, motion direction, or
speed. In areas downstream from V1, feature selectivity
Visual receptive fields: Caught between labeled becomes segregated into two main streams, the “what”
lines and dynamic resource allocation and “where” pathways. In the former, reaching from V1
to inferotemporal cortex, neurons show selectivity for
Neurons in the primate visual system represent visual orientation, color, shape, and even complex objects
information by variations in their firing rate (fre- such as faces and houses. In the latter, which comprises
quency of action potentials) when different visual a chain of areas leading into parietal cortex, neurons
stimuli are presented within a specific, circumscribed show selectivity for motion features such as direction
region of visual space. This region, or receptive field and speed, and for stereoscopic disparity (depth). The
(RF), is considered a neuron’s essential physiological selectivity of a neuron’s RF for a given stimulus feature
property; it acts as a spatial filter, shaping the encod- can be described by the cell’s tuning curve, a plot of
ing of retinal signals across the visual system. Since mean response as a function of the feature value pre-
the seminal work of Hubel and Wiesel in the 1960s sented inside its RF.
(e.g., Hubel & Wiesel, 1968), electrophysiological A predominant and long-held view has been that the
methods in nonhuman primates have been continu- spatial structure of the neurons’ RF, as well as their
ously refined to determine the location and shape of nonspatial tuning, are hard-wired properties. Such
neuronal RFs throughout the visual cortex. Such invariance ensures a stable “labeled line code” for the
RF FC
FC Tuning
pattern
90
80
70
60
Spikes/second
50
40
30
20
10
0
–150 –100 –50 0 50 100 150
Tuning pattern
Antipreferred
pattern
Figure 20.1 Switching attention between two moving stimuli pattern) and the other could change direction from trial to
inside the receptive field (RF) of MT neurons. The top panels trial (tuning pattern). The lower panel illustrates tuning
illustrate the experimental design of Treue and Martinez- curves (responses as a function of the tuning pattern direc-
Trujillo (1999). Two random dot patterns (RDPs) were pre- tion) obtained when the animal attended to the fixation point
sented inside the RF of an MT neuron (white area) while the (black), to the tuning pattern (red), and to the anti-preferred
animal fixated a dot (small square). One RDP always moved pattern (blue). (See color plate 16.)
in the neurons’ anti-preferred direction (anti-preferred
necessary to map RFs while spatial attention is allo- neuron’s RF separately for each of the two spatial
cated to one or the other of two locations inside a attention conditions. When the animal directed spatial
given RF. Womelsdorf and colleagues (Womelsdorf, attention to one of the patterns, the RF shifted its
Anton-Erxleben, Pieper, & Treue, 2006) recorded the center toward that attended stimulus’s location. In
responses of single neurons in area MT of rhesus addition, there was also a small shrinkage of the RF
monkeys while the animals attended to one of two around that stimulus (figure 20.2A). This modulation
identical RDPs positioned inside the recorded neu- of RF profiles indicates that early in the hierarchy of
ron’s RF. The task for the animals was to sustain atten- visual processing, unattended information is filtered
tion on one pattern, wait for a change in its direction, out from the RF and only behaviorally relevant
and release a lever within a short time window after the attended signals are further processed in downstream
change onset. Because the second, unattended pattern areas.
could also change direction, and trials were terminated One feature of RFs in many visual areas is their cen-
without reward if the animal released the lever in ter-surround structure; the center is an excitatory
response to the distractor change, they were motivated region where stimuli produce an increase in response,
to attend to the target RDP and ignore the distractor. and the surround is an inhibitory region where stimuli
By flashing probes at different locations inside and produce response inhibition. One interesting question
around the RF while the animal was waiting for the arising from the results of Womelsdorf and colleagues
stimulus to change, the researchers could map a is whether both of these regions shift during the
Probe
locations
c d
S3 S1 S2 Receptive
field
–40 0 40 5°
Figure 20.2A Left: Sketch of the layout in the study of Wom- RF, to S3 (panel b). The surface color at each point in the
elsdorf et al. (2006), depicting an example of the placement plots indicates the increase in the neuron’s response elicited
of the three moving random dot patterns (shown here as by the presentation of a probe stimulus at that position, over
textured circles) that were present in every trial as well as the the response observed in the absence of a probe (that is, when
grid of locations at which a series of small probes could briefly only S1 and S2 were present). Panel d depicts a difference
appear within a trial. The single dot represents the fixation map computed by subtracting the RF when attention was on
point where the animal has to maintain his gaze throughout S1 from the RF when attention was on S2. The map illustrates
every trial. Right: RF profiles of an example neuron, when that shifting attention from S1 to S2 enhances responsiveness
attention was directed inside the RF, to stimulus S1 (panel a) around S2 and reduces it near S1. (See color plate 17a.)
or S2 (panel c), or when attention was directed outside the
allocation of attention. Anton-Erxleben and colleagues football game or staring at a menu in a restaurant). In
(2009) answered this question using a similar experi- these circumstances it is very likely that more than two
mental design as in figure 20.2B, with the difference stimuli will share a single visual RF. This is even more
that they also stimulated the RF surround with the small frequent in extrastriate visual areas, where RFs com-
probes. They found that both RF regions shift with prise several degrees of visual angle. Nevertheless,
shifts in the allocation of attention (Anton-Erxleben, humans can allocate spatial attention to one stimulus
Stephan, & Treue, 2009). within a crowd, selectively processing details for that
These findings provide support for Moran and stimulus while ignoring other stimuli nearby (He, Cava-
Desimone’s hypothesis that spatial attention can change nagh, & Intriligator, 1996; Palmer & Moore, 2009).
a neuron’s RF profile. Moreover, they indicate that the Moreover, studies of multi-object tracking have
sensitivity of the attended RF region increases at the shown that human subjects can simultaneously track
expense of one of the unattended parts. Thus, the origi- (attend to) multiple, independently moving objects
nal view outlined above of neurons in visual cortex as without moving their gaze; that is, they can split the
hard-wired, invariant filters for the location and other “spotlight” of spatial attention into more than one
properties of visual stimuli has to be abandoned in favor focus (Cavanagh & Alvarez, 2005). An interesting
of a system where neurons receive retinal input from a question arising from this observation is whether the
restricted region of the visual field, but the sensitivity attentional mechanism that causes changes in RF pro-
of different RF regions can dynamically change depend- files is flexible enough to selectively enhance process-
ing on the allocation of spatial attention. ing in two or more attended regions of a given RF
while suppress processing in a third region. For the
When more than two stimuli share a visual particular scenario of two attended objects separated
receptive field: Splitting the spotlight of attention by an irrelevant distractor in between, that may mean
that the RF would be virtually split by attention into
In many visual scenes, stimuli can be cluttered over a two regions of enhancement, separated by a region of
small region of visual space (e.g., while watching a inhibition.
‘Inward’
B C D MI
80
0
−0.1
Cells
80
0
Trials
−0.2
0 −0.4
−6 −4 −2 0 2 4 6
Distance to RF-pattern (°)
Stimuli onset Stimuli onset
‘outward’ ‘inward’ 0
−0.1
MI
Response (Hz)
60
−0.2
29 70 80 80 80 70 29
−3 −2 −1 0 1 2 3
0 RF region
−4 −2 0 2 4 −4 −2 0 2 4
Distance to RF-pattern (°) Preferred Antipreferred
Figure 20.2B Splitting receptive fields with attention. translating RDP position during attend RF (green) and tracking
(A) Sketch of the experimental paradigm of Niebergall et al. (red) trials. (C) Responses when the translating pattern dots
(2011b). The experiment was identical to the one described moved in the anti-preferred direction. (D) Response modula-
in figure 20.4, except that a third RDP (with local dot motion tion for all neurons. The cells have been sorted according to
in the neuron’s preferred direction) was placed and stayed in their RF size, and the strength of the modulation appears in
the center of the RF, and the animal was instructed to attend color. The arrows indicate the direction of the translating
either to the translating RDPs (tracking) or to the RF stimulus patterns (up: preferred, down: anti-preferred). The bottom
(attend RF). (B) Responses for attend RF (green rasters) and graph plots the mean modulation for both directions (two
tracking (red rasters) trials. Data from an example cell for the lines) as well as the difference in modulation between them
stimulus configuration with the translating RDPs dots locally (gray band, representing the average difference of the two
moving in the preferred direction. The bottom plot shows line plots ± 95% confidence interval). (See color plate 17b.)
average responses (± standard error) as a function of the
A recent study by Niebergall, Khayat, Treue, and hypothesis that splitting the attentional spotlight across
Martinez-Trujillo (2011b) using single-cell recordings the two tracked stimuli would produce an area of sup-
in area MT of macaque monkeys provides a neural cor- pression at the RF center, affecting the processing of
relate of the ability to split the focus of spatial attention the irrelevant distractor RDP.
between two stimuli within a visual RF separated by an They found that when the two tracked patterns
irrelevant distractor. They trained rhesus monkeys to passed alongside or entered the RF, the responses to
attentively track two RDPs that translated across a com- the central stimulus in the receptive field were reduced,
puter screen along parallel paths and passed a third with the maximum suppression when the three patterns
RDP positioned inside a recorded neuron’s RF (figure were aligned (figure 20.3B). Interestingly, they tested
20.3A). During task trials, the animals maintained two stimulus configurations: in the first, the three
their gaze on a light spot. This allowed testing of the stimuli moved in the neuron’s preferred direction. In
‘Inward’
B
Outward Inward
60
100
Tracking
Response [Hz]
Att.-fixation
Figure 20.3 Expanding receptive fields with attention. time. (B) Result from an example neuron. Left: Data from
(A) Sketch of the experimental paradigm of Niebergall et al. “outward” trials. Right: Data from “inward” trials. Data points
(2011a). Two RDPs moved across the screen following parallel represent the average responses evoked by RDPs with local
trajectories (moving outward or inward relative to the fixation dots moving in the neuron’s preferred direction during track-
point) and moving through the peripheral parts of a neuron’s ing (gray), and attend fixation (white). Gaussian fit–predicted
receptive field (RF, dashed circle). During attend-fixation trials, values are superimposed (dashed line, tracking; solid line,
animals ignored the RDPs and had to detect a luminance attend fixation). The DFR50 (downward arrows) represents
reduction in the fixation point. During tracking trials, animals the distance from the Gaussian center to the point of half-
had to detect a change in the local speed of the dots in one maximum response during attend fixation (horizontal line) as
of the RDPs, which occurred at an unpredictable point in a measure of receptive field size.
the second, the translating patterns moved in the anti- distant stationary stimuli (McMains & Somers, 2004;
preferred direction and the pattern at the center of the Morawetz, Baudewig, Treue, & Dechent, 2010). It dem-
RF moved in the preferred direction. They found that onstrates that attention is a flexible and adaptable
in both configurations, directing attention to the trans- mechanism able to produce dramatic changes in RF
lating patterns effectively suppressed the responses to profiles according to task demands.
the pattern positioned in between, relative to a condi-
tion where the latter pattern was attended and the Expansion of RFs during attentive tracking
translating patterns were irrelevant. However, in the
configuration with the translating patterns moving in The studies described above have demonstrated shift-
the neuron’s anti-preferred direction, the suppression ing and contraction of RF profiles with spatial attention
of the central pattern was larger. These results demon- in order to reduce the influence of irrelevant stimuli
strate that focusing spatial attention on the two translat- on neuronal responses. An additional important func-
ing patterns produced a suppressive area of inattention tion of attention is to dynamically allocate processing
between them. Moreover, such suppression was depen- resources to the relevant visual input. This flexible allo-
dent on the selectivity of the neurons for the attended cation is restricted across visual space toward attended
objects. This finding matches the observation reported locations and is constrained by the RF boundaries. This
by functional MRI studies of spatially separated peaks is because the RF constrains a neuron’s processing
of activity across the retinotopic spatial maps in visual resources to input from a highly restricted region of
cortex when splitting spatial attention between two space. Whether and to what extent the RF boundaries
MT receptive field
MT neuron
V1 neurons
V1 receptive
fields
Figure 20.4 A possible mechanism for the attentional shift ellipse represents the MT neuron’s RF. The gray band at the
of receptive fields. The two panels are sketches of the putative bottom represents the even attentional gain applied to all V1
V1 inputs into an MT neuron. The small circles at the bottom neurons in the absence of a spatial focus of attention. (B) If
indicate the RFs of the V1 input neurons along a horizontal attention is directed to the right, the attentional gain increases
axis crossing the middle of the MT receptive field. Each small for the right V1 neurons and decreases for the left V1 neurons.
triangle represents the cell body of a V1 neuron, and the This causes the MT RF profile to shift toward the right. The
connected lines, the projection toward the MT neuron (large dashed circle represents the old RF, and the filled circle the
triangle). The bell-shaped function represents the strength of new RF in the presence of spatial attention on the right side.
the weights from the V1 neurons into MT. (A) The large
alter its RF properties, the visual system can fulfill two Born, R. T., & Bradley, D. C. (2005). Structure and function
core goals of attentional modulation of information of visual area MT. Annu Rev Neurosci, 28, 157–189.
Cavanagh, P., & Alvarez, G. A. (2005). Tracking multiple
processing, namely, the suppression of irrelevant infor-
targets with multifocal attention. Trends Cogn Sci, 9(7),
mation and the allocation of additional resources to the 349–354.
processing of relevant incoming sensory information. Ghose, G. M., & Maunsell, J. H. (2008). Spatial summation
Given the importance of a balance between the benefits can explain the attentional modulation of neuronal
and disadvantages of an invariant system and a highly responses to multiple stimuli in area V4. J Neurosci, 28,
adaptive and dynamic system, evolution seems to have 5115–5126.
He, S., Cavanagh, P., & Intriligator, J. (1996). Attentional
opted for a compromise. The changes to RF profiles resolution and the locus of visual awareness. Nature, 383,
induced by attention seem to be highly predictable and 334–337.
stay within relatively well-circumscribed boundaries Heeger, D. J., Simoncelli, E. P., & Movshon, J. A. (1996).
while still adapting to the demands of a given task. In Computational models of cortical visual processing. Proc
essence, the brain combines the benefits of both a sta- Natl Acad Sci USA, 93, 623–627.
Hubel, D. H., & Wiesel, T. N. (1968). Receptive fields and
tionary and a dynamic system. functional architecture of monkey striate cortex. J Physiol,
195, 215–243.
REFERENCES
Khayat, P. S., Niebergall, R., & Martinez-Trujillo, J. C.
Albrecht, D. G., & Hamilton, D. B. (1982). Striate cortex (2010a). Frequency-dependent attentional modulation of
of monkey and cat: Contrast response function. J Neuro- local field potential signals in macaque area MT. J Neurosci,
physiol, 48, 217–237. 30, 7037–7048.
Anton-Erxleben, K., Stephan, V. M., & Treue, S. (2009). Khayat, P. S., Niebergall, R., & Martinez-Trujillo, J. C.
Attention reshapes center-surround receptive field struc- (2010b). Attention differentially modulates similar neuro-
ture in macaque cortical area MT. Cereb Cortex, 19(10), nal responses evoked by varying contrast and direction
2466–2478. stimuli in area MT. J Neurosci, 30, 2188–2197.
ABSTRACT Selective visual attention is realized in brain circuits of an IT neuron will typically contain multiple objects.
by the dynamic restructuring of cortical information flow in However, when attention is directed to only one of
favor of sensory information that is behaviorally relevant, those objects, the IT neuron’s response is biased toward
rather than irrelevant and distracting. Electrophysiological
evidence suggests that this attentional selection of neuronal
the response that would be obtained if only the attended
information routing is implemented flexibly through selective object were presented (Chelazzi, Miller, Duncan, &
synchronization of neuronal activities at multiple frequencies Desimone, 1993).
and across all spatial levels of neuronal interactions. This Such dynamic biasing of responses in later visual pro-
chapter delineates the underlying neuronal processes that cessing stages could be achieved by selective enhance-
could give rise to selective neuronal communication through
ment (suppression) of the impact of those afferent
coherence and highlights recent insights about (1) selective
attentional routing through synchronized gamma activity in inputs from neurons in earlier visual areas coding for
sensory cortices, (2) long-range reverberation of selective top- the attended (nonattended) input (Reynolds, Chelazzi,
down information at beta and gamma-band frequencies, and & Desimone, 1999). However, the mechanisms underly-
(3) the role of low-frequency theta band rhythms to selectively ing this up-and down-modulation of neuronal input
sample and switch between attentionally relevant sensory
gain for subsets of converging connections are only
information. This survey suggests that core processes underly-
ing attentional structuring of neuronal communication poorly understood. They likely entail a selective increase
employ precise phase synchronization that is selective in of temporally precise and coincident inputs from those
space, time, and frequency. neurons activated by an attended stimulus in earlier
areas. Such a role of spike timing is suggested by fine-
grained attentional modulation of precise neuronal
The role of selective neuronal synchronization in synchronization within area V4. Enhanced synchroni-
the implementation of attentional top-down biases zation of the spiking output among those neuronal
groups activated by attended sensory input (Fries,
Attentional top-down biases on perceptual processes Womelsdorf, Oostenveld, & Desimone, 2008) translates
are ubiquitous, ensuring that neuronal circuitry is into enhanced coincident arrival of spikes at their post-
devoted to prioritize processing resources to the typi- synaptic target neurons at later stages of visual process-
cally small subset of sensory inputs that should control ing. These temporally coincident inputs will be highly
behavior. In visual cortex, this identification process has effective in driving neuronal activity (Azouz & Gray,
to act highly selectively, because the same sensory 2003; Tiesinga, Fellous, & Sejnowski, 2008). It is there-
aspects can pertain to different visual objects. Attention fore likely that selective synchronization within area V4
is thus required to functionally connect those neurons underlies attentional biasing within IT cortex, and
that represent information from attended visual objects, could thus underlie effective spatial routing of informa-
and decouple or filter out the contribution of cells tion flow within visual cortex (Bosman et al., 2012).
encoding distracting information (figure 21.1A). For Synchronized spike output not only leads to increased
example, neurons at the highest visual processing stage impact on postsynaptic target neurons in a feedforward
in IT cortex have receptive fields that span much of a manner, it also rhythmically modulates the group’s
visual field and respond selectively to complex objects ability to communicate such that rhythmic synchroniza-
composed of simpler visual features. Part of this selectiv- tion between two neuronal groups likely supports their
ity arises from their broad and convergent anatomical interaction, because rhythmic inhibition within the two
input at earlier processing stages from neurons that groups is coordinated and mutual inputs are optimally
have smaller receptive fields and simpler tuning proper- timed. We capture these implications in the framework
ties (Kravitz, Saleem, Baker, Ungerleider, & Mishkin, of selective neuronal communication through neuro-
2013). During natural vision, the large receptive field nal coherence (Fries, 2005, 2009).
synchronization synchronization
“Good”-phase
Effective strength
interactions
0 0.2 0.4
Ineffective –pi
Phase relation
interactions
“Bad”-phase
0
pi
Neuronal Neuronal
groups action potentials
A A Sensory impulse
C D Light Light
+ + + -
+ Trials with “good” B C B C
phase relations 1 2
2
c(AB) for ( [ 1 + 2 ] – [ 3 + 4 ] ) / 2 0
0 ms 25 ms
c(AB) for ( [ 1 + 3 ] – [ 2 + 4 ] ) / 2
0.08 Gamma
cycle
correlation
Power
V4
0.04
0.1
Spikes
/ sec. 5 ms
0
10 20 60 100 140
Frequency (Hz)
Figure 21.1 (A) Attention renders a subset of cortical con- synchronization. (D) Evidence that rhythmic optogenetic acti-
nections effective. (B) Rhythmic activity provides briefly vation in mouse somatosensory cortex modulates gain of
recurring time windows of maximum excitability (LFP responses to whisker stimulation. A whisker was stimulated at
troughs), which are either in phase (black and dark gray different phases of an optogenetic activation rhythm. The
groups), or in anti-phase (black and light gray groups). spiking responses of pyramidal cells (y-axis) varied as a func-
Mutual interactions (right panel, upper axis) are high during tion of the phase (x-axis). ((B) and (C) adapted from Wom-
periods of in-phase synchronization and lower otherwise. elsdorf et al., 2007; (D) adapted from Cardin et al., 2009.
(C) The trial-by-trial interaction pattern between neuronal Reprinted by permission from Macmillan Publishers Ltd:
groups (A to B, and A to C) is predicted by the pattern of Nature.)
Selective Communication Through Coherence brief time periods between inhibition provide time
Local neuronal groups frequently engage in periods windows for effective neuronal interactions with other
of rhythmic synchronization. During activated states, neuronal groups, because they reflect enhanced post-
rhythmic synchronization is typically evident in the synaptic sensitivity to input from other neuronal groups
gamma-frequency band (30–90 Hz; Fries, 2009). In vitro as well as maximal excitability for generating spiking
experiments and computational studies illustrate that output to other neuronal groups (Azouz & Gray, 2003;
gamma-band synchronization can emerge from the Fries, Nikolic, & Singer, 2007; Tiesinga et al., 2008). As
interplay of excitatory drive and rhythmic inhibition a consequence, when two neuronal groups open their
imposed by interneuron networks (Bartos, Vida, & temporal windows for interaction at the same time, they
Jonas, 2007; Börgers, Epstein, & Kopell, 2005). Inter- will be more likely to influence each other (Womelsdorf
neurons impose synchronized inhibition onto the local et al., 2007). The consequences for selective neuronal
network (Bartos et al., 2007; Cardin et al., 2009). The communication are illustrated in figure 21.1B: if the
V1
V4
D E F
u g d g
c i c i
G H I
J K L
Figure 21.2 (A) The spatial coverage of ECoG electrode relative to pre-stimulus baseline in V1a (E) and V1b (F).
locations. Shaded regions indicate areas V1 and V4. (G, H) Coherence spectra between V4 and V1a (G) and
(B, C) Attention conditions with arrows denoting that only between V4 and V1b (H). (I, J) Bottom-up Granger causal
the behaviorally relevant and therefore attended stimulus was influence of V1a onto V4 (I) and of V1b onto V4 ( J).
routed from the respective V1 site to the V4 site. The shading (K, L) Top-down Granger causal influence of V4 onto V1a
of the box contours (black, gray) is used in the remaining (K) and of V4 onto V1b (L). Gray bars in (G)–(L) indicate
panels to denote the attention conditions. (D) Spectral power the frequencies with a significant effect. (Adapted from
change relative to pre-stimulus baseline in V4 (gray line Bosman et al., 2012. Reprinted by permission from Macmillan
mostly occluded by black line). (E, F) Spectral power change Publishers Ltd: Nature.)
(Dhamala, Rangarajan, & Ding, 2008). Applying the 2011), because feedforward connectivity originates
Granger causal metric, Bosman et al. (2012) revealed a from superficial layers in primary visual cortex.
predominance of Granger causal influences from the
attended V1 neuronal group onto the neuronal group Origins of attentional top-down biases onto visual
in V4 in the gamma-frequency band (figures 21.2I and cortex: Selective long-range coordination at
21.2J), and only a subtle Granger causal feedback gamma- and beta-frequency synchronization
influence of V4 to V1 (figures 21.2K and 21.2L). These
findings resonate well with the predominance of The described findings illustrate that the behavioral
gamma-band synchronization in superficial versus deep relevance of a stimulus location or feature determines
visual cortical areas reported above (Buffalo et al., how the neuronal circuitry in V4 synchronizes
Figure 21.3 (A) Average coherence of spike-trains of visu- memory task required encoding of a sample object or sample
ally tuned neurons in the frontal eye field (FEF) to the local object location. Following a 0.8–1.2 sec delay, the macaque
field potential (LFP) when the monkeys attended inside (red) monkeys had to identify which of two objects or object loca-
or outside (blue) the visual receptive field of the FEF cell. tions matched the sample. (G) Average selectivity of coher-
(B) Same as in (A), but for “motor” cells in the FEF that are ence for object identity (upper panel) and location (bottom
tuned to saccadic target directions and are located predomi- panel) during sample and delay epochs (separated by white
nantly in deeper cortical layers in the FEF. (C) Long-range vertical lines). The LFP coherence was computed between
coherence of visually tuned FEF cells with the LFP recorded pairs of frontal and posterior parietal sites. (H) Average
in visual area V4 FEF. (D) Long-range coherence of motor- coherence, for each of three objects, rank-ordered for each
tuned FEF cells with the LFP recorded in visual area V4. recorded pair according to their object preference. The time
(E) Illustration of cells in the FEF that are primarily respon- course shows differences in coherence with the start of the
sive to visual stimuli and located in supra-granular layers delay period. (I) The proportion of frontoparietal LFP-LFP
(red), visuomotor neurons (half-red, half-green), and motor- pairs that showed individually significant selectivity of beta
tuned neurons (green) that are predominantly located in band coherence, split according to their anatomical locations.
deeper layers. The solid/dashed arrows to V4 cells indicate (Panels (A)–(D) adapted from Gregoriou et al., 2012; panels
selective coupling of visual FEF to visual V4 cells, but a lack (F)–(I) adapted from Salazar et al., 2012. Reprinted with
of coupling from motor-related FEF cells. (F) The working permission from AAAS.) (See color plate 18.)
A Attention inside RF C E
Attention outside RF
Spike-LFP coherence
Spike-LFP coherence
Motor-
FEF cells
B D
FEF spikes— FEF spikes—
Spike-LFP coherence
Spike-LFP coherence
FEF
FEF LFP V4 LFP
V4
Macaque
cortical surface
F I
60
Fronto-parietal network topography
Coherence
Percent of sign. LFP-LFP pairs
20
selectivity index
10
Coherence
0 20
Object location
50
40
30
20
10
0
Figure 21.4 (A) Schematic illustrating the communication 7.1 Hz, for the last 120 ms prior to target onset and standard-
through coherence hypothesis in an attentional task where ized performance after the phases were aligned for each par-
one location (apple) is attended while the other (pepper) is ticipant so that the optimal phase would be lined up with the
ignored. Lower visual areas represent both objects, yet gamma- zero degrees bin. Note that from the optimal phase, perfor-
band synchronization supports the communication of the mance declines and reaches a minimum at the opposite
attended object for further processing and curtails the com- phase. (E) Schematic illustration of a task that measures the
munication of the unattended object from further processing sampling rhythm using psychophysics. (F) Detection perfor-
in higher visual areas. (B) Incorporating the nesting of mance as a function of temporal interval between target event
gamma-band synchronization within a theta sampling rhythm and flash event (attention reset event). Zero point denotes
when, for example, two objects are behaviorally relevant. onset of flash event; negative values denote trials in which the
Within each theta cycle, a given object is selected, and this target preceded the irrelevant flash. Black (gray) line: detec-
selection is implemented in the gamma-band synchronization tion for targets appearing in the same (opposite) visual field
much like illustrated in panel (A). In each sampling cycle, the as the flash. These time courses of performance reveal a
selection alternates between the two objects. (C) Statistical rhythmic sampling of each location sequentially. (Panels (C)
significance of a phase bifurcation between detected and and (D) adapted from Busch, Dubois, & van Rullen, 2009,
missed targets. The white rectangle highlights a time-frequency and van Rullen, Busch, Drewes, & Dubois, 2011.)
region of significance. (D) The relationship between phase at
“Lower”
+ Visual
Areas
“Higher”
Visual
Areas
Time
Time
C D E F
p < 0.05 (corrected)
Detection accuracy
(proportion correct)
Frequency (Hz)
0.8
0.6
Target event
0.4
Flash event
0.2 Same
Opposite
0
–0.6 –0.4 –0.2 0 0.2 0.4 0.6 0.8 1
noudoost, albarran, and moore: neural signatures, circuitry, and modulators 233
Table 22.1
Summary of signatures of top-down attention within the visual system. An overview of currently known signatures of attention and their
potential implications. Abbreviations: LFP, local field potential; RF, receptive field; FEF, frontal eye field; LIP, lateral intraparietal area; IT,
inferotemporal cortex; SC, superior colliculus; LGN, lateral geniculate nucleus; Pulv, pulvinar nucleus of the thalamus; PFC, prefrontal
cortex.
Signature Potential implication Area
↑ Spiking rate Increases output signal LGN, TRN, Pulv, SC, V1, V2,
V4, MT, IT, LIP, FEF, PFC
↑ γ-band LFP power Increases locally synchronous synaptic activity FEF, V4
↑ Local γ-band coherence Potentially increases signal efficacy at postsynaptic FEF, V4, LIP
targets
↑ Cross-areal γ-band coherence Potentially facilitates long-range interactions by FEF-V4, PFC/FEF-V4, LIP-MT
providing a common temporal reference frame
↓ Response variability Increases reliability of encoded information V4
↓ Low-frequency synchrony Potentially reduces output redundancy by decreasing V4
correlation
↓ Correlated noise at low Potentially increases the sensitivity of pooled responses V4
frequencies
Receptive field shift Potentially increases output signal by recruiting MT, V4
neurons
of signals by reducing the phase lag of inputs to down- can be attributed to independent sources of noise
stream structures and thus potentially increasing their within each neuron, the correlated fluctuations shared
postsynaptic efficacy. Spikes reflect the output of across groups of neurons, or both. For individual
neurons, while LFPs are thought to largely reflect the neurons, the Fano factor, a measure of neural firing
inputs (excitatory and inhibitory post-synaptic poten- variability quantified as the variance divided by the
tials). Coherence in neural activity can be measured mean response, is reduced when attention is directed
between the spiking activity within different sets of toward the respective RFs of those neurons (Mitchell,
neurons, between their LFPs, or between the spikes of Sundberg, & Reynolds, 2007). Recent studies have
one set of neurons and the LFPs of the other. Specifi- found that attention reduces the correlated variability
cally, it has been argued that gamma-band (25–100 Hz) among pools of many neurons in area V4, which
synchronization of spiking from selected groups of appears to improve the SNR more effectively than
neurons can increase the influence of these spikes on increases in firing rate or decreases in independent
downstream areas (Salinas & Sejnowski, 2001). Syn- firing variability (Cohen & Maunsell, 2009; Mitchell,
chrony in the gamma band among neurons encoding Sundberg, & Reynolds, 2009). This decorrelation arises
an attended representation could facilitate the integra- mainly from a suppression of low-frequency (<5 Hz)
tion of spikes from these populations converging on rate fluctuations shared across neuronal populations
their postsynaptic targets, enhancing the attended rep- and is thought to reflect the attention-dependent low-
resentation. Various studies have reported increases in frequency desynchronization observed in area V4
the local gamma band (Fries, Womelsdorf, Oostenveld, (Fries et al., 2008).
& Desimone, 2008; Gregoriou, Gotts, Zhou, & Desim- Another measurable signature of attention is the
one, 2009), and that synchrony has been found to cor- number of neurons involved in processing particular
relate with behavioral performance (Womelsdorf, Fries, visual information. The act of attention potentially
Mitra, & Desimone, 2006). Although these observations amplifies and sharpens selected signals through the
support the positive role of synchrony in attentional recruitment of additional neurons to represent selected
selection, there are differences in the actual reported information and, additionally, through the narrowing
frequency ranges in which significant effects of atten- down of their spatial tuning (RFs). Studies that show
tion are observed, and the role of synchrony in atten- shifting and shrinking of RFs toward attended stimuli
tion remains an active area of research (see chapter 21, provide evidence for this adaptive prioritization of
this volume). visual representations (Connor, Preddie, Gallant, &
Other studies have found that attention also Van, 1997)
decreases the variability of neuronal firing of individ- In summary, attention increases the efficacy of visual
ual neurons. Variability in firing patterns of neurons processing via various different potential mechanisms,
Sources of attention
SC Pulvinar
It is clear that visual perception depends on the direc-
tion of attention, and as is now well established, this
dependence is reflected in the responses of neurons
throughout the visual system via changes in the SNR of Figure 22.1 Macaque brain highlighting structures impli-
cated in the control of visual selective attention and their
visual information (table 22.1). However, until recently
connections, namely, the lateral intraparietal area (LIP) in
little was known about the neural circuitry that controls posterior parietal cortex, the frontal eye field (FEF) in pre-
the perceptual effects of attention and their neural cor- frontal cortex, the superior colliculus (SC) in the midbrain,
relates within the visual system. In the following section, and the pulvinar nuclei in the thalamus. Dotted outlines indi-
we discuss several brain structures suspected of playing cate subcortical structures. (See color plate 19.)
a causal role in attention (figure 22.1 and table 22.2 ).
Frontal eye field addition, like neurons within earlier stages of the visual
system, FEF visual responses are modulated by covert
The frontal eye field (FEF), located within the prefron- attention (e.g., Armstrong, Chang, & Moore, 2009;
tal cortex (PFC), is an area originally known for its role Thompson, Bichot, & Sato, 2005). However, more
in overt attention, specifically saccadic eye movements recent studies revealed a prominent role of the FEF in
(Bruce, 1990). FEF neurons exhibit a spectrum of visual covert spatial attention. FEF neurons have a rich con-
and motor properties. Most FEF neurons respond to nectivity with posterior visual areas, both feedforward
visual stimulation within their RFs. A subset of those and feedback, the latter being implicated in the modu-
neurons will also respond prior to saccades of a particu- lation of posterior visual signals by the PFC. Electrical
lar direction and amplitude, most often (but not always) microstimulation of sites within the FEF (through
to locations within the visual RF. The remaining subset microelectrodes) with sufficiently high currents can
of FEF neurons consists of those that do not respond evoke highly reproducible saccades. More importantly,
to visual stimulation, but only prior to saccades. In microstimulation of FEF sites with currents too weak to
Table 22.2
Candidate brain areas implicated in the control of visual processing
Method of Signature
causality of
test attention FEF SC LIP Pulvinar
Stimulation Neural Increased visual response ? ? ?
and stimulus selectivity
Behavioral Increased visual detection Increased visual Spatially non- ?
and visual guidance of detection specific increases
saccades in visual
detection
Inactivation Neural Decreased stimulus Decreased stimulus ? ?
selectivity selectivity
modulation
Behavioral Distractor-dependent Distractor- Distractor- Distractor-
decrease in visual dependent dependent dependent
detection decrease in visual decrease in visual decrease in
detection detection (LIPv visual
not LIPd) discrimination
noudoost, albarran, and moore: neural signatures, circuitry, and modulators 235
evoke saccades (subthreshold microstimulation) none-
theless improves performance in an attention-demand-
ing task (Moore & Fallah, 2001). Specifically, FEF
microstimulation improves the ability of monkeys to
detect minute changes in the luminance of targets posi- ms
5 00
tioned within the part of space represented by neurons –1
at the stimulation site. In addition, subthreshold FEF 30
0
x
microstimulation also enhances the degree to which
voluntarily generated saccades are guided by the visual
features of saccadic targets (Schafer & Moore, 2007).
Thus, activation of FEF neurons is sufficient to increase
x
the influence of particular visual stimuli on behavior.
Subthreshold microstimulation of FEF sites also pro-
duces a brief enhancement of visually driven responses
of area V4 neurons with RFs in the region of space
represented by neurons at the stimulated FEF site
(Moore & Armstrong, 2003). Furthermore, it has been
shown that the magnitude of this enhancement -
increases when more effective RF stimuli are used and
1
Target discrimination
when non-RF (i.e., distractor) stimuli are present. Con-
Percent RF misses
versely, microstimulation of FEF sites that do not overlap 10 0.5
with particular V4 RFs suppresses the responses of those
0
V4 neurons, which reflects the effects observed in 5
endogenous attention studies. Additionally, this −0.5
enhancement of V4 responses is observed only with RF
0 −1
stimuli that align with the endpoint of the saccade Up Down −200 0 200 400
vector represented at the FEF site. As a result of this, Voluntary control Time from array
when two competing stimuli are present within a V4 RF, condition onset (ms)
FEF microstimulation drives visual responses toward Figure 22.2 Operant control of FEF neurons and its effects
those observed when an aligned stimulus is presented on selective attention measured behaviorally and neurophysi-
alone (Armstrong, Fitzgerald, & Moore, 2006). Func- ologically. Top: The operant control task in which the monkey
tional magnetic resonance imaging (fMRI) has been fixated a central spot on an otherwise blank video display and
was rewarded for increasing or decreasing the firing rate of
used to demonstrate the influence of FEF microstimula- FEF neurons. Dotted circle shows the FEF RF. Speaker icon
tion on visual activation throughout the cortex and musical note depict auditory feedback of FEF neuronal
(Ekstrom, Roelfsema, Arsenault, Bonmassar, & Vanduf- activity (spike train). Subsequent panels depict the visual-
fel, 2008). This work shows that FEF microstimulation search probe trials in which a search array appeared, the
enhances the visual activation of retinotopically corre- auditory feedback ceased (“x” on speaker icon) and the
monkey was rewarded (blue droplet) for directing a saccade
sponding foci within multiple visual areas and increases
toward an oriented bar target. Bottom left: The frequency of
the contrast sensitivity within multiple visual areas trials in which the monkey failed to respond to the target
(Ekstrom, Roelfsema, Arsenault, Kolster, & Vanduffel, (“misses”) opposite the RF was increased during DOWN
2009). operant control of FEF activity in both monkeys (square and
In addition to studies using electrically induced triangle symbols). Bottom right: Target discrimination by FEF
neurons was increased during upward (red) operant control
changes in FEF activity to test its role in attention,
relative to downward (blue). (Adapted from Schafer and
Schafer and Moore (2011) tested the hypothesis that Moore, 2011.) (See color plate 20.)
behaviorally conditioned, voluntary changes in FEF
neuronal activity are sufficient to bring about the
deployment of visual attention (figure 22.2). The blocks of trials), while maintaining central fixation on
authors used operant training techniques to examine a blank visual display. The authors probed the behav-
the impact of voluntary control of FEF activity on visu- ioral and neurophysiological consequences of operant
ally driven behavior. Monkeys were provided with real- control of FEF activity. They did this by introducing
time auditory feedback based on the firing rate of FEF probe trials in which the monkey performed a visual
activity, and rewarded for either increasing or decreas- search task while exerting operant control over FEF
ing that activity (in alternating “UP” and “DOWN” activity. They found that when the search targets
V4 RF V1 RF IPC RF
Modulation index
60
Firing rate (sp/s)
30 50
0.1
40
20 n = 86 30
0
20
10
10
–0.1
0 0
0.1 0.8 1.2 2.4 0 4 8 12
RF stimulus Bar length (degree) Distracter salience
(loom degree/s)
Figure 22.3 (A) Visual responses of a V4 neuron with a RF toward the stimulus within the neuron’s RF compared to
that overlapped the FEF RF measured during passive fixation when attention is directed elsewhere. Indices measured
(top panel). The bottom plot shows mean and standard error during control trials (gray) and during iontophoretic applica-
of the mean (SEM) visual responses to bar stimuli presented tion of acetylcholine (black) are shown. (C) Cholinergic
at varying orientations and the baseline-firing rate in the neurons in the owl’s IPC nucleus signal the physical salience
absence of visual stimulation (dashed lines) before (black) of stimuli by a characteristic switch response. The exemplar
and after (gray) the FEF D1R manipulation. (B) Enhance- neuron responds almost invariantly to RF stimuli across a
ment of attention effects in macaque V1 by ACh. The cartoon range of stimulus intensities as long as the stimulus is more
above depicts the behavioral task in which attention was physically salient than the other stimulus on the screen (dis-
directed covertly to a neuron’s RF stimulus (V1 RF) or to a tractor). When the RF stimulus is less salient, the neuron
stimulus outside of the RF (not shown) during fixation of a responds at a uniformly low rate. Salience is manipulated by
central spot (gray lines). The effect of spatial attention on the varying the speed at which a given stimulus looms (target
responses of V1 neurons to visual stimuli is quantified with a salience in this example is set at 7 degrees/sec). IPC neurons
modulation index for stimuli of varying lengths. Positive were recorded in owls during passive viewing (top panel).
indices indicate greater responses when attention is directed
appeared in the FEF RF, monkeys were less likely to responses of V4 neurons to visual stimulation, (2)
detect the target (i.e., they had more “misses”) on the increased stimulus selectivity, and (3) reduced cross-
DOWN-trials than UP-trials. In addition, the authors trial variability of visual responses. Note that similar
found that FEF neurons could discriminate targets changes to each of these three measures have been
from distractors better during UP trials compared to observed during endogenously directed attention
DOWN trials. This change in target discriminability was (McAdams & Maunsell, 1999; Mitchell et al., 2007). The
dependent on the direction of operant control and not results of Noudoost and Moore (2011a) provide two
on spontaneous fluctuations in firing rate. These results crucial pieces of evidence. First, they demonstrate
show that endogenous, voluntary changes in FEF neural changes in FEF neuronal firing rate are sufficient to
activity are sufficient to bring about both the behavioral bring about changes in sensory signals in posterior
and neurophysiological effects of visual attention, and visual cortex. Although the previous microstimulation
that explicit learning of an attention task is not required. results suggest this to be the case, they do not rule out
Noudoost and Moore (2011a) pharmacologically other possibilities such as antidromic spread of micro-
altered FEF activity by infusing small quantities of stimulation-evoked activity to other cortical or subcorti-
dopamine receptor agonists and antagonists at local- cal structures. Second, the results also demonstrate that
ized cortical sites. They found that, similar to FEF the influence the FEF has on posterior sensory signals
microstimulation, manipulation of FEF neuronal activ- can itself be influenced by changes in dopaminergic
ity through dopamine D1 receptors could enhance the activity. This latter point may establish a basis by which
visual responses of area V4 neurons at corresponding prefrontal dopamine contributes to attentional control,
spatial locations (figure 22.3A). Specifically, they found as appears to be the case in studies of attention-deficit
that blocking D1 receptors (1) increased the spiking hyperactivity disorder (Castellanos & Tannock, 2002).
noudoost, albarran, and moore: neural signatures, circuitry, and modulators 237
In addition to the studies taking a gain-of-function 2004). Other studies further demonstrated that the
approach, inactivation experiments have consistently behavioral benefits of SC stimulation on performance
shown that removal of FEF output disrupts behavioral are not dependent on whether microstimulation occurs
measures of attention. Wardak and colleagues studied in the visual or motor layers of the SC (Cavanaugh,
effects of FEF inactivation on behavioral performance Alvarez, & Wurtz, 2006), indicating that the benefit
in visually guided saccades and covert visual search tasks does not result from an evoked visual experience.
and found that FEF inactivation causes deficits in both The results of pharmacological inactivation of the SC
saccade and attention tasks (Wardak, Ibos, Duhamel, & also suggest a causal role of this area in visuospatial
Olivier, 2006). Monosov and Thompson have also attention. Lovejoy and Krauzlis (2010) trained monkeys
shown that inactivation of the FEF produces spatially in a covert spatial attention task. Monkeys were required
selective perceptual deficits in covert attention tasks to deploy covert attention to a cued location and report
(Monosov & Thompson, 2009). These deficits are stron- the motion direction of stimuli appearing at that loca-
gest on invalid cue trials that require endogenous atten- tion. The authors found that inactivation of the SC
tion shifting. Experiments using search classification resulted in profound deficits for targets in the corre-
tasks to record responses of inferotemporal (IT) sponding part of space. This deficit was only observed
neurons before and after FEF inactivation show that when target stimuli were presented in the presence of
both object-selective responses of IT neurons and distractors outside of the affected visual field, consistent
behavioral performance are affected by changes in FEF with the idea that the observed deficits are attentional
activity (Monosov, Sheinberg, & Thompson, 2010). in nature, rather than merely visual. More recently, the
Thus, not only does FEF activation (electrically or phar- authors tested the effects of SC inactivation on the
macologically) enhance behavioral and neuronal mea- neural signatures of attention within visual areas MT
sures of attention, its inactivation reduces performance and MST. Surprisingly, they found that in spite of prom-
on attention-based tasks, suggesting that FEF activity is inent behavioral deficits in visual attention following SC
not only sufficient to drive attentional deployment, but inactivation, the measured neural signatures of atten-
that it is also necessary for that deployment. tion within the two visual areas were unaffected (Zenon
& Krauzlis, 2012). This work suggests that although the
Superior colliculus SC may be necessary for attentionally directed behavior,
it may not be a source of attention-related modulation
Another oculomotor area suspected of playing a causal within visual cortex.
role in attention is the superior colliculus (SC). As with
the FEF, SC neurons exhibit a spectrum of visual and Lateral intraparietal area
motor properties. Most SC neurons respond to visual
stimulation within their RFs (Goldberg & Wurtz, 1972). Another brain region that has been implicated as a
A subset of those neurons will also respond prior to source of attentional modulation is the lateral intrapa-
saccades of a particular direction and amplitude, most rietal area (LIP), an area located in the lateral bank of
often to locations within the visual RF. The remaining the intraparietal sulcus of the parietal lobe. As with the
subset of FEF neurons consists of those that do not FEF and the SC, LIP neurons exhibit a spectrum of
respond to visual stimulation, but only prior to sac- visual and motor properties (Mazzoni, Bracewell,
cades. In general, neurons with largely visual properties Barash, & Andersen, 1996). As with the other struc-
are found in more superficial layers of the SC, while tures, LIP visual responses are also modulated by
neurons with largely motor properties are found in covert attention (e.g., Bisley & Goldberg, 2003). The
deeper layers. In addition, like neurons within earlier first evidence for a causal role of the LIP in visual
stages of the visual system, SC visual responses are mod- attention comes from inactivation studies. Wardak et al.
ulated by covert attention (Ignashchenkova, Dicke, found that inactivation of the LIP had no direct influ-
Haarmeier, & Their, 2004). As with the FEF, micro- ence on saccade mechanisms for visually and memory-
stimulation of the SC enhances performance on atten- guided saccades (i.e., saccades maintained the same
tion-demanding tasks. Subthreshold microstimulation accuracy, omission rate, latency, duration, and ampli-
of sites within the SC spatial map enhances visual per- tude; Wardak, Olivier, & Duhamel, 2002). However, LIP
formance in the corresponding region of space. This inactivation did cause monkeys to choose targets
has been observed in studies in which monkeys appearing on the ipsilateral side of visual field over
performed a direction-of-motion discrimination task contralateral ones and induced a delay in searching for
(Muller, Philiastides, & Newsome, 2005) or they visual targets in the ipsilateral field. Follow-up experi-
detected changes in visual stimuli (Cavanaugh & Wurtz, ments testing the monkeys’ performance on covert
noudoost, albarran, and moore: neural signatures, circuitry, and modulators 239
from the PFC, suggesting a means by which PFC control Studies employing behavioral paradigms that manip-
can exert network-wide attentional effects. For a more ulate bottom-up attentional orienting—for example,
detailed review of the role of neuromodulators in atten- using spatial cues (Posner, 1980)—have generally found
tion, see Noudoost and Moore (2011c). Here, we briefly that lesions of the basal cholinergic nuclei impair such
discuss recent evidence of a role for two major neuro- orienting (Voytko et al., 1994), while increased cholin-
modulators, acetylcholine and dopamine. ergic activity (e.g., via nicotine agonist) increases ori-
enting (Witte, Davidson, & Marrocco, 1997). Moreover,
both systemic administration of the muscarinic antago-
Acetylcholine Within the past twenty years, a nist scopolamine (Davidson, Cutrell, & Marrocco, 1999)
number of studies using human and animal subjects and its local injection into posterior parietal cortex
have yielded evidence for the role of acetylcholine have been shown to slow bottom-up orienting of atten-
(ACh) in attention (Warburton & Rusted, 1993). Sys- tion in nonhuman primates (Davidson & Marrocco,
temic increases in ACh activity have been shown to 2000), suggesting a role of ACh in the mechanism of
enhance visual selective attention in healthy human bottom-up attention. In contrast to top-down attention,
subjects (Warburton & Rusted, 1993). Cholinergic in which selection among different sensory stimuli
receptors are classified into two classes: metabotropic depends solely on the relevance of those stimuli to
muscarinic receptors (mAChRs) and ionotropic nico- behavioral goals, bottom-up driven selection is based
tinic receptors (nAChRs; Cooper, Bloom, & Roth, solely on the (physical) salience of stimuli. As with top-
2003). Although much of the evidence for behavioral down attention, bottom-up salience enhances the
enhancement through cholinergic stimulation involves responses of neurons within visual cortex (Burrows &
nAChRs, there is evidence for a role of both receptors Moore, 2009). Yet it remains unclear how ACh contrib-
in certain aspects of attentional control. Rodent studies utes to these effects. Recent studies employing owl
suggest that the processing of sensory signals within models reveal that neurons in a cholinergic nucleus
posterior areas might be influenced by the interaction exhibit response characteristics consistent with a
of PFC with ascending cholinergic projections (Sarter potential role in the selection of visual objects based on
et al., 2005), and the interaction appears to involve salience (Asadollahi, Mysore, & Knudsen, 2010).
nAChRs (Guillem et al., 2011). Within posterior areas, Neurons within the nucleus isthmi pars parvocellularis
basal forebrain stimulation enhances sensory signals (IPC) of owls transmit cholinergic inputs to the tectum
within somatosensory (Tremblay, Warren, & Dykes, and respond to both auditory and visual stimuli. Inter-
1990) and visual cortex (Goard & Dan, 2009), and in estingly, the visual responses of these neurons depend
all cases the effects appear to involve mAChRs. However, heavily on whether the stimulus in their RF is more
other studies have recently shown that within primary salient than stimuli outside of their RF; the magnitude
visual cortex (V1) it is in fact nAChRs that are involved of their responses decreases sharply at the boundary
in gain control (Disney, Aoki, & Hawken, 2007). Within where the relative salience of the RF stimulus falls below
V1, nAChRs are localized presynaptically at geniculo- that of a stimulus outside of the RF (figure 22.3C).
cortical inputs to layer IVc neurons, where they enhance Similar effects are observed within the owl optic tectum
responsiveness and contrast sensitivity of thalamorecipi- (Mysore & Knudsen, 2011), which is reciprocally con-
ent neurons. Recent experiments concerning the role nected with IPC in a precisely topographic manner.
of ACh in attentional modulation recorded visual These results suggest that salience-driven selection may
responses of V1 neurons in monkeys performing a originate in part from cholinergic inputs, or at least it
covert attention task (Herrero et al., 2008), and, consis- involves those inputs.
tent with previous results (e.g., McAdams & Reid, 2005),
an increase in V1 responses was found when monkeys Dopamine Dopamine receptors are grouped into two
attended toward RF stimuli compared to when they classes, D1 and D2. The D1 family includes D1 and D5
attended to non-RF stimuli (figure 22.3B). However, receptors, whereas D2, D3, and D4 receptors make up
iontophoretic application of ACh augmented the atten- the D2 family (Seamans & Yang, 2004). Within the PFC,
tional modulation of V1 responses. Furthermore, appli- D1Rs exhibit a bilaminar pattern of expression, while
cation of scopolamine, an mAChR antagonist, reduced D2Rs are less abundant and appear to be expressed
the attentional modulation, while application of the primarily within infragranular layers (Lidow, Goldman-
nAChR antagonist, mecamylamine, had no effect. Rakic, Gallager, & Rakic, 1991; Santana, Mengod, &
These results demonstrate a robust interaction between Artiga, 2009). Recent studies have explored the impact
attentional deployment and mAChR activity on the rep- of manipulating D1R-mediated activity within the FEF
resentation of stimuli within visual cortex. on saccadic target selection and on visual responses of
noudoost, albarran, and moore: neural signatures, circuitry, and modulators 241
and subcortical mechanisms. Cold Spring Harb Symp Quant Mirpour, K., Ong, W. S., & Bisley, J. W. (2010). Microstimu-
Biol, 55, 963–971. lation of posterior parietal cortex biases the selection of eye
Disney, A. A., Aoki, C., & Hawken, M. J. (2007). Gain modu- movement goals during search. J Neurophysiol, 104(6),
lation by nicotine in macaque V1. Neuron, 56, 701–713. 3021–3028.
Ekstrom, L. B., Roelfsema, P. R., Arsenault, J. T., Mitchell, J. F., Sundberg, K. A., & Reynolds, J. H. (2007).
Bonmassar, G., & Vanduffel, W. (2008). Bottom-up Differential attention-dependent response modulation
dependent gating of frontal signals in early visual cortex. across cell classes in macaque visual area V4. Neuron, 55,
Science, 321, 414–417. 131–141.
Ekstrom, L. B., Roelfsema, P. R., Arsenault, J. T., Kolster, Mitchell, J. F., Sundberg, K. A., & Reynolds, J. H. (2009).
H., & Vanduffel, W. (2009). Modulation of the contrast Spatial attention decorrelates intrinsic activity fluctuations
response function by electrical microstimulation of the in macaque area V4. Neuron, 63, 879–888.
macaque frontal eye field. J Neurosci, 29(34), 10683–10694. Monosov, I. E., Sheinberg, D. L., & Thompson, K. G. (2010).
Fries, P., Womelsdorf, T., Oostenveld, R., & Desimone, R. Paired neuron recordings in the prefrontal and inferotem-
(2008). The effects of visual stimulation and selective visual poral cortices reveal that spatial selection precedes object
attention on rhythmic neuronal synchronization in identification during visual search. Proc Natl Acad Sci USA,
macaque area V4. J Neurosci, 28, 4823. 107(29), 13105–13110.
Goard, M., & Dan, Y. (2009). Basal forebrain activation Monosov, I. E., & Thompson, K. G. (2009). Frontal eye field
enhances cortical coding of natural scenes. Nat Neurosci, 12, activity enhances object identification during covert visual
1444–1449. search. J Neurophysiol, 102(6), 3656–3672.
Goldberg, M. E., & Wurtz, R. H. (1972). Activity of superior Moore, T., & Armstrong, K. M. (2003). Selective gating of
colliculus in behaving monkey. I. Visual receptive fields of visual signals by microstimulation of frontal cortex. Nature,
single neurons. J Neurophysiol, 35(4), 542–559. 421, 370–373.
Gregoriou, G. G., Gotts, S. J., Zhou, H., & Desimone, R. Moore, T., & Fallah, M. (2001). Control of eye movements
(2009). High-frequency, long-range coupling between pre- and spatial attention. Proc Natl Acad Sci USA, 98,
frontal and visual cortex during attention. Science, 324, 1273–1276.
1207–1210. Muller, J. R., Philiastides, M. G., & Newsome, W. T. (2005).
Guillem, K., Bloem, B., Poorthuis, R. B., Loos, M., Smit, Microstimulation of the superior colliculus focuses atten-
A. B., Maskos, U. … Mansvelder, H. D. (2011). Nicotinic tion without moving eyes. Proc Natl Acad Sci USA, 102(3),
acetylcholine receptor b2 subunits in the medial prefrontal 524–529.
cortex control attention. Science, 333, 888–891. Mysore, S. P., & Knudsen, E. I. (2011). Flexible categoriza-
Herrero, J. L., Roberts, M. J., Delicato, L. S., Gieselmann, tion of relative stimulus strength by the optic tectum.
M. A., Dayan, P., & Thiele, A. (2008). Acetylcholine con- J Neurosci, 31, 7745–7752.
tributes through muscarinic receptors to attentional modu- Noudoost, B., Chang, M. H., Steinmetz, N. A., & Moore,
lation in V1. Nature, 454, 1110–1114. T. (2010). Top-down control of visual attention. Curr Opin
Ignashchenkova, A., Dicke, P. W., Haarmeier, T., & Neurobiol, 20, 183–190.
Thier, P. (2004). Neuron-specific contribution of the supe- Noudoost, B., & Moore, T. (2011a). Control of visual corti-
rior colliculus to overt and covert shifts of attention. Nat cal signals by prefrontal dopamine. Nature, 474, 372–375.
Neurosci, 7(1), 56–64. Noudoost, B., & Moore, T. (2011b). A reliable microinjec-
Lidow, M. S., Goldman-Rakic, P. S., Gallager, D. W., & trode system for use in behaving monkeys. J Neurosci Methods,
Rakic, P. (1991). Distribution of dopaminergic receptors 194, 218–223.
in the primate cerebral-cortex: Quantitative autoradio- Noudoost, B., & Moore, T. (2011c). The role of neuromodu-
graphic analysis using [3H]raclopride, [3H]spiperone and lators in selective attention. Trends Cogn Sci, 15, 585–591.
[3H]Sch23390. Neuroscience, 40, 657–671. Petersen, S. E., Robinson, D. L., & Keys, W. (1985). Pulvinar
Liu, Y., Yttri, E. A., & Snyder, L. H. (2010). Intention and nuclei of the behaving rhesus monkey: Visual responses and
attention: Different functional roles for LIPd and LIPv. Nat their modulation. J Neurophysiol, 54(4), 867–886.
Neurosci, 13(4), 495–500. Posner, M. I. (1980). Orienting of attention. Q J Exp Psychol
Lovejoy, L. P., & Krauzlis, R. J. (2010). Inactivation of A, 32, 3–25.
primate superior colliculus impairs covert selection of Saalmann, Y. B., Pinsk, M. A., Wang, L., Li, X., & Kastner,
signals for perceptual judgments. Nat Neurosci, 13(2), S. (2012). The pulvinar regulates information transmission
261–266. between cortical areas based on attention demands. Science,
Mazzoni, P., Bracewell, R. M., Barash, S., & Andersen, R. 337(6095), 753–756.
A. (1996). Motor intention activity in the macaque’s lateral Salinas, E., & Sejnowski, T. J. (2001). Correlated neuronal
parietal area. I. Dissociation of motor plan from sensory activity and the flow of neural information. Nat Rev Neurosci,
memory. J Neurophysiol, 76(3), 1439–1456. 2, 539–550.
McAdams, C. J., & Maunsell, J. H. R. (1999). Effects of atten- Santana, N., Mengod, G., & Artiga, F. (2009). Quantitative
tion on orientation-tuning functions of single neurons in analysis of the expression of dopamine D1 and D2 recep-
macaque cortical area V4. J Neurosci, 19, 431–441. tors in pyramidal and GABAergic neurons of the rat pre-
McAdams, C. J., & Reid, R. C. (2005). Attention modulates frontal cortex. Cereb Cortex, 19, 849–860.
the responses of simple cells in monkey primary visual Sarter, M., Hasselmo, M. E., Bruno, J. P., & Givens, B.
cortex. J Neurosci, 25, 11023–11033. (2005). Unraveling the attentional functions of cortical
McAlonan, K., Cavanaugh, J., & Wurtz, R. H. (2006). Atten- cholinergic inputs: Interactions between signal-driven and
tional modulation of thalamic reticular neurons. J Neurosci, cognitive modulation of signal detection. Brain Res Brain
26, 4444–4450. Res Rev, 48, 98–111.
noudoost, albarran, and moore: neural signatures, circuitry, and modulators 243
23 Computational Models of Attention
LAURENT ITTI AND ALI BORJI
ABSTRACT This chapter reviews recent computational models studies have shown that the majority of eye fixations
of visual attention. We begin with models for the bottom-up during execution of many tasks are directed to task-
or stimulus-driven guidance of attention to salient visual relevant locations that may or may not also be salient,
items, which we examine in seven different broad categories.
We then examine more complex models, which address the
and fixations are coupled in a tight temporal relation-
top-down or goal-oriented guidance of attention toward items ship with other task-related behaviors, such as reaching
that are more relevant to the task at hand. and grasping (Hayhoe, Shrivastava, Mruczek, & Pelz,
2003). Several of these studies have used naturalistic
A large body of psychophysical evidence on attention interactive or immersive environments to give high-
can be summarized by postulating two forms of visual level accounts of gaze behavior in terms of objects,
attention (James, 1981). The first is driven by the visual agents, “gist of the scene” (Potter & Levy, 1969;
input; this so-called exogenous, bottom-up, stimulus- Torralba, 2003), and short-term memory to describe,
driven, or saliency-based form of attention is rapid, for example, how task-relevant information guides eye
operates in parallel throughout the entire visual field, movements while subjects make a sandwich (Hayhoe
and helps mediate pop-out, the phenomenon by which et al., 2003; Land & Hayhoe, 2001), or how distractions
some visual items tend to stand out from their sur- such as setting the radio or answering a phone affect
roundings and to instinctively grab our attention (Itti eye movements while driving (Sodhi, Reimer, & Llamaz-
& Koch, 2001; Koch & Ullman, 1985; Treisman & ares, 2002). While these more complex attentional
Gelade, 1980). The second, endogenous, top-down, or behaviors have been more difficult to capture in com-
task-driven form of attention depends on the exact task putational models, we review below several recent
at hand and on subjective visual experience, takes efforts that have successfully modeled attention guid-
longer to deploy, and is volitionally controlled. Normal ance during complex tasks, such as driving a car or
vision employs both processes simultaneously to control running a hot-dog stand that serves many hungry
both overt and covert shifts of visual attention (Itti & customers.
Koch, 2001). Covert focal attention has been described While we focus on purely computational models
as a rapidly shiftable “spotlight” (Crick, 1984), which (which autonomously process visual data without the
serves the double function of selecting particular loca- requirement of a human operator, or of manual parsing
tions and objects of interest and of enhancing visual of the data into conceptual entities), we also point the
processing at those locations and for specific attributes reader to several relevant previous reviews on attention
of those objects. Thus, attention acts as a shiftable infor- theories and models more generally (Frintrop, Rome,
mation-processing bottleneck, allowing only objects & Christensen, 2010; Gottlieb & Balan, 2010; Itti &
within a circumscribed visual region to reach higher Koch, 2001; Paletta, Rome, & Buxton, 2005; Toet, 2011;
levels of processing and visual awareness (Crick & Koch, Tsotsos & Rothenstein, 2011).
1998).
Most computational models of attention to date have Computational models of bottom-up attention
focused on bottom-up guidance of attention toward
salient visual items. Many, but not all, of these models Development of computational models of attention
have embraced the concept of a topographic saliency started with the feature integration theory of Treisman
map (Koch & Ullman, 1985) that highlights scene loca- and Gelade (1980), which proposed that only simple
tions according to their relative conspicuity or salience. visual features are computed in a massively parallel
This has allowed for validation of saliency models manner over the entire visual field. Attention is then
against eye movement recordings, by measuring the necessary to bind those early features into a united
extent to which human or monkey observers fixate loca- object representation, and the selected bound repre-
tions that have higher predicted salience than expected sentation is the only part of the visual world that passes
by chance (Parkhurst, Law, & Niebur, 2002; Tatler, Bad- though the attentional bottleneck (figure 23.1A). Koch
deley, & Gilchrist, 2005). However, recent behavioral and Ullman (1985) extended the theory by proposing
WTA
Saliency map
Feature maps
C D
Input image
7 features
Feature Maps (6 maps per
feature)
Conspicuity Maps
Linear combinations
Saliency map
Inhibition of return
WTA
Central representation
Figure 23.1 Early bottom-up attention theories and models. computational models. They included many elements of the
(A) Feature integration theory of Treisman and Gelade Koch and Ullman framework and added new components,
(1980) posits several feature maps, and a focus of attention such as an alerting subsystem (motion-based saliency map)
that scans a map of locations and collects and binds features and a top-down subsystem (which could modulate the saliency
at the currently attended location (from Treisman & Souther, map based on memories of previously recognized objects).
1985). (B) Koch and Ullman (1985) introduced the concept (D) Itti et al. (1998) proposed a complete computational
of a saliency map receiving bottom-up inputs from all feature implementation of a purely bottom-up and task-independent
maps, where a winner-take-all (WTA) network selects the most model based on Koch and Ullman’s theory, including multi-
salient location for further processing. (C) Milanese, Wechsler, scale feature maps, saliency map, winner-take-all, and inhibi-
Gill, Bost, and Pun (1994) provided one of the earliest tion of return.
multiscale contrast and color spatial-distribution; R*: lumi- target patch receives higher saliency than a randomly chosen
nance, contrast, luminance-bandpass, contrast-bandpass; SM*: negative one); DR means models have used a measure of
orientation and motion; J*: CIO, horizontal line, face, people detection/classification rate to determine how successful the
detector, gist, etc.; S*: color matching, depth and lines; and model was. PR stands for precision recall. In the data set ( f13)
:) represents the face. In the model type ( f11) column, R column: self data means that authors gathered their own data.
means that a model is based on reinforcement learning. In For a detailed definition of these factors, please refer to Borji
the measures ( f12) column, K* means the Wilcoxon-Mann- and Itti (2012b).
Whitney test was used (the probability that a randomly chosen
Figure 23.3 Examples of recent top-down models. prediction. Models compared are as follows. MEP: mean eye
(A) Model of Ban et al. (2010), which integrates bottom-up position over all frames during the game play (control model).
and top-down components; r, g, b: red, green, and blue color G: trivial Gaussian map at the image center. BU: bottom-up
channels. I: intensity feature. E: edges. R, G: red-green color. saliency map of the Itti model. Mean BU: average saliency
B, Y: blue-yellow color. CSD&N: center-surround differences maps over all video frames. REG(1): regression model that
and normalization. ICA: independent component analysis. maps the previous attended object to the current attended
GFT_ART: growing fuzzy topology adaptive resonance theory. object and fixation location. REG(2): similar to REG(1), but
SP: saliency point. (B) Graphical representation of the the input vector consists of the available objects at the scene
dynamic Bayesian network (DBN) approach of Borji et al. augmented with the previously attended object. SVM(1) and
(2012b), unrolled over two time-slices. Xt is the current SVM(2) correspond to REG(1) and REG(2) but using a
saccade position, Yt is the currently attended object, and Ft i support vector machine (SVM) classifier. Similarly, DB(5) and
is the function that describes object i at the current scene. All DB(3) correspond to REG(1) and REG(2), meaning that in
variables are discrete. It also shows a time-series plot of prob- DB(5) the network considers just one previously attended
ability of objects being attended and a sample frame with object, while in DB(3) each network slice consists of the previ-
tagged objects and eye fixation overlaid. (C) Sample pre- ously attended object as well as information about the previ-
dicted saccade maps of the DBN model (shown in B) on three ous objects in the scene. REG(Gist): regression based only on
video games and tasks: running a hot-dog stand (HDB; top the gist of the scene. kNN: k-nearest-neighbors classifier.
three rows), driving (3DDS; middle two rows), and flight Rand: white noise random map (control). Overall, DB(3)
combat (TG, bottom two rows). Each red circle indicates the performed best at predicting where the player would look
observer’s eye position superimposed with each map’s peak next (Borji et al., 2012b). (See color plate 21.)
location (blue squares). Smaller distance indicates better
aspects of attention computationally. Tens of new While many models have approached the problem of
models have been developed, each bringing new modeling top-down attention, a fully implemented cog-
insights into the question of what makes some nitive system that reasons about objects, their relation-
stimuli more important to visual observers than other ships, and their locations to guide the next shift of
stimuli. attention remains an elusive goal to date.
25 RIEKE 271
26 FAIRHALL 283
27 OLSHAUSEN 295
29 SCHNEIDMAN 319
30 RUST 337
ABSTRACT The evolution of sensory systems is driven by the The design of artificial systems is sometimes inspired
need to survive and reproduce in a naturalistic sensory envi- by biological systems. This represents an additional
ronment. However, neural architectures are also shaped by motivation to seriously consider the constraints on bio-
their own internal constraints. These constraints include the
evolutionary-developmental constraints that restrict the pos-
logical systems. Because artificial systems are not subject
sible effects of mutations, as well as stochastic developmental to many of the constraints discussed here, an imple-
noise. Neural systems are further limited by metabolic costs, mentation that works well in biology may be suboptimal
particularly the cost of maintaining ionic gradients. Finally, in an artificial system, and thus engineers must be
neural systems are limited by the properties of their own careful not to draw the wrong lessons from biological
electronics—namely, the existence of intrinsic electrical
systems (Stafford, 2010).
noise, the limited speed of signal propagation, the restricted
dynamic range of synapses and firing rates, and the quasi- This chapter reviews the key constraints on neural
linear nature of synaptic integration. Importantly, many of systems and their implications for sensory processing.
these internal constraints are fundamentally in conflict with Organisms have evolved intriguing ways to cope with
each other, insofar as they pressure evolution in opposite some of these constraints. Such cases are generally
directions. Thus, neural systems are shaped by the need to
clearer in sensory neuroscience than in other branches
find a satisfactory trade-off between competing factors. Many
such trade-offs are common to many sensory modalities, and of neuroscience, simply because the function of sensory
they are shared by organisms as different as flies and humans. systems is particularly transparent. For this reason,
As such, they can help explain why some features of neural sensory systems are a good setting for investigating how
systems are also shared. neural systems evolve in the context of constraints.
“The enemy of art is the absence of limitations.”
—Orson Welles Evolutionary inheritance
The ability of neural systems to process sensory infor- Neural systems in related species have a similar organi-
mation is subject to internal constraints. A constraint zation. This is true even in species that inhabit very
(as defined here) is a limitation on the performance of different ecological niches. For example, the relative
a sensory system that arises from the intrinsic properties size of major brain divisions is remarkably constant
of the neural system itself. This is distinct from limita- across mammals. A meta-analysis of 131 species showed
tions arising from the sensory environment, or the that the volumes of all major brain divisions (including
nature of the behavioral task. the medulla, hippocampus, cerebellum, striatum, and
Any given internal constraint may not directly limit neocortex) were highly systematically related to total
performance, because it may be possible to compensate brain volume, except the olfactory bulb. Different
for the constraint. However, compensation may push brain divisions showed different relationships to brain
the nervous system toward a particular architecture, volume—in particular, neocortex grew particularly
rather than other architectures that might hypotheti- steeply with increasing volume—but for all brain divi-
cally produce better performance in the absence of this sions there was a systematic dependence on brain
constraint. Thus, constraints often limit performance volume that extended to species with widely varying
indirectly, by pressuring neural systems toward compro- body sizes and lifestyles. Notably, this analysis included
mise architectures. A compromise may be optimal, or species as diverse as simians, prosimians, insectivores,
it may simply be good enough to permit the organism and bats. This result argues that the expansion of brain
to survive and reproduce (Marder & Goaillard, 2006; volume resulting from natural selection for any behav-
Rieke, 2014). ioral trait is constrained to be a coordinated growth of
The constraints discussed here are common to all the entire nonolfactory brain (Finlay & Darlington,
sensory systems. Moreover, they are common to very 1995). This finding raises a provocative question: might
different organisms. As such, they can potentially there be excess signaling capacity in some brain divi-
explain why certain neural architectures occur again sions, as a by-product of a strong pressure to expand
and again: they are good compromises. other divisions?
Output
duces total population firing rates that are relatively
equal for different stimuli, as compared with a scenario
where there is no compressive nonlinearity. Again
drawing an example from the Drosophila olfactory
0 0 system, it is notable that there is a compressive nonlin-
0 Channel 2 0 Input earity in the relationship between the odor responses
of olfactory receptor neurons and postsynaptic second-
C After within-channel D After divisive order neurons. Implementing this nonlinearity in data-
compressive nonlinearity normalization
driven simulations makes it easier to construct linear
classifiers that respond sparsely and selectively to single
odors (figure 24.2; Luo et al., 2010; Olsen et al., 2010).
Channel 1
Channel 1
0
0
0
0
Conclusion
Channel 2 Channel 2
Figure 24.2 Compression, normalization, and linear sepa- To be a film director, in the words of Orson Welles,
rability. (A) Consider a sensory representation in multiple is simply to “preside over accidents.” This could stand
dimensions, with each dimension corresponding to a coding as a description of how a nervous system evolves and
channel (a neuron or a chain of neurons). For concreteness, develops—accident by accident. Neural systems are
we might imagine the two-dimensional space in this schematic
subject to constraints, and this influences which random
corresponds to two different types of olfactory receptor
neurons. Every odor stimulus is a point in this space. A subset variations are passed on to successive generations. As
of points are highlighted for comparison across panels (black we have seen, the history of which accidents survive
circles). (B) A compressive nonlinearity transforms signals becomes another constraint on neural system architec-
within each channel (black curve). Divisive normalization ture, in the form of the organism’s evolutionary inheri-
adjusts the steepness of this nonlinearity according to the tance and developmental programs.
total input to both channels. Namely, when total activity is
high, the nonlinearity is less steep (gray curve; Carandini & The central argument of this chapter is that internal
Heeger, 2012). Different stimuli will therefore fall on differ- constraints leave their imprint on the architecture of
ent curves, because they produce different amounts of total neural systems. Because these constraints are ubiqui-
input to both channels. Curves in this schematic are fit to data tous, they can potentially explain why some architec-
recorded from Drosophila olfactory neurons in vivo (Olsen tures are so common. Understanding how this might
et al., 2010). (C) A steep compressive nonlinearity (black
curve in B) preferentially boosts weak input signals (compare
occur will require a comparative approach that embraces
to A). (D) Normalization tends to equalize the distance of all a variety of sensory modalities and organisms.
representations from the origin. As a consequence, stimuli Thinking about constraints is more important than
tend to lie near the surface of a circle. The vector w corre- ever before, because the field of experimental neuro-
sponds to the weights on the two input channels that define science is undergoing a revolution in techniques. New
a linear separation (dashed line) between one highlighted
techniques allow us to precisely perturb neural activity,
symbol and the other symbols. Note that, in this example, we
can separate each highlighted symbol from the rest using only and to test how this affects perception and behavior. In
nonnegative weights. In the Drosophila olfactory system, there practice, one often begins with a specific element of a
are 50 coding channels (glomeruli), as compared to the two neural system (e.g., a cell type) and one searches for
channels in this schematic. This system has a large capacity behaviors that fall apart when this element is per-
for generating sparse and selective representations, and com-
turbed. This search assumes we will understand the
pression and normalization together improve this capacity
(Luo et al., 2010; Olsen et al., 2010). function of each element by identifying the behaviors
that rely on it. However, the nature of the behavioral
task is only one pressure that drives the evolution of
neural systems: equally important is the pressure to
rieke: are neural circuits the best they can be? 271
A 1000 events/integration time (dim interior) priate metric to evaluate performance and incorporat-
1100 ing key hardware constraints. In the best case, both
0.10
provide parameter-free predictions about how well a
Probability
Count
0.05 neural system can perform a given task and how it could
achieve this limiting performance.
900 0.00 Ideal observer or optimization approaches provide
0 50 100 900 1000 1100
key tools to compare performance across different
levels of a system. Thus the limiting performance set by
B 10 events/integration time (dusk)
signals at one level of a system can be compared with
20
the actual performance realized at a later stage. These
Probability
0.10
two levels could be, for example, light inputs to the eye
Count
10
0.05 and behavior (reviewed by Donner, 1992; Field,
Sampath, & Rieke, 2005; Geisler, 2003, 2011), or pho-
0 0.00
0 50 100 0 10 20
toreceptor signals and retinal output (reviewed by Field
et al., 2005), or pre- and post-synaptic signals. Ideal
C 0.1 events/integration time (moonlight) observer analysis provides a natural progression in our
1 1.0
understanding of a system. Thus, initial applications
often focus on overall performance (e.g., a comparison
Probability
Count
spontaneous activation of an enzyme). Below we focus Vision in starlight and photon detection
on one such source of noise created by spontaneous
Vision at low light levels exemplifies how a focus on
activation of the same photopigment molecules that
optimal design can guide investigation of the relevant
absorb photons and initiate the visual process; sponta-
neural circuits. Specifically, the approach of behavioral
neous activation of these molecules generates photon-
performance to fundamental limits has led to a set of
like noise events that provide an important source of
precise questions and constraints about how the under-
intrinsic noise in the responses of rod photoreceptors.
lying computations work. Many of the issues have been
reviewed in detail previously (Donner, 1992; Field et al.,
Hardware constraints and the view of the ideal 2005).
observer
Optimization approaches are closely related to ideal Probing Behavioral Sensitivity It has been known
observer analysis (Geisler, 2003, 2011; Green & Swets, for more than 100 years that dark-adapted human
1966), which determines the optimal performance observers can detect a small number of absorbed
obtainable from observing signals at one level of a photons. The experiments described below support a
system. Optimal performance should not be confused picture in which visual sensitivity is limited as much by
with perfect performance. Instead, noise in sensory the division of light into discrete photons and the
inputs or in neural responses (e.g., those described in unavoidable statistical fluctuations in photon absorp-
the previous section) will set a limit to per formance— tion as by biological noise or inefficiency. These obser-
the limit identified by ideal observer analysis. Ideal vations about behavioral fidelity motivate making and
observer analysis and optimization approaches share testing predictions about the operation of key compo-
similar challenges—in particular, identifying an appro- nents of the neural circuits supporting night vision.
Fraction seen
analyzed assuming that all flashes producing more than
0.5
a threshold number of absorbed photons were seen,
and that the number of absorbed photons was described
by Poisson statistics. In this case, the probability of
seeing is simply the probability that a flash of a given 0.0
strength produces more than the threshold number of
1.0
absorbed photons. This analysis does not incorporate
rieke: are neural circuits the best they can be? 273
noise or true photon absorption—are available to cor- We start with a seemingly innocuous question about
tical circuits involved in the decision of whether the photon absorption: what fraction of the incident
flash was seen. In the second, a thresholding nonlin- photons should a rod capture to optimize sensitivity? A
earity in the visual circuitry limits information about reflexive answer is that maximal sensitivity is achieved
individual photon-like events. The first model makes a by capturing as many incident photons as possible, but
clear prediction about how detection threshold and as we will see that answer is wrong (see Bialek, 2012,
the rate of false-positive responses interact (figure for more details). The argument is based on two prop-
25.2B, Poisson noise curve). The actual decrease in erties of rhodopsin: self-screening and spontaneous
perceptual threshold with increased false-positive activation.
rate falls short of this prediction, indicating that the To get some intuition for self-screening, approximate
“every photon” model is unlikely to be strictly correct the outer segment of a rod photoreceptor as a cylinder
(Koenig & Hofer, 2011). This is consistent with a filled with a uniform concentration of rhodopsin mol-
thresholding nonlinearity that eliminates some ecules (figure 25.3). As photons travel along the long
responses to absorbed photons. axis of the cylinder, some of them are absorbed. This
The ability to express the noise that limits behavior means that the density of photons at the location of a
as a rate of equivalent photon-like events is convenient, given rhodopsin molecule depends on its position
but does not specify where the noise originates. Barlow within the cylinder; specifically, the photon density
suggested that internal noise in fact resulted from the declines exponentially from the base to the tip of the
thermal activation of rhodopsin molecules in the rods cylinder. The probability of photon absorption by a
(Barlow, 1956). Subsequent recordings from single-rod rhodopsin molecule at location x depends on the
photoreceptors revealed discrete noise events associ- photon density at the molecule’s position. As a
ated with spontaneous rhodopsin activation (Baylor,
Matthews, & Yau, 1980; Baylor, Nunn, & Schnapf, 1984).
These noise events are indistinguishable from events
produced by photon absorption. They occur about
once every 100–200 sec in a mammalian rod, a rate
quite close to that required to account for the noise-
limiting behavioral sensitivity, although the connection 1
Normalized amplitude
cross-section, 70% absorption is achieved for a rod ∼60 Normalized amplitude 0.8
rieke: are neural circuits the best they can be? 275
phototransduction process, specifically on the rate of likely generating single-photon responses while reject-
spontaneous phosphodiesterase activation. ing noise from the remaining rods (Baylor et al., 1984;
Figure 25.4 explores these issues using a model for van Rossum & Smith, 1998).
the phototransduction cascade based on measurements Similar issues of how to integrate sparse, noisy inputs
in toad rods. Figure 25.4B plots the noise variance and recur in many other neural circuits. Photon detection
response integration time as the rate of spontaneous in the retina offers several key advantages for investigat-
phosphodiesterase activation is increased. High phos- ing this general issue. First, the signal and noise proper-
phodiesterase activity produces a brief response, but ties of the rod’s single-photon responses can be
also produces so much noise that individual single- thoroughly characterized. This permits construction of
photon responses are not identifiable (top right). Low parameter-free predictions of the optimal strategies for
phosphodiesterase activity produces little noise, but separating rod signal and noise. Second, separation of
responses last tens of seconds (top left). The measured single-photon responses from noise is effective only
rate of spontaneous phosphodiesterase activation places prior to mixing of rod signals. This restricts the possible
the actual rod behavior at a location in the middle. anatomical locations for such signal processing to sites
Evaluating whether or not this is a good compromise, at or prior to the synapse from rods to rod bipolar cells.
however, requires some means of weighing the relative Thus we can generate parameter-free predictions about
costs of response duration and noise, and it is not clear the properties, location, and functional importance of
how to do so. This example highlights a central chal- a key signal-processing step supporting night vision.
lenge facing ideal observer calculations: the task The predictions about processing of rod responses
facing the system is often multifaceted, and weighing have been tested by recording from rod bipolar cells
different aspects of system function can be difficult or (Berntson, Smith, & Taylor, 2004; Field & Rieke, 2002).
impossible. Responses of mouse rod bipolar cells indeed differ
substantially from those of the rods from which they
Retinal Readout Rod-mediated signals traverse the receive input. In particular, rod bipolar responses
mammalian retina through the specialized rod bipolar depend supralinearly on flash strength, and are more
circuit (reviewed by Bloomfield & Dacheux, 2001; Field discrete than responses of the rods. These properties
et al., 2005). Behavioral sensitivity requires that this are consistent with the presence of a threshold-like
circuit operate effectively when <0.1% of the rods nonlinearity at the synapse between rods and rod
absorb a photon within the 100–200 ms integration bipolar cells. Further, the synaptic nonlinearity has
time of rod signals. This sensitivity poses substantial properties closely aligned to those predicted from the
constraints. First, rod photoreceptors must respond to rod signal and noise. Surprisingly, the synapse discards
single photons. Second, the resulting signals must be many of the rod’s single-photon responses. This is,
reliably transmitted across the retina and protected however, a good strategy when photon absorptions are
from noise. Third, absorption of one or a few photons exceedingly rare, as the change in current produced
spread across many rods must noticeably impact the by a small or average single-photon response is more
spike responses sent from the retina to the brain. These likely to have originated from noise from spontaneous
constraints have guided investigation of the underlying phosphodiesterase activity (figure 25.4) than from a
mechanisms, and the resulting focus has provided mul- true photon absorption.
tiple insights into circuit function. Thresholding of the rod signals occurs through a
At light levels near perceptual threshold, photons simple synaptic mechanism (Sampath & Rieke, 2004).
arrive rarely at individual rods, but all of the rods gener- Rods release glutamate continuously in the dark, and
ate noise. This noise threatens to overwhelm the sparse decrease this release during the single-photon response.
single-photon responses when signals from multiple Rod bipolar cells sense glutamate released by the rods
rods are combined. How should downstream retinal via metabotropic receptors that control ion-channel
neurons combine signals across the rods from which activity through a second-messenger cascade. Gluta-
they receive input? This is a situation in which averaging mate release from the rods saturates this signaling
is a disaster. Averaging is a good strategy when both cascade; as a consequence, small changes in rod voltage
signal and noise are spread uniformly across receptors. and glutamate release fail to modulate the rod bipolar
But when the signal is sparse, averaging inextricably voltage. Larger changes in glutamate release—such as
mixes signal from a few receptors with noise from those associated with a large single-photon response—
receptors that do not carry any signal. Instead, sensitiv- relieve this saturation and produce a response. An
ity can be dramatically improved by a nonlinear mecha- interesting and unresolved issue is how the level of
nism that selectively retains signals from those rods synaptic saturation is controlled.
rieke: are neural circuits the best they can be? 277
d+ d– The optics of the eye blur the lines in the Vernier task
of figure 25.5 so that they are spread over many cones
(Westheimer, 1984). Thus the distribution of mean
cone photon fluxes (without any noise) resembles that
in figure 25.5 (second panel from top). The asymmetry
in the distribution—particularly the different depths of
the minima corresponding to the regions between the
center and flanking lines—is indicative of the displace-
ment of the central line; from this simple picture it is
clear that, without noise, an ideal observer could detect
exceedingly small displacements even given the finite
cone sampling and optical blur. Figure 25.5 (second
panel from bottom) depicts these distributions on a
single 0.6 sec trial of such a task for a displacement of
1/20th of the cone spacing; noise due to quantal fluc-
tuations is included. This illustrates the quality of signal
available to an ideal observer of the photon flux at the
cone photoreceptors.
Human behavioral performance in a Vernier task like
that in figure 25.5 falls short of that possible given the
Mean photon flux photon flux at the cones. The difference between actual
and ideal performance is a factor of 3–10 depending
on details of the spatial configuration (Banks, Sekuler,
& Anderson, 1991). A similar difference between per-
Population response
rieke: are neural circuits the best they can be? 279
vast array of stimulus conditions under which such mea- Barlow, H. B. (1956). Retinal noise and absolute threshold.
surements can be made. Further complicating matters, J Opt Soc Am, 46(8), 634–639.
Barlow, H. B. (1972). Dark and light adaptation: Psychophys-
models that characterize responses for one set of stimu-
ics. Handbook of sensory physiology, 8(4), 1–24.
lus conditions rarely generalize well to describe Baylor, D. A., Matthews, G., & Yau, K. W. (1980). Two
responses for different stimulus conditions. What, then, components of electrical dark noise in toad retinal rod
is a neuroscientist to do? outer segments. J Physiol, 309, 591–621.
Predictions based on the idea that neural circuits Baylor, D. A., Nunn, B. J., & Schnapf, J. L. (1984). The
approach ideal performance provide one way to navi- photocurrent, noise and spectral sensitivity of rods of the
monkey Macaca fascicularis. J Physiol, 357, 575–607.
gate these complexities. This chapter has focused on Berntson, A., Smith, R. G., & Taylor, W. R. (2004). Trans-
illustrating how this approach can be applied to the mission of single photon signals through a binary synapse
operation of the retina in the context of simple detec- in the mammalian retina. Vis Neurosci, 21(5), 693–702.
tion tasks. Limits to performance in these tasks due to Bialek, W. (1987). Physical limits to sensation and percep-
unavoidable stimulus variations or early sensory noise tion. Annu Rev Biophys Biophys Chem, 16, 455–478.
Bialek, W. S. (2012). Biophysics: Searching for principles. Princ-
are particularly clear, and the computations required to eton, NJ: Princeton University Press.
realize such performance provide experimentally test- Bialek, W., & Owen, W. G. (1990). Temporal filtering in
able predictions about the relevant computational retinal bipolar cells. Elements of an optimal computation?
dynamics of retinal circuits. Biophys J, 58(5), 1227–1233.
There are many reasons such optimization approaches Bloomfield, S. A., & Dacheux, R. F. (2001). Rod vision:
Pathways and processing in the mammalian retina. Prog
could fail. Calculations of the performance of an ideal
Retin Eye Res, 20(3), 351–384.
observer necessarily focus on simple measures of system Borghuis, B. G., Sterling, P., & Smith, R. G. (2009). Loss
performance; in reality, a host of factors, many unknown, of sensitivity in an analog neural circuit. J Neurosci, 29(10),
are certain to have imposed evolutionary pressures that 3045–3058.
have shaped circuit operation. Indeed, other aspects of Brivanlou, I. H., Warland, D. K., & Meister, M. (1998).
Mechanisms of concerted firing among retinal ganglion
the operation of the retina can be predicted from argu-
cells. Neuron, 20(3), 527–539.
ments based on minimizing energy expenditure or Curcio, C. A., Sloan, K. R., Kalina, R. E., & Hendrickson,
wiring length. Thus it is somewhat surprising that pre- A. E. (1990). Human photoreceptor topography. J Comp
dictions based on optimizing a simple performance Neurol, 292(4), 497–523.
metric are ever successful. The fact that they can be DeVries, S. H. (1999). Correlated firing in rabbit retinal gan-
indicates some order in the apparent complexity of glion cells. J Neurophysiol, 81(2), 908–920.
Donner, K. (1992). Noise and the absolute thresholds of cone
neural computation. and rod vision. Vis Res, 32(5), 853–866.
Field, G. D., & Rieke, F. (2002). Nonlinear signal transfer
ACKNOWLEDGMENTS I thank Juan Angueyra and Rachel Wilson
from mouse rods to bipolar cells and implications for visual
for many helpful comments, Darren Koenig and Heidi Hofer sensitivity. Neuron, 34(5), 773–785.
for providing data for figure 25.2, and past and present Field, G. D., Sampath, A. P., & Rieke, F. (2005). Retinal pro-
members of the laboratory for helping develop and refine cessing near absolute threshold: From behavior to mecha-
many of the ideas here. nism. Annu Rev Physiol, 67, 491–514.
Geisler, W. S. (2003). Ideal observer analysis. In L. Chalupa
& J. Werner (Eds.), The visual neurosciences (pp. 825–837).
REFERENCES
Cambridge, MA: MIT Press.
Aho, A. C., Donner, K., Hyden, C., Larsen, L. O., & Reuter, Geisler, W. S. (2011). Contributions of ideal observer theory
T. (1988). Low retinal noise in animals with low body tem- to vision research. Vis Res, 51(7), 771–781.
perature allows high visual sensitivity. Nature, 334(6180), Green, D. M., & Swets, J. A. (1966). Signal detection theory and
348–350. psychophysics. New York, NY: Wiley.
Ala-Laurila, P., Greschner, M., Chichilnisky, E. J., & Greschner, M., Shlens, J., Bakolitsa, C., Field, G. D.,
Rieke, F. (2011). Cone photoreceptor contributions to Gauthier, J. L., Jepson, L. H., … Chichilnisky, E. J.
noise and correlations in the retinal output. Nat Neurosci, (2011). Correlated firing among major ganglion cell types
14(10), 1309–1316. in primate retina. J Physiol, 589(Pt 1), 75–86.
Angueyra, J. M., & Rieke, F. (2013). Origin and effect of Hecht, S., Shlaer, S., & Pirenne, M. H. (1942). Energy,
phototransduction noise in primate cone photoreceptors. quanta, and vision. J Gen Physiol, 25, 819–840.
Nat Neurosci, 16(11), 1692–1700. Klein, S. A., & Levi, D. M. (1985). Hyperacuity thresholds of
Banks, M. S., Geisler, W. S., & Bennett, P. J. (1987). The 1 sec: Theoretical predictions and empirical validation.
physical limits of grating visibility. Vis Res, 27(11), J Opt Soc Am A, 2(7), 1170–1190.
1915–1924. Koenig, D., & Hofer, H. (2011). The absolute threshold of
Banks, M. S., Sekuler, A. B., & Anderson, S. J. (1991). cone vision. J Vis, 11(1), 1–24.
Peripheral spatial vision: Limits imposed by optics, photo- Luo, D. G., Yue, W. W., Ala-Laurila, P., & Yau, K. W. (2011).
receptors, and receptor pooling. J Opt Soc Am A, 8(11), Activation of visual pigments by light and heat. Science,
1775–1787. 332(6035), 1307–1312.
rieke: are neural circuits the best they can be? 281
26 Adaptation and Natural
Stimulus Statistics
ADRIENNE FAIRHALL
ABSTRACT In trying to understand the structure and dynamics neuron or neural system’s response to a specific stimu-
of sensory neural representations, it has been productive to lus often reduces over time. While this is often thought
examine the complex structure of natural images to discover of as a decay in the response, it can be interpreted as
the nature of the task that the nervous system performs. Two
different forms of adaptation help sensory systems to handle
part of the dynamical coding properties of the neural
the large variations and multiple spatial and temporal scales system. Adaptation is also used to describe more general
of natural inputs. Slow changes in excitability can be under- nonstationarities in the neural code: shifts in the form
stood as an encoding of slowly varying stimulus properties that of neural representation over time. These shifts often
highlights change, while adaptive changes to neural coding appear to allow neural systems to represent inputs more
strategies lead to more efficient representations of stimuli
efficiently.
drawn from time-varying distributions.
arrangements that can be viewed from a range of dis- self-similar and exhibit a power-law spectrum (Catrakis
tances and that exist in a wide range of sizes (figure & Dimotakis, 1996).
26.1C), images exhibit self-similarity: as one zooms Not all of the structure in a natural scene is captured
through different scales, the statistical composition of by the power spectrum: one can easily construct a
the scene is relatively invariant (Ruderman & Bialek, random image with a 1/f 2 spectrum that has no object-
1994; Simoncelli & Olshausen, 2001). For an image like properties (figure 26.1F). The existence of higher-
with intensity as a function of space given by I(x, y) we order correlations is apparent when one examines the
can compute the image’s autocorrelation function, distribution of two-point differences. If images were
C(δx, δy) = I(x + δx, y + δy)I(x, y), where the average is fully explained by the mean and power spectrum, these
over all locations x and y. As images are translation- distributions would be Gaussian; however, they are typi-
invariant, one can equivalently compute this function cally strongly peaked at zero and have long tails (figure
in the Fourier domain to obtain the power spectrum. 26.1G), showing that higher-order moments, or equiva-
This two-dimensional function can be reduced to a lently, multipoint correlations, are needed to fully
function of spatial frequency f by averaging over phase. describe such images.
For natural images, this function behaves as a power Despite the overall self-similarity, natural inputs often
law, 1/f α, where the exponent α is very close to 2 have a separation of meaning between large-scale and
(figure 26.1H; Ruderman & Bialek, 1994; Tolhurst, small-scale features (Attias & Schreiner, 1998; Joris, Sch-
Tadmor, & Chao, 1992). The power-law form of the reiner, & Rees, 2004). Large-scale features might be
power spectrum is likely a result of the scale invariance termed objects, whereas small-scale features may be
of natural scenes; the particular exponent of 2 may be termed textures; features on these different scales convey
a result of the prevalence of edges, as edges themselves different types of information. Here, “scale” can refer
have a power spectrum of 1/f 2. This property is not to either time or space. For example, the large-scale
unique to vision: under ideal conditions, odorants that features that identify object boundaries are filled with
are advected by a turbulent airflow also become finer-scale textural structure that conveys distinct
recent past of the stimulus (for example, the fact that 26.2A. If, as is often observed, the output activity
it just underwent a step, or that it has been constant for responds transiently to changes in the stimulus, f (τ) is
some time), one can approximate it as some function f likely to have a differentiating nature, as in f2 in figure
of the recent history of the stimulus: 26.2A. Generally, one might expect the response to
R (t ) = f (s (t ′ ≤ t )) . have a sensitivity to both the recent mean stimulus and
its derivative, so that f (τ) might be a composition of an
In the simplest case, this response might be a linear averaging filter and a differentiating filter. If the
function: response is truly approximately linear, these compo-
∞ nents will simply add. It is likely, however, that the
R (t ) = f ∗ s = ∫ dτf ( τ ) s(t − τ), (4) response is a nonlinear function of this weighted sum:
0
ABSTRACT Although the idea of thinking of perception as an basis of “deep learning” models in machine learning
inference problem goes back to Helmholtz, it is only recently and have achieved unprecedented success on both
that we have seen the emergence of neural models of percep- image- and speech-recognition benchmarks.
tion that embrace this idea. Here I describe why inferential
computations are necessary for perception, and how they go
Might such hierarchical, feedforward processing
beyond traditional computational approaches based on models provide insight into what is going on in the
deductive processes such as feature detection and classifica- multiple areas of sensory cortex? I shall argue here that
tion. Neural models of perceptual inference rely heavily upon despite the strong parallels between these models and
recurrent computation in which information propagates both cortical anatomy and physiology, these models are still
within and between levels of representation in a bidirectional
missing something fundamental. The problem lies not
manner. The inferential framework shifts us away from think-
ing of “receptive fields” and “tuning” of individual neurons, just with their computational architecture, but also the
and instead toward how populations of neurons interact via class of problems they have been designed to solve.
horizontal and top-down feedback connections to perform Namely, benchmark tasks such as image or speech rec-
collective computations. ognition—while appearing to capture human percep-
tual capabilities—define the problem of perception too
One of the vexing mysteries facing neuroscientists in narrowly. Perception involves much more than a passive
the study of perception is the plethora of intermediate- observer attaching labels to images or sounds. Arriving
level sensory areas that lie between low-level and high- at the right computational framework for modeling
level representations. Why in visual cortex do we have perception requires that we consider the wider range
a V2, V3, and V4, each containing a complete map of of tasks that sensory systems evolved to solve.
visual space, in addition to V1? Why S2 in addition to So, what are these tasks? What do animals use their
S1 in somatosensory cortex? Why the multiple belt senses for? Answering these questions is a research
fields surrounding A1 in auditory cortex? problem in its own right. One thing we can say with
A common explanation for this organization is that certainty is that visual systems did not start out process-
multiple stages of processing are needed to build pro- ing high-definition images, and auditory systems did
gressively more complex or abstract representations of not start out with well-formed cochleae providing time-
sensory input, beginning with neurons signaling pat- frequency analysis of sound. Rather, sensory systems
terns of activation among sensory receptors in lower began with crude, coarse-grained sensors attached to
levels and culminating with representations of entire organisms moving about in the world. Visual systems,
objects or properties of the environment in higher for example, began with simple light detectors situated
areas. For example, numerous models of visual cortex in the epithelium. Remarkably, over a relatively short
propose that invariant representations of objects are period of time (estimated to be 500,000 years), they
built up through a hierarchical, feedforward processing evolved into the wide variety of sophisticated eye designs
architecture (Fukushima, 1980; Riesenhuber & Poggio, we see today (Nilsson & Pelger, 1994). What was the
1999; Wallis & Rolls, 1997). Each stage is composed of fitness function driving this process? Presumably, it was
separate populations of neurons that perform feature the ability to plan useful actions and predict their out-
extraction and spatial pooling, with information flowing comes in complex, three-dimensional (3D) environ-
from one stage to the next, as shown in figure 27.1. The ments. For this purpose, performance at tasks such as
idea here is that each successive stage learns progres- navigation or judging scene layout is crucially impor-
sively more complex features of the input that are built tant. From an evolutionary perspective, the problem of
upon the features extracted in the previous stage. By “recognition”—especially when distilled down to one of
pooling over spatial position at each stage, one also classification—may not be as fundamental as it seems
obtains progressively more tolerance to variations in the introspectively to us humans.
positions of features, culminating in object-selective The greater problem faced by all animals is one of
responses at the top level that are invariant to variations scene analysis (Lewicki, Olshausen, Surlykke, & Moss,
in the pose of an object. Such networks now form the 2014). It is the problem of taking incoming sensory
Figure 27.1 Hierarchical feedforward processing model of object-selective responses at the top stage that are invariant to
visual cortex. Each stage contains separate populations of variations in pose.
neurons for feature extraction and pooling, resulting in
T O
S
S I S B
B T
P
A
Figure 27.2 Components of scene analysis. The scene itself contain multiple levels of representation and analysis. The
contains not just a single target object, but other objects, level of “intermediate features” is where inferential processes
terrain, and background, all of which may be important for come into play (Lewicki et al., 2014).
behavior. The neural structures enabling scene analysis
information and interpreting it in terms of what it or the pose of an object—in an analog manner that
conveys about the surrounding environment: terrain, captures the components of a scene and how one might
obstacles, and navigable surfaces or routes, in addition act upon them. It is at this intermediate level where
to specific objects of interest and their pose and posi- inferential processes come into play.
tion within the scene. In contrast to a simple feedfor- In this chapter, I shall describe why inference pro-
ward processing pipeline in service to the single goal of vides a suitable computational framework for percep-
classification, scene analysis involves multiple types of tion, the basic computations it entails, and specific
representation with different functions (figure 27.2). models that have been proposed for how it is instanti-
Most actions (the interesting ones we care about) are ated in neural systems. As we shall see, the inferential
not simply reflexive behaviors in direct response to framework forces us to consider neural architectures
sensory messages. Rather, they depend on goals, behav- other than the standard feedforward processing pipe-
ioral state, and the past history of what has occurred line. An open problem, though, is to understand the
(memory). In other words, meaningful behavior relation between perception and action, and how infer-
requires having a model of the world and one’s place ential computations fit into the larger framework of
in it. The model needn’t be particularly detailed— sensorimotor systems.
indeed, what aspects of the environment are modeled, For other excellent reviews of “perception as infer-
and to what degree of accuracy, is an important empiri- ence” from the perspective of psychophysics, see Knill
cal question—but we may reasonably assume that it and Richards (1996), Kersten, Mamassian, and Yuille
must be assimilated from diverse types of sensory input (2004), Kersten and Yuille (2003), and Yuille and
into a common format that mediates planning and Kersten (2006), in addition to chapters 32 and 34 in
execution of behavior. this volume.
The goal of intermediate levels of representation,
then, is to disentangle from the raw input stream aspects Why inference?
of the scene appropriate for driving behavior. In con-
trast to classification that collapses over variability to The Problem of Disentangling The properties of
make a discrete categorical assignment, the goal here the world that we care about—which drive behavior—
is to describe the variability—such as the slope of terrain, are not directly provided by sensory input. There are
120 120
140 140
Figure 27.4 The outlined region around the boundary of each line segment indicates the magnitude of response of a
the log (left panel) is shown expanded in the middle panel. neuron whose receptive field is situated at that position and
The right panel shows how a hypothetical array of model V1 orientation. An array of such neurons provides only weak or
neurons (Gabor filters at four different orientations) would ambiguous cues about the presence of object boundaries in
respond to the image subregion shown at left. The length of natural scenes.
orientation-selective units analyzing a local patch of Importantly, the computation of each of these maps
the image, with the boundary of the log superimposed involves propagating information between maps to
as a faint gray line. As one can see, almost nowhere obey photometric constraints and within maps to obey
along this boundary are there neurons firing indicat- continuity and occlusion constraints. That is, they
ing the position and orientation of the boundary. cannot be computed in independent streams in a purely
Instead, one finds neurons firing at many different feedforward fashion, but must cooperate to reach a
positions and orientations that signal structure in the solution.
background and foreground, but with little relation to Although Barrow and Tenenbaum appreciated the
the boundary itself. Thus, simply measuring oriented importance of disentangling and outlined some of the
contrast in an image does not give us a direct measure computational problems that need to be solved, they
of the boundaries or intersection of objects which can stopped short of proposing a specific algorithm and
then be fed into a reasoning engine about 3D shape. testing it on real images. Somewhat surprisingly, the
The best we will get from early levels of representation intervening years have seen only a handful of efforts
is a collection of ambiguous 2D shape cues that devoted to these problems (e.g., Jojic & Frey, 2001;
have aspects of illumination, shape, and reflectance Wang & Adelson, 1994), and as a result there has been
intermingled. These must then be aggregated and little progress in developing practical solutions for
refined by higher-level processes to disambiguate these dealing with real-world images. Recently, however,
cues and what aspects of scene structure they corre- Barron and Malik (2012) have made an important
spond to. advance by using priors over shape, reflectance, and
illumination to recover intrinsic images for these quan-
Intrinsic Images One of the first attempts to grapple tities from photographs of real objects. To date, their
with the computational aspects of the disentangling method obtains the best performance on this challeng-
problem was Barrow and Tenenbaum’s work on “intrin- ing problem. And, notably, it is based on inferential
sic images” (Barrow & Tenenbaum, 1978). They computation in which representations of shape, reflec-
attempted to specify the rules by which scene compo- tance, and illumination interact in order to settle to a
nents such as surface shape, reflectance, and illumina- solution.
tion could be recovered from the raw-intensity image. The intrinsic-image approach takes an important
They argued that these attributes should be separated step in introducing the idea of a structured or layered
at an early level of representation, and that by doing so representation that moves away from a flat, monolithic
it greatly facilitates the process of segmentation and representation of image properties (such as an array of
object recognition. Specifically, they proposed repre- Gabor filters) and toward a representation of properties
senting these attributes as a stack of 2D maps, or intrin- of the scene. But still, attributes of the scene are repre-
sic images, that are in register with the original intensity sented in 2D Cartesian coordinates, in a retinotopic or
image, where each pixel location is labeled according camera-centric frame of reference, whereas animals
to its shape, reflectance, or illumination properties. must act in complex 3D environments. Ultimately, then,
A B
External Internal .
.
world model .
I(x,y) i (x,y) ai
Figure 27.6 Sparse-coding model of images. (A) Images are patches extracted from natural images. Each patch shows a
represented in terms of a set of basis patterns ϕi (x ) that are different learned pattern ϕi (x ). The learned patterns are ori-
learned from the data. Each coefficient ai expresses how ented, localized, and bandpass (selective to structure at dif-
much of each basis pattern is needed to describe the image. ferent spatial scales), similar to the measured receptive fields
Sparsity is imposed through a prior that encourages coeffi- of V1 neurons.
cients to be zero. (B) Basis patterns learned from image
ABSTRACT An animal’s brain must extract from the onslaught How does the brain attack this problem? At the very
of raw sense data the few bits of information that actually input, it converts photons into electrical signals in nerve
matter for the guidance of behavior. Accordingly, one finds cells. At the output, it converts electrical signals of nerve
neurons in the upper echelons of a sensory system that report
very selectively on high-level features of the stimulus while
cells into movement. In fact, all the world events with
remaining invariant to many low-level perturbations. For which we interact are represented by neuronal mem-
example, “face cells” in the primate visual cortex respond brane potentials: sight, sound, smell, touch, speech,
selectively to one person’s face regardless of the view angle, movement, sweating, and internal phenomena like
scale, or illumination. The emergence of such complex thought, dreams, and emotions. This is a truly remark-
pattern detectors is one of the stunning phenomena in
able step of abstraction, by which phenomena of entirely
sensory neuroscience. In studies that trace signal flow through
the associated neural circuits, the most common tool is the different physical nature are encoded with the same
measurement of sensory receptive fields. Here we review the symbol set of membrane voltages. From that point on
tenets and basic results of receptive field analysis and place it the brain can use general purpose devices—namely,
in a common framework with the task of selective feature neurons and synapses—to create connections between
detection. We show that classic receptive field measurement—
these world events; for example, combining sights and
with its focus on the linear summation of stimulus variables—
has little to contribute on the subject of feature computation. sounds to cause thoughts and speech. We sometimes
On the other hand, a cascade of simple nonlinear operations forget the immense power of this abstraction, but it
can indeed account for high-level pattern detector neurons, offers another parallel to man-made electronic comput-
and hints of such organization are found in the brains of ers, which similarly represent all world events by voltage
diverse animals. signals.
In this chapter, we consider how these sensory com-
Consider a human typist copying a handwritten manu- putations happen; namely, how the neurons and syn-
script. His brain converts visual signals that enter the apses are arranged to accomplish the selective extraction
eye into motor signals that drive the fingers. How much of relevant bits from the sensory stream. We start with
information is involved in this task? At the input, the some illustrative examples of selective computation. In
typist’s eye can receive about 109 bits per second of raw tracing this process through the associated neural cir-
visual information.1 At the output, the typist generates cuitry, the primary tool has been measurement of
about 20 bits per second of movement information.2 receptive fields, and we discuss and critique this
This modest number is in line with other estimates for approach. How complex neuronal-response properties
the information rate of human behavior (Eriksson, come about through the actual signal flow in neural
1996; Pierce & Karlin, 1957). So the task of the typist’s circuits presents a more daunting challenge, and we
sensory system is to extract, every second from a data cite some examples of progress in this area. Finally, we
deluge of 109 bits, the 20 bits that really matter for visu- consider several theories of brain function that purport
ally driven behavior. Of course, those few important bits to explain why sensory computation is organized the
are hidden in the raw sensory input in convoluted ways. way it is. Throughout, we make liberal use of examples
The same applies to all other cases of sensory-driven from the visual system, partly because of our own expo-
behaviors. This process of highly selective extraction is sure bias, but the chapter focuses on principles thought
appropriately called computation; indeed, many of the to apply in the other senses as well.
tasks that we hand to our man-made computing machines
involve similar challenges of pattern extraction.
Selectivity and invariance
1
Our back of the envelope reads: Six million cone Continuing our observation of the typist: note how his
photoreceptors, each modulated at up to 25 Hz with a signal-
to-noise ratio of 100. right thumb flicks downward only when he needs to
2
One hundred words per minute, six letters per word, two bits type a space. So there is a motor neuron innervating
of entropy per letter in the English language. his thumb muscle that has become a perfectly selective
B C D
E F G
Figure 28.1 A geometric view of sensory computations. function of the stimulus variables. Note the contour lines are
(A) Each stimulus is a vector in a high-dimensional space, straight and orthogonal to the receptor’s axis of sensitivity
whose axes are defined by the excitation of all the sensory (here the S1 axis). (D) A receptive field is represented by a
receptors, for example, the intensity on the 6 million cones vector L in stimulus space. Integration of the stimulus S by
in a human eye. One retinal image of a face is a point in that this receptive field is the projection of S onto L. The response
space. The same face seen under different views and illumina- of a linear-nonlinear (LN) neuron depends only on this pro-
tions defines a high-dimensional surface (DiCarlo & Cox, jection: the contour lines of its response function are straight
2007). (B) Two-dimensional stimulus space. A “face cell” and orthogonal to L. (E–G) How to create a shoelace-shaped
responds only when the stimulus is in a thin, convoluted, response region in three simple steps. See text for details.
shoelace-shaped region. (C) The response of a receptor as a (See color plate 24.)
appreciate this, it is helpful to think about the space of like scaling, translation, illumination by a point source
stimuli in geometric terms. at various angles, background illumination, diverse
Your eye has about 6 million cone photoreceptors. facial expressions. Together these define a high-dimen-
For daylight vision, this is the number of pixels of the sional and highly convoluted surface in the stimulus
raw visual stimulus. Suppose the image of a face appears space (figure 28.1A). The face cell fires only if the
on the retina: like all other retinal images, it can be stimulus is on or near that surface.
represented as a point in an abstract space that has 6 For those few readers with difficulty imagining a
million dimensions (DiCarlo & Cox, 2007). Each coor- 6-million-dimensional space, let us consider a rudimen-
dinate of that space represents the intensity on one tary sensory system with just two receptors. The space
photoreceptor (figure 28.1A). Now rotate the face of all stimuli is a two-dimensional plane (figure 28.1B).
about one axis: this produces a series of stimulus points Any given image is a point on that surface. The activity
that trace out a curved line in the stimulus space. of those two receptors can be represented with two
Rotate it also about another axis: now one has a two- response functions on that surface, whose contour lines
dimensional surface. Add several other transformations, run straight and parallel to the axes (figure 28.1C). But
7
6
Frequency (Hz)
5
4
3
2
B
1000
7
6
5
4
0 .01 .02 .03 .04
Time (s)
C
Oct
0
Hex
D 0
Oct/Hex
0 1
Time (s)
Figure 28.2 Receptive fields in various sensory systems. time course again is biphasic, as in panel C. (E) Spectrotem-
(A–C) Spatiotemporal receptive fields in the early visual poral receptive field of a neuron in the auditory area field L
system. Here the components of the stimulus vector S are the of the zebra finch (Nagel & Doupe, 2008). Here the stimulus
light intensity at different locations and time points. Any given S(f, t) is represented as a spectrogram, plotting sound power
stimulus vector consists of a short “movie” S(x, y, t). The recep- as a function of time and frequency. The receptive field of
tive field L(x, y, t) can similarly be viewed as a movie. For this cell shows excitatory (red) and inhibitory (blue) regions.
displays on paper, one often shows a single frame L(x, y, t0) Along the spectral dimension one finds a “Mexican hat”
of this movie (panels A–B) along with the time course L(x0, profile, with a central region of excitatory frequencies and
y0, t) at a particular location (panel C). Note that L has both adjacent inhibitory regions. The action of the surround is
positive and negative components, meaning that light is excit- delayed relative to the center. In the time domain, the recep-
atory (red) at some locations and times, and inhibitory (blue) tive field is strongly biphasic or even triphasic. This is only
at others. For retinal bipolar cells, ganglion cells, and tha- one of many receptive field shapes encountered in this brain
lamic relay cells, the receptive fields look very similar except region. (F) Osmotemporal receptive field of a neuron in the
for some scaling along the spatial and temporal axes. The olfactory pathway of the locust (Geffen et al., 2009). Here the
spatial profile (B) contains a center region (C) and a sur- stimulus S(o, t) is the time-varying concentration of two dis-
round region (S) in which light has opposite action. When crete odors, octanone or hexanal. The receptive field L(o, t)
displayed in three dimensions (A), this profile has the appear- specifies the contribution to the response of each odor at
ance of a “Mexican hat.” The time course of the receptive different times. Top and middle: Receptive field derived from
field (C) is biphasic, both in the center and the surround. experiments in which only one odor was presented at a time.
Generally, the surround response is slightly delayed relative Bottom: Receptive field measured when both odors were varied
to the center (C). (D) For a simple cell in primary visual simultaneously. Note the neuron’s response to hexanal
cortex, the receptive field profile typically shows elongated changes polarity under this condition. (See color plate 25.)
side-by-side regions in which light has opposite action. The
ABSTRACT The brain relies on the spiking patterns of large circuits. It is not clear, however, how we should interpret
groups of neurons to represent information. Uncovering the the code of such a large number of neurons.
design principles of this “neural code” is fundamental to our Representation of information in the brain is neces-
understanding of information processing and computation in
the brain. This task is made difficult by the fact that neural
sarily noisy because of the stochastic nature of neurons
responses to their stimuli are highly selective and noisy, and and synapses. Overcoming this noise requires some
because population activity patterns are often correlated and form of error correction, but our understanding of how
the number of potential population patterns is exponential the brain might do this is, at best, limited. While differ-
in the size of the population. We present here the fundamen- ent brain areas show distinct spiking patterns and are
tal questions regarding the design and structure of probabi-
coding stimuli of different modalities, universal fea-
listic and correlated neural population codes and their
implication for encoding and decoding. We show that correla- tures of the code include population spiking patterns
tions are neither good nor bad, but that the combination of that are sparse, noisy, correlated, and adaptive (Rieke,
stimulus-induced correlations and joint fluctuations of the Warland, de Ruyter van Steveninck, & Bialek, 1997). We
cells balance the capacity of the code and its error-correction present here some of the fundamental design princi-
properties. We then show that the typically weak signal and
ples of the code of populations of neurons and, in
noise correlations observed at the level of cell pairs can add
up to give strong signal and noise correlations in large popula- particular, how noise and correlations are shaping it.
tions. Finally, we suggest that population codes may be We suggest that neural noise means that we must rely
designed to adapt to stimulus statistics in a way that would on probability distributions in describing and analyzing
give collective coding that balances the information content the code, and that the brain must do so as well. We
and the code’s readability or learnability. argue that the combinatorial number of potential activ-
ity patterns and correlations implies that understanding
the code of large populations must rely on identifying
Information is represented in the brain by the action simplifying principles of the design of neural codes. We
potentials of large groups of neurons. These basic present an information theory–based framework for
symbols form temporal sequences of multi-neuron assessing the role and impact of correlations and noise
spiking and silences that are the universal language that between cells and how it shapes the nature of the code.
our brain uses. Extending the language metaphor, to We review the dominant theoretical idea of redundancy
decipher this code we need to understand how this reduction as a design principle of neural population
alphabet is used to form the vocabulary, what the codes and its limitations, and suggest instead that neural
grammar is, and, ultimately, the content of this neural population codes rely on correlated collective coding
code. that balances the amount of conveyed information with
Much of our understanding of how information is the readability or “learnability” of the code.
“encoded” in the brain relies on studies of single
neurons. As most of what we care about is the result of The neural code
the joint activity of many neurons working together, the
nature of the code of populations is central to neurosci- The universality of spikes as the basic symbols of neu-
ence and our understanding of the brain. Experimen- ronal activity has made it common to talk about the
tally, it has been hard to record the joint activity of many “neural code” (Rieke et al., 1997). But although the
neurons over long periods of time. Since this task has “letters” of the code are universal in the brain (i.e.,
become more feasible in recent years using multi- spikes are the key carrier of information), the organiza-
electrode arrays or imaging, we will most likely have in tion of these letters into words and their meaning
the not-too-distant future recordings of very large popu- differs between brain areas, between conspecifics, and
lations and even recordings from full, large neural presumably between species. Moreover, the mapping
B B
C C
Figure 29.1 Reliability and reproducibility of in vivo spiking shown in panel A. The responses to constant stimuli gave
patterns in response to different stimuli. A fly (Calliphora highly irregular and noisy spiking patterns. Right: Spike-pat-
vicina) viewed a pattern of random bars that was moved across tern reliability for dynamic stimuli. (A) The fly viewed the
the visual either with constant or dynamic velocity. The spiking same spatial pattern as in the left panel but now moving with
activity of H1, a motion-sensitive neuron in the fly’s visual a time-dependent velocity, part of which is shown. The motion
system, was recorded from an immobilized fly over multiple approximates a random walk with diffusion constant D ∼ 14
repetitions of the same visual stimulus. Left: Spike-pattern degrees2/sec. For illustration, the waveform shown is low-pass
reliability for constant stimuli. (A) A random bar pattern was filtered. In the experiment, a 10-sec waveform is presented
presented across the visual field at constant speed (0.022 900 times, every 20 sec. (B) A set of 50 response traces to the
degrees/sec) and in the preferred direction of the H1neuron repeated presentations of the stimulus waveform shown in
in the fly. (B) A set of 50 response traces to thestimulus in panel A. (C) Averaged rate (PSTH) for the same segment.
panel A, each lasting 1 sec, taken 20 sec apart. The occurrence The rate is strongly modulated, but its time-average is very
of each spike is shown as a dot. The traces were taken from a close to that in the left panel. Variability and reproducibility
segment of the experiment where transient responses have were much higher than for the constant stimulus, but still
decayed. (C) The peristimulus time histogram (PSTH; bin showed diverse, stimulus-dependent reliability. (Adapted
width 3 ms, 96 presentations), which describes the rate at from de Ruyter van Steveninck et al., 1997.) Such results have
which spikes were generated in response to the stimulus parallels in many experiments on sensory neurons.
Rudolph, Fellous, & Sejnowski, 2001; Seung, 2003; networks (see Timme, Wolf, & Geisel, 2002; van
Deweese & Zador, 2004; Stevens & Wang, 1994). Yet Vreeswijk & Sompolinsky, 1996). The combination of
another source is the stochastic nature of ion channel all these sources of randomness and chaos shapes the
opening and closing (DeFelice, 1981; White, Rubin- probabilistic mapping from stimuli to responses, giving
stein, & Kay, 2000), which means that even when pre- a surprisingly high level of variability in many cases
sented with the exact same intracellular input, we can (see more on this below).
expect variability of the resulting spike train (Mainen Thus, rather than describing neural encoding in
& Sejnowski, 1995; Schneidman, Freedman, & Segev, terms of their high selectivity (which is the basis of the
1998). In addition to inherent randomness or “thermal” notion of “grandmother cells”), the mapping from
noise, we might get variance in the response to the stimuli to responses must be described in terms of a
same exact inputs without true “randomness.” Variabil- probability distribution over the responses to each stim-
ity may result from chaotic behavior at the level of ulus. This implies a computational cost in terms of
single cells, or from the dynamics of neuronal learning the code and processing it, which then requires
B
From correlations to redundancy, independence,
and synergy in neural population codes
Intuitively, the design of information coding by a
neural population can be described in terms of the
uniquely by some cells but not by others (independent In relative terms we can quantify the synergy or
coding), and information that is carried by the joint redundancy in terms of the ratio SR(r1,r2;s)/min(I(s;r1)
activity of cells in a way that can only be read from ,I(s;r2)), which measures what fraction of the informa-
their joint activity (synergistic coding; Amari & Naka- tion that is carried by the cells is redundant, and is often
hara, 2006; Averbeck & Lee, 2004; Latham & Niren- denoted as the fractional redundancy.
berg, 2005; Schneidman et al., 2003a). It is not The different mutual information terms we presented
immediately clear, though, how to translate the corre- and the relations between them do not tell us which
lation structure among cells, and between the cells and stimuli are associated with which responses. Nor do they
stimuli, into these notions of redundant, synergistic, tell us what is the redundant or synergistic information
and independent coding. First, the relation between that is encoded by the cells. Other kinds of measures
cells and the stimuli they respond to can be quantified must be used to give a breakdown of the encoded infor-
using various correlation measures. For example, while mation and relations between cells or stimuli, which
it is straightforward (and common) to assess the would have to rely on additional assumptions, such as
Pearson correlation between stimuli and responses, it metric approaches to the stimuli or decoding errors
is not clear how to use such linear measures for the (Brunel & Nadal, 1998; Kang & Sompolinsky, 2001;
correlation between the stimulus and the combined Shamir & Sompolinsky, 2004), or measures of informa-
responses of two cells (or more). Second, and more tion carried by specific patterns of activity (Brenner,
importantly, it is not clear how these measures relate to Strong, Koberle, Bialek, & de Ruyter van Steveninck,
one another and to these coding notions. 2000; Deweese & Meister, 1999; Pola, Thiele, Hoff-
Information theory gives a principled and self-consis- mann, & Panzeri, 2003; Schneidman et al., 2011). The
tent way to combine these notions of dependency power and benefit of using information-theory based
between stimuli and cells, without making arbitrary measures here is exactly their assumption- and metric-
assumptions—since the mutual information between free nature, which gives bounds on the relations
variables measures their dependency in a way that is not between cells and stimuli.
limited to linear correlations, or any specific metric on Somewhat surprisingly, the correlations between
similarity. We can then compare the information that pairs of cells are typically weak—in terms of both the
two cells convey together about the stimulus I(s;r1,r2) signal correlations and the average of noise correlations
and the information that each of the cells carries on its over stimuli—in many different sensory systems (Bair,
own, I(s;r1) and I(s;r2). The synergy or redundancy of Zohary, & Newsome, 2001; Ecker et al., 2010; Niren-
the code can be then expressed as the difference berg, Carcieri, Jacobs, & Latham, 2001). This has often
between the two, namely, been interpreted to suggest that we could either neglect
noise correlations or signal ones. Moreover, estimates
SR (r1, r2; s ) = I (s ; r1, r2 ) − ( I (s ; r1 ) + I (s ; r2 )) . of pairwise redundancy between cells have also given
relatively low values of redundancy, typically on the
Then, the cells are redundant when SR < 0, since in order of a few percent (Chechik et al., 2006; Gawne &
this case they must be carrying redundant information. Richmond, 1993; Narayanan, 2005; Puchalla, Schneid-
The cells are synergistic if SR > 0, since together they man, Harris, & Berry, 2005; Reich, Mechler, & Victor,
convey more information than taken separately. If SR = 2001; see an example in figure 29.4). Yet even these
0, then the cells carry independent information. This typically weak correlations can shape the reliability of
framework also allows us to make direct link to the rela- neural coding and computation (Cafaro & Rieke, 2010;
tions between the activity of the cells and the stimulus Lee, Port, Kruse, & Georgopoulos, 1998).
I(r1,r2;s), the signal correlations between cells I(r1;r2), The results for pairs of cells seem to be consistent
and the average noise correlations over stimuli I(r1;r2|s)s. with one of the more influential theoretical ideas in
The synergy/redundancy (SR) measure defined above neuroscience, known as “redundancy reduction.”
is also equal to the difference between the correlations Attneave (1954) and Barlow (1961, 2001) suggested
between neurons measured over all stimuli measured that neural populations “aim” to encode information
as the information between cells, and the average noise efficiently, and since redundancy is prevalent in natural
correlations, measured as the average over stimuli of sensory scenes, neurons would decorrelate their
the stimulus-dependent information between cells responses with respect to one another, to optimally
(figure 29.3A): use their joint coding capacity. This idea of a design
Population rate
population activity patterns (see figure 29.5A and
Amari, Nakahara, Wu, & Sakai, 2003; Ganmor, Segev, &
Schneidman, 2011b; Marre, El Boustani, Frégnac, &
Destexhe, 2009; Ohiorhenuan et al., 2010; Schneidman
et al., 2006; Shlens et al., 2006, 2009; Tang et al., 2008).
Going beyond the overall vocabulary of the code or
signal correlation, the immediate question is what is Time (s)
the role of correlations in encoding specific stimuli, or Figure 29.5 Strong signal and noise correlations at the level
what is the nature of population noise correlations? of large populations of neurons. (A) An example of strong
Similar to the case of the overall vocabulary, the nature signal correlations in the vertebrate retina and the accuracy
of the responses of large populations of neurons to of pairwise maximum-entropy models. The response of 10
simultaneously recorded ganglion cells in an intact patch of
the same stimulus are often poorly described by a
salamander retina, in response to a natural movie, was
model that assumes no noise correlations (Granot- recorded using a multi-electrode array. Neural responses were
Atedgi, Tkacik, Segev, & Schneidman, 2013; Pillow discretized into 20 ms bins, and the activity of the group was
et al., 2008; Schneidman et al., 2006). Explicitly, this represented in each time bin as a binary word of length 10,
means that p(r1,r2,…,rN|s) can be significantly different with xi = 1 if a neuron was active, and 0 if it were silent. The
rate of occurrence of each pattern predicted if all cells are
from p(r1|s)p(r2|s)… p(rN|s). This difference between
independent is plotted against the measured rate shown in
these distributions reflects the impact of noise correla- gray. Each dot stands for one of the 210 possible binary activity
tion at the level of the network for a particular stimulus patterns for the 10 cells. Black line shows equality. For the
(Granot-Atedgi et al., 2013), which for a population of same group of cells, the rate of occurrence of each firing
N cells we can quantify by I(r1;r2;…;rN|s) = H[p(r1|s)p(r2 pattern predicted from the pairwise maximum-entropy model
|s)…p(rN|s)] − H[p(r1,r2,…,rN|s)]. Importantly, while p(2) that takes into account all pairwise correlations is plotted
against the measured rate (black dots). The rates of com-
I(r1;r2;…;rN|s) can be very high for particular s, it is often monly occurring patterns are predicted with better than 10%
the case that averaging over stimuli gives a relatively low accuracy, and scatter between predictions and observations
value of 〈I(r1;r2;…;rN|s)〉s since for many other stimuli the is confined largely to rare events for which the measurement
noise correlations at the level of the group can be low of rates is itself uncertain. (Adapted from Schneidman
(Granot-Atedgi et al., 2013). As it has been customary et al., 2006.) (B) The stimulus-dependent maximum-entropy
(SDME) model (see main text) was fit to describe the response
in the literature to report the average noise correlations of 100 retinal ganglion cells to full-field flicker Gaussian stim-
over stimuli, and these reports often suggested weak ulus. The pairwise SDME model predicts population activity
noise correlations (see e.g., Ecker et al., 2010), one patterns for N = 100 neurons better than a model that assumes
should note that these averages may indeed mask strong that the cells are independent encoders, given by a set of
noise correlations for particular stimuli (see, e.g., conditionally independent LN models for each of the cells.
The log-likelihood ratio of the population firing patterns
Granot-Atedgi et al., 2013).
under the pairwise SDME model and under the conditionally
Two families of models have been used to show how independent model is shown as a function of time (black dots,
taking the relations between cells into account gives a scale on left) for an example stimulus repeat (which was not
significant improvement in describing the correlated used to train the model). For reference, the average firing
nature of population encoding of specific stimuli. The rate of the population is shown in gray (scale on right). Thus,
the noise correlations in the population are very strong at
generalized linear model (GLM; Pillow et al., 2008)
particular times along the stimulus. (Adapted from Granot-
describes the response of each cell in a population Atedgi et al., 2013.)
to a stimulus s by its instantaneous firing rate, given by
I= I=
I=
ts
I=
8
2
bi ioral information can be read fast and reliably, even if
I= not perfectly (Gollisch & Meister, 2008; Haddad et al.,
2010; VanRullen & Thorpe, 2001); predictive coding of
6
stimulus features by neural populations (Hosoya,
High Baccus, & Meister, 2005; Palmer, Marre, & Berry, 2013;
4 noise Srinivasan, Laughlin, & Dubs, 1982); sparseness and
Low overcompleteness of the code (Olshausen & Field,
noise
2 1997; Simoncelli & Olshausen, 2001); time scale of rep-
2 4 6 8 10
Output entropy (bits)
resentation (Burak & Fiete, 2012); and energy efficiency
(Laughlin, 2001; Laughlin & Sejnowski, 2003; Levy &
Figure 29.6 The effect of noise and correlation on the Baxter, 1996). It is not unlikely that the principles would
nature of optimal information encoding by model networks. be different in different parts of the brain. In particular,
(A and B) A toy SDME model with 10 neurons was studied
the effect of noise in neural circuits goes beyond the
under Gaussian correlated stimuli for the case of no coupling
between the cells, aside from signal correlations (in panel A), capacity and accuracy of representing sensory informa-
and with the optimal couplings that would maximize the tion that we presented here. It also imposes limits on
encoded information (in panel B). The output entropy (total the nature of persistent activity in networks and propa-
population entropy) and the population noise entropy are gation in the brain (London, Roth, Beeren, Häusser, &
shown for uncoupled and optimal coupled networks para-
Latham, 2010), and is likely to have a crucial role in
metrically as neural reliability (or 1/noise) changes from high
(bright symbols) to low (dark symbols), The information that shaping the computations that underlie decision
the population transmits I, is the difference between output making based on population activity (Beck et al., 2008;
and noise entropies and is shown by the gray scale gradient. Churchland et al., 2011; Lee, 2008) and the representa-
Networks with low reliability transmit less information and tion of probabilistic information in the brain (Beck et
thus lie close to the diagonal, while networks that achieve high al., 2008; Berkes et al., 2011; Daw, O’Doherty, Dayan,
information rates lie close to the lower right corner. The
optimal network uses its couplings to maintain a consistently Seymour, & Dolan, 2006; Deneve, Latham, & Pouget,
low network noise entropy despite a 10-fold variation in 1999; Lee et al., 1998; Pouget, Dayan, & Zemel, 2000).
neural reliability. Error bars are computed over 30 replicate The ability to record large populations of neurons
optimizations for each reliability (1/noise) level. Thus, these over long time periods would enable us in the not-so-
two models show distinct information coding capability, based distant future to address the design and content of very
on optimal coupling between cells, but also distinct profiles
of coding vocabulary and coding noise. (Adapted from Tkacik
large population codes and their dynamics. Recent
et al., 2010.) experimental studies suggest that network correlation
structure may change during development or learning
features of our brains. However, with 1011 neurons and (Berkes, Orban, Lengyel, & Fiser, 2011; Tkacik et al.,
1015 synapses in the human brain, it is clear that if this 2010) to match stimulus statistics. Theoretical analysis
were the key aspect of optimality, the evolutionary pres- and experimental evidence suggest that very different
sure for more reliable neural circuits would be immense. correlation structures would be optimal for different
The brain’s performance despite its noisy hardware stimuli and noise conditions (Chechik et al., 2006;
ABSTRACT Many of our everyday perceptual and cognitive objects (DiCarlo & Cox, 2007; Hung, Kreiman, Poggio,
tasks require our brains to transform information from implicit & DiCarlo, 2005a).
representations in which task-relevant information exists but in
The need to group neural responses into sets applies
a format that is difficult to extract, into explicit representations
in which this information is accessible. For example, deter- to more cognitive challenges as well. For example, sig-
mining the identities of objects that are currently in view naling whether an item is a member of a particular
across naturally occurring variation, such as changes in an category (e.g., whether a chair is a piece of furniture)
object’s position, requires our brains to reformat the pattern requires our brains to group the sensory representa-
of light-based representations encoded by our photoreceptors tions of different items (e.g., Freedman, Riesenhuber,
into representations that explicitly reflect object identity. The
brain faces similar challenges for other perceptual tasks, such Poggio, & Miller, 2001). Moreover, we know that our
as identifying words spoken by different voices, as well as more brains can flexibly group the same items in different
cognitive challenges, such as determining whether a chair ways, depending on task demands, as exemplified by
belongs to the category of furniture. Insight into these chal- the Wisconsin card sorting task in which subjects are
lenges, and the brain’s solutions, can be understood using instructed to take a deck of cards and switch between
geometrical, population-based coding approaches. Once for-
mulated, these population-based descriptions can be linked
groupings of cards with the same shape, color, and
to the single- and multi-neuron mechanisms that support number of items (Berg, 1948). In these and similar
a successful task solution as well as provide important cases, our brains transform lower-level population
insights into the computations that the brain uses to process representations in which the task solution exists but
information. is hidden (because the neural responses to the
members of different groups are intermingled in the
Many of the computations that our brains perform can population representation), into higher-level popula-
be restated as transformations of information from tion representations in which the appropriate neural
implicit representations in which task-relevant information responses are grouped together and the task solution
exists but in a format that is difficult to extract, into can be extracted.
explicit representations in which this information is
accessible. Transformations from implicitly to explicitly Understanding neural transformations at
formatted information are often required when our multiple levels
brains need to “group” the neural responses to differ-
ent conditions into sets, and this requirement is ubiq- Explicit representations tend not to emerge until higher
uitously present in the tasks that we perform every day. stages of neural processing. In these high-level brain
For example, consider the perceptual challenge of areas, neural response properties tend to be diverse,
identifying the objects that are currently in view. Each and this diversity is thought to be advantageous insofar
of our photoreceptors encodes light intensity at a par- as a population that contains a diversity of neural
ticular position, and thus the photoreceptor population responses is capable of performing a diversity of tasks
represents the visual environment as patterns of light. (Rigotti et al., 2013). However, response heterogeneity
These light patterns can differ substantially with natural also makes these high-level brain areas difficult to
variation in an object, such as changes in its position on understand using classical single-neuron approaches,
our retina, or its retinal size as the object moves which inherently rely on identifying regularities in the
toward us, yet we have no problems identifying objects response properties of individual neurons across a pop-
across these identity-preserving transformations. This is ulation (e.g., discovering that the majority of V1 neurons
because our brains successfully “group” all the light are tuned for orientation).
patterns that contain the same object and differentiate Inspired in large part by the proposals of David Marr
those from the light patterns that contain different (1982), we and many others have converged on using
A
Response
| |
Neuron 1 || | || ||| vector
Spike count, neuron 2
Neuron 2 | | | | |
...
|
|| | |||
| || | | |
| | || |
| | | |
| |
| || | | Response
|
|| || | | cloud
|| | |
| | | |
Spike count, neuron 1
Counting
window
Neuron 1
Figure 30.1 Population representations. (A) The spike requires parsing the white and black sets of response clouds.
count responses for N neurons combine to form a “response Panels show scenarios in which the information required for
vector” of length N. This response vector exists in an N- this task does not exist (left); the information required for this
dimensional space but is illustrated in the two-dimensional task is present but requires a highly nonlinear population
space defined by the responses of two neurons (one dot). readout (i.e., is implicit or tangled; middle); the information
Because neurons are noisy, repeated trials of the same condi- required for this task can be accessed via a linear readout of
tion produce slightly different response vectors, and together the population (i.e., is explicit or untangled; right).
all trials form a “response cloud.” (B) Shown are the hypo-
thetical responses for a two-way discrimination task that
B2
Spike count
N
ED
B1
d’
B A1
~B
C1
Neuron 2
Proportion
A2
Proportion
d’ C2 ~A A
Neuron 1
Spike count
C Noise correlations
Beneficial Detrimental
Uncorrelated correlations correlations
D Tuning correlations
Beneficial Detrimental
Uncorrelated correlations correlations
Figure 30.2 Factors that determine population perfor- correlations are determined by the trial-by-trial variability
mance. (A) The distance between population-response clouds between neurons, and they determine the shape of the indi-
depends on the information conveyed by single neurons and vidual response clouds. (D) Tuning correlations are deter-
grows as a function of population size. Shown are the responses mined by the mean firing-rate responses within and across
of two hypothetical neurons, each with a d′ = 3 for two condi- sets, and they determine the relative positions of the different
tions. The normalized Euclidean distance (NED) for these response clouds. For (C) and (D), shown are hypothetical
two conditions in the 2D population space can be calculated scenarios that include no correlations, correlations that are
as NED = 32 + 32 = 4.2 . (B) An N-way classification can be beneficial because they are aligned to the decision boundary,
envisioned as parsing the population space into N two-way and correlations that are detrimental because they are per-
classifications (e.g., A/not A, B/not B, etc.). (C) Noise pendicular to the decision boundary.
decision boundaries, each of which separates the individual response clouds (i.e., uncorrelated noise has
response clouds corresponding to one set from all the a spherical shape whereas correlated noise is oblong;
other sets. figure 30.2C). Whether this type of correlation is ben-
Population performance also depends on “correla- eficial or detrimental to population performance
tions,” which come in two varieties. “Noise correla- depends on how these correlations are aligned relative
tions,” defined by the correlated trial-by-trial variability to the decision boundaries (figure 30.2C). In contrast,
between neurons, determine the shapes of the “tuning correlations” determine how the response
Squirrel
Not squirrel
C Which of 10 objects D
1 is in view? Example neurons
IT
Not linearly Linearly
0.8
separable separable
Performance
V4
Transformation
0.6
0.4
0.2
0 Object # Object #
0 50 100 150
Number of neurons
Figure 30.3 Explicit representations of object identity. As classification performance in V4 and IT using a linear readout,
reported by Rust and DiCarlo (2010). (A) A subset of images reprinted from Rust and DiCarlo (2010). (D) Firing-rate
used to probe the representation of object identity, across identity- response surfaces for 10 objects presented under six different
preserving transformations, in V4 and the inferotemporal transformations for one example V4 neuron that is not lin-
cortex (IT). In total, 10 objects were presented under six dif- early separable, and one example IT neuron that is linearly
ferent transformations that included changes in the objects’ separable. Single-neuron linear separability translates into the
position, size, and background context. (B) The problem of preservation of object preferences across these transforma-
invariant object recognition can be formulated as multiple, tions, or equivalently a separable response matrix.
two-way classifications (see also figure 30.2B). (C) Object
Information about objects is processed in the primate presented at different positions and sizes and within
brain along the ventral visual pathway, a hierarchically different background contexts (figure 30.3A; Hung et
arranged collection of neural structures that includes al., 2005a; Li, Cox, Zoccolan, & DiCarlo, 2009; Rust &
the retina and lateral geniculate nucleus (LGN) as well DiCarlo, 2010). We then evaluated the degree to which
as cortical brain areas V1, V2, and V4 and inferotempo- object identity was explicitly represented at each stage
ral cortex (IT; Felleman & Van Essen, 1991; reviewed by probing how well a linear decision boundary could
by DiCarlo et al., 2012). Within each structure, the separate the responses corresponding to one object
response vectors corresponding to all possible identity- from the others using a cross-validated linear classifica-
preserving transformations of the same object (e.g., tion scheme (figure 30.3B). These results revealed that
changes in position, scale, pose, etc.) define an object a linear readout of object identity applied to the IT
manifold (DiCarlo & Cox, 2007). As described above, population performs robustly with only a few hundred
the population of photoreceptors represents the visual neurons and that the IT population performs invariant
scene as patterns of light, and because natural object object recognition tasks much better than real or simu-
variation produces markedly different light patterns, lated populations at earlier stages, such as V1 and V4
the manifolds for different objects are “tangled” in the (figure 30.3C; Hung et al., 2005a; Freiwald & Tsao,
photoreceptor representation. Thus, to extract object 2010; Li et al., 2009; Rust & DiCarlo, 2010; figure
identity information, the brain must untangle these 30.3A). These results support the notion that the ventral
signals. To evaluate where and how the brain accom- visual pathway reformats object manifolds to make
plishes this, we and others have recorded and/or simu- object identity explicit at its final stage (DiCarlo & Cox,
lated the responses of neurons within different ventral 2007).
visual pathway brain areas to images of objects
Target
matches
Time
C D Population average
Is the image a target match Working
Normalized modulation
or a distractor? Visual memory IT
1 PRH Visual
PRH
0.9
Performance
0.8 Diagonal
Looking for
1
0.7 IT 2 Working
3 memory
0.6
4
0.5 1 2 3 4 Diagonal
0 50 100 150 Looking at 0 100 200
Number of neurons Time (ms)
Response, output 2
Response, input 2
Target
matches
Response, input 1 Response, pairing 1 Response, output 1
Figure 30.4 Explicit representations during target search. Amounts of each type of signal as function of time relative to
As reported by Pagan et al. (2013). (A) Monkeys performed stimulus onset and averaged across each population, com-
a delayed match to sample task in which a cue image was puted via a bias-corrected, ANOVA-like decomposition (right;
presented, followed by a random number of distractor images described by Pagan et al., 2013). (E) A linear-nonlinear model
and then a target match. The task required the monkeys to provided a good account of the transformation from IT to
fixate throughout the presentation of the distractors and PRH. This model untangled target match information by
make a downward saccade in response to the match. The combining IT neurons that had tuning correlations for target
experimental design involved presenting four images in all matches and distractors that were asymmetric (e.g., positive
possible combinations as a visual stimulus and a target. tuning correlations for target matches and negative correla-
Throughout the figure, the different images are indicated tions for distractors are shown in this hypothetical example).
with different shapes, with target match conditions in black Pairing such neurons with appropriate weights resulted in a
and distractors in white. (B) This task can be formulated as a rotation in the population space that resulted in variance dif-
two-way classification of the same images presented as target ferences between the firing-rate response distributions to
matches and as distractors. For simplicity, only a subset of the target matches and distractors (depicted at the bottom and
conditions is depicted. (C) Target match/distractor linear left of the plot). These variance differences were then
classification performance measured in IT and perirhinal exploited to increase linear separability via an instantaneous
cortex (PRH; reprinted from Pagan et al., 2013). (D) Firing- nonlinearity that included thresholding and/or saturation.
rate response matrices for four example neurons (left).
ABSTRACT One of the major goals of systems neuroscience is stimulus vary tremendously in their functional and ana-
to understand how the activity of sensory neurons gives rise tomical properties. The sensory information they
to our perceptual experience. When scientists first began encode may or may not be useful for the task at hand
recording from sensory neurons while subjects performed
perceptual tasks, Parker and Newsome (1998) wrote a seminal
or sufficiently sensitive to explain the detail with which
paper laying out a rubric for how to establish that the electri- a subject perceives a stimulus. Different cortical areas
cal impulses on the end of the electrode were actually respon- or other groupings of cells work as a network, so simply
sible for a specific percept. The intervening years have seen determining the effect of removing or activating a
an explosion of interest in this question, technical develop- group of neurons might be either too crude or too
ments that have paved the way for new types of answers, and
subtle a manipulation to yield specific behavioral effects.
theoretical advances that provide a context for the new exper-
imental results. In this chapter we will update the rubric of To make matters worse, most neurons do more than
Parker and Newsome to incorporate recent work and to pose simply provide sensory information, so dissociating
questions whose answers will provide a new level of under- their contributions to perception rather than a cogni-
standing in the coming years. tive or motor process can be challenging. In general,
establishing a link between neurons and task perfor-
Since the fourth century BC, when Aristotle claimed mance entails monitoring the activity of and manipulat-
that the heart was the seat of the mind and the soul, ing specific (but often large) groups of neurons while
scientists and philosophers have been searching for a a subject performs a task, which can be experimentally
link between biology and our internal perception of the challenging.
world around us. Modern neuroscience has long In 1998, Parker and Newsome wrote an influential
recognized that the physical source of our internal review summarizing the state of the field and providing
experience is the brain. Over the last few decades, neu- a rubric for establishing a link between a group of can-
roscientists have begun to amass a body of evidence didate neurons and a specific percept (Parker &
linking the activity of groups of sensory neurons in Newsome, 1998). Their rubric has become an invalu-
specific brain areas with individual percepts. able framework for interpreting and combining the
Although the claim that our internal experience is results of many studies. The nature and quantity of the
due solely to the activity of neurons is no longer con- experimental evidence linking sensory neurons to per-
troversial, associating a specific group of neurons with ception has exploded since that time, leading to many
a particular perceptual experience is challenging for new insights. Out of technological necessity, earlier
both conceptual and experimental reasons. First, it is experiments typically monitored the activity of one
necessary to measure a subject’s percept, which is a neuron at a time and extrapolated their responses to
fuzzy, subjective experience, in a quantifiable way. This the large groups of neurons thought to underlie any
challenge has largely been met thanks to centuries of percept. The technology for activating or inactivating
work by psychologists and psychophysicists who have neurons was crude, akin to using a chain saw when a
designed clever tasks to measure subjects’ perceptual scalpel was called for. The techniques for analyzing
abilities. Still, we are left relating neuronal activity to data focused on these single neuron recordings or
performance on a task, rather than perception. gross causal manipulations. In the intervening decade
Determining whether the activity of a particular and a half, new experimental technology and tech-
group of neurons is in a position to underlie perfor- niques for analyzing data have revolutionized the field.
mance is even trickier. Many thousands of neurons These developments have recently begun to bear
across sensory cortex as well as in subcortical areas fruit, and hard drives are being filled with data as
respond every time we see, hear, touch, smell, or taste quickly as engineers can increase their size. As the field
a sensory stimulus. The neurons that respond to any adjusts to this wealth of new information, it is more
ruff and cohen: relating the activity of sensory neurons to perception 349
important than ever to establish a conceptual frame- 2. Most experimental methods for recording or
work for understanding the meaning of these new data. manipulating the activity of neurons are invasive, so
For those who spend their days in the lab or at the they cannot be used in humans except in rare cases
computer, such a framework will be the only way of when a patient is undergoing brain surgery for another
recognizing when we have accomplished the goal. What reason. Therefore, the vast majority of data concerning
evidence will convince us that we’ve found the biologi- the neuronal basis of perception comes from animal
cal basis for a percept? studies, in which the subjects cannot verbally describe
Our goals in this chapter are fourfold. We aim to their experiences.
1. provide an updated framework that is based largely To link neuronal activity with perception, we there-
on the rubric of Parker and Newsome but reflects new fore rely on the field of psychophysics to quantify the
insights for linking groups of neurons to perception; relationship between physical stimuli and a subject’s
2. review the current state of the art for performing perceptual abilities. Psychophysical experiments can be
experiments and analyzing the results; carefully designed to minimize the ambiguity inherent
3. review the results of influential experiments using in perceptual reports, often in ways that can be general-
older technology and describe early experiments using ized from human subjects to nonhuman subjects.
the newest methods; Rather than asking a subject how fast something went,
4. highlight current and future avenues of research for example, one could probe their ability to judge
aimed at understanding how neural activity gives rise to speed by asking them which of two stimuli is moving
perception. faster. This experimental design removes the ambiguity
Although we will discuss evidence linking different sets associated with the subject’s internal speed calibration,
of neurons with particular percepts from a variety of because it forces the subject to evaluate one stimulus
model systems and organisms, our goal is not to provide with respect to another stimulus. The subject’s response
a comprehensive review of this field. We aim to explore on a single trial could be compared to the activity of
the depth of knowledge necessary to present a compel- the candidate neurons at that moment (e.g., could a
ling case that a particular set of neurons underlies a subject’s mistake be predicted from the activity of that
specific perception. group of neurons?). The subject’s performance on the
task could also be compared to that group of neurons
(e.g., how well could a hypothetical subject do if all the
information they had to go on came from the activity
Quantifying a percept
of that group of neurons?).
Perception is an inherently subjective experience. We The experiments described in the rest of this chapter
can easily quantify aspects of a physical stimulus, such attempt to determine whether the responses of a group
as the speed of a moving object, the chemicals that give of candidate neurons are necessary and sufficient to
rise to an odor, or the pitch of a musical note. But the explain performance on perceptual tasks. Therefore, if
subjective experience of seeing your puppy sprint by the title of this chapter were to describe the status
you, smelling your mother’s homemade chocolate chip rather than the goal of this field, it would be called the
cookies, or hearing the crescendo in a Beethoven sym- more cumbersome “relating the activity of sensory
phony is not something that is readily accessible to neurons to performance on perceptual tasks.”
experimentalists.
In everyday life, we try to understand the perceptual
Quantifying performance
experiences of others by asking them what something
looks, smells, or sounds like. Such perceptual reports, Measuring a subject’s overall performance on a task is
however, are an ineffective way of linking neuronal therefore critical for assessing perception. Simple mea-
activity to perception for two reasons: sures like total percent correct are affected by factors
1. Perceptual reports are notoriously unreliable. other than the subject (such as the difficulty of the
Although it is beyond the scope of this chapter, there task). Psychophysicists usually measure a psychometric
is abundant experimental evidence that people’s reports curve, which is a plot of performance (either percent
of what they perceive (or do not perceive) are very dif- correct or percent of choices in favor of a particular
ferent from what quantitative tests reveal are the limits decision) as a function of a measure of the sensory
of their perceptual capabilities. This has consequences stimulus. For example, in a speed-discrimination task,
far beyond neuroscience, including the validity of eye- the psychometric curve might plot percent correct as a
witness testimony in the courtroom. function of the difference in speed between the two
ruff and cohen: relating the activity of sensory neurons to perception 351
used to establish what is currently the best link between Limitations:
perception and sensory neurons. Newsome and his
• For nonhuman subjects, discrimination tasks are
colleagues (Britten, Newsome, Shadlen, Celebrini, &
often more difficult to train than detection tasks.
Movshon, 1996; Britten, Shadlen, Newsome, & Movshon,
• Decisions are often made quickly, limiting the time
1992; Newsome & Paré, 1988; Salzman, Britten, &
available to record neuronal responses. If subjects are
Newsome, 1990; Shadlen, Britten, Newsome, &
forced to observe a stimulus for a long period of time
Movshon, 1996) performed a series of experiments
before making a decision, they may not use all of the
linking the responses of direction-selective neurons in
available sensory information, making it difficult to
the middle temporal area (MT) and performance on
determine when the decision actually occurred.
the task in figure 31.1B. In these studies, rhesus monkeys
viewed a dynamic random dot display in which a per-
centage of the dots moved coherently in one of two Linking sensory neurons to perception
opposite directions (up or down in figure 31.1B), while
the rest of the dots moved randomly. The monkeys We will devote the rest of the chapter to establishing a
were required to indicate which of the two directions series of questions (based heavily on the rubric of
contained the coherent motion. When a large percent- Parker & Newsome, 1998) that must be answered to
age of dots moved coherently, this task proved to be demonstrate that a candidate group of neurons under-
very easy. At low coherence, the random dots provided lies performance on a particular perceptual task. The
a masking stimulus, making the discrimination earliest attempts to characterize sensory neurons
difficult. focused on describing the general properties they
There are two types of discrimination tasks. Some, encode. On average, neurons in early sensory areas are
like the direction-discrimination task (figure 31.1B), selective for simple features like the orientation of a
require subjects to observe a stimulus and choose line, the pitch of a note, or which of a rodent’s whis-
between two or more options. When there are two kers has moved. In higher cortical areas, cells are
options, these are called two-alternative forced-choice tuned for more complex features like object identity.
tasks (the “forced” is because the subject must pick an Building on this knowledge, one can make a somewhat
option rather than indicating “I don’t know”). The educated guess about which neurons seem like good
forced choice can often reveal perceptual abilities candidates to underlie the perception of a particular
unknown to the subject. In the direction-discrimination feature. The goal of the rest of this chapter is to pose a
task, human subjects often perform better than would series of experimentally tractable questions that test
be expected by chance even when they report seeing this hypothesis.
no coherent motion. By far, the best-established connection between
In the second type of discrimination task, two-interval sensory neurons and perception is the one studied by
forced-choice tasks, subjects are required to indicate in Newsome and colleagues (figure 31.1B) between direc-
which of two time intervals a stimulus (or a stimulus tion-selective neurons in the middle temporal visual
with a certain property) occurred. An example of this area (MT) and performance in a motion-direction dis-
type of task was used by Romo and colleagues (Romo, crimination task. To our knowledge, this is the only
Hernández, Zainos, & Salinas, 1998) to determine the connection between sensory neurons and perception
role of somatosensory areas in the perception of vibrat- for which all of the questions below can be answered in
ing tactile stimuli. Two stimuli were presented in the affirmative. In each section, we will describe the
sequence and monkeys were required to indicate which evidence linking MT with motion perception and also
of the two vibrated at a higher frequency. provide recent examples showing how these concepts
have been applied to other systems or tested using new
Strengths: experimental techniques.
• Thresholds are immune to differences in criterion
because a bias in favor of one option (e.g., “up” answers Do the responses of the candidate neurons encode detailed
in the motion-direction discrimination task) can be enough sensory information to support the percept? The
measured by comparing performance on trials when most basic requirement for the neural underpinnings
the dots actually moved up versus down. of a percept is that the candidate neurons carry enough
• While cognitive factors like alertness still improve sensory information to explain performance on a per-
performance and modulate neuronal responses, they ceptual task. For example, if, as a group, MT neurons
do so equally for all trial types. Therefore, increases in could only discriminate strong upward from downward
arousal will affect “up” and “down” trials equally. motion with 75% accuracy but the monkey can get 90%
ruff and cohen: relating the activity of sensory neurons to perception 353
underlie perception, however, they must together A
encode sufficient information to explain performance
on psychophysical tasks. The results highlighted here
show that this is not usually the most difficult require-
ment to fulfill in linking sensory neurons with
perception.
ruff and cohen: relating the activity of sensory neurons to perception 355
techniques were refined, experimenters realized that A
smaller currents delivered by fine electrodes placed
no stim
inside the cortex of animals could lead to readily repro- stim
ducible motor responses (for a review, see Graziano,
2006).
Newsome and colleagues used microstimulation to
determine how activating small populations of MT
neurons could bias the monkey’s choice during the
motion-direction-discrimination task (Salzman et al.,
1990). This clever experiment relied on the fact that in
MT, neurons located near each other tend to have
similar preferred directions. Therefore, when the
experimenters applied small currents, they could rea- – – –
sonably expect that they were modulating the activity of
a group of neurons with similar tuning. B
The authors put an electrode into MT, placed the
motion stimulus in the receptive fields of the neurons intact
lesion
they recorded, and had the monkey discriminate
motion in the neurons’ preferred direction from
motion in the neurons’ null direction. Unbeknownst to
the monkey, the experimenters passed small amounts
of current through the electrode on a subset of trials,
which increased the firing rates of the neurons near the
electrode tip.
The monkey behaved as if the stimulation had
increased the strength of the motion in the neurons’
preferred direction, which can be seen in the leftward
shift in the animal’s psychometric curve (figure 31.4A).
That this technique works is astounding: the currents Figure 31.4 (A) Microstimulation biases the subject’s choice
used in this experiment should have modulated the toward the option that matches the preferred direction of the
responses of only tens or hundreds of neurons (Histed, stimulated neurons (schematic based on the results of Salzman
Bonin, & Reid, 2009). The fact that tickling the et al., 1990). (B) Performance on a motion-discrimination
task is not much better than chance following ibotenic acid
responses of such a small number of neurons affects the
lesion of MT (schematic based on the results of Newsome &
monkeys’ behavior in a measurable way is strong evi- Paré, 1988).
dence that MT neurons underlie performance in the
direction-discrimination task.
In other contexts, microstimulation has an even
more astounding effect: it has been used to create a information that the animals could use to make deci-
percept in the absence of a physical stimulus. Romo and sions about stimulus frequency.
colleagues trained monkeys to perform a two-interval While microstimulation is a powerful tool for evaluat-
somatosensory discrimination where the monkey’s job ing the causal contribution of neurons to perception,
was to indicate whether the second stimulus in a pair it is a relatively coarse method both temporally and
vibrated at a higher or lower frequency than the first spatially, and does not allow for the precise targeting of
(Romo et al., 1998). During most trials, a mechanical cell types or neuronal subpopulations. New techniques
vibrating flutter stimulus was applied to the monkey’s make it possible to precisely target particular neurons
fingertips for both stimuli, but on some trials, one of at particular times. In particular, the field of optogenet-
the mechanical stimuli was absent and instead, electri- ics allows researchers to use light to activate neurons
cal microstimulation was applied directly in somatosen- that express proteins such as channelrhodopsin (ChR2;
sory cortex. Amazingly, the animals were able to Luo, Callaway, & Svoboda, 2008; Peron & Svoboda,
compare electrical and mechanical stimuli after train- 2010; Scanziani & Häusser, 2009).
ing only on mechanical stimuli. This experiment The greatest promise of these techniques lies in their
provided strong evidence that the electrically stimu- potential specificity: in mice and some other species,
lated somatosensory neurons encoded vibrotactile ChR2 can be genetically expressed into particular
ruff and cohen: relating the activity of sensory neurons to perception 357
could be responsible for perception and a motor plan, identify faces. To test whether activity of inferotemporal
so the requirement should not be for identical neurons is sufficient to lead to the percept of a face,
responses for different motor outputs. Rather, some Afraz and colleagues performed an analogous experi-
aspects of the response (e.g., early in the behavioral ment in primate IT to the one performed by Newsome
trial) should not depend on the motor output used to and colleagues in area MT. These authors trained
signal the choice. monkeys to perform a task where they reported whether
an image contained a face or not (Afraz, Kiani, & Esteky,
Examples of systems for which the link between 2006). This task was made difficult by adding various
sensory neurons and perceptions is most well amounts of noise to the image set that degraded the
established clarity of the images, thus making them harder to dis-
criminate. Similar to the results of the MT study, the
To our knowledge, MT and the motion-direction dis- authors found that microstimulation in face-selective
crimination task represents the only system for which portions of cortex led the animals to report seeing a
all of the above questions have been answered in face in a noisy stimulus more often than when stimula-
the affirmative. However, there is growing evidence tion was not applied.
linking other brain areas with specific percepts. Two With the advent of increasingly sophisticated molecu-
other experimental systems have proven to be particu- lar, genetic, and imaging techniques has come a strong
larly fertile grounds for linking sensory neurons to interest in developing both rodent and rat models for
perception. linking the activity of neurons to behavior. Progress on
Motion direction is a relatively low-level feature of a this front has been made across a range of tasks and
visual stimulus, but the same experimental methods sensory modalities, including visual (Carandini &
used in the MT experiments have been used to link the Churchland, 2013) and auditory (Brunton, Botvinick,
activity of neurons in inferior temporal (IT) cortex, in & Brody, 2013) discriminations. Of particular interest
both humans and monkeys, with the perception of is a series of work linking the activity across multiple
higher-level features like faces. Early electrophysiologi- brain areas in rats to olfactory discriminations.
cal experiments identified neurons in primate IT that Uchida, Mainen, and colleagues have demonstrated
were extremely selective for faces compared to other that rats can accurately discriminate between mixtures
complex objects (Desimone, Albright, Gross, & Bruce, of odor pairs with just a single sniff (Uchida & Mainen,
1984). The existence of small clusters of face-selective 2003). Neurons in the olfactory bulb, the target of olfac-
IT neurons was revealed by work from Doris Tsao and tory receptor neurons in the nose, exhibit odor-specific
colleagues, who used functional imaging to guide their selectivity and have been shown to reliably distinguish
electrophysiological recordings and identified patches between small sets of odorants on similarly brief time
in macaque temporal cortex that were overwhelmingly scales to a sniff, as well as to correlate with reaction
selective for faces, even compared to other similar times during an odor-discrimination task (Cury &
shapes (Tsao, Freiwald, Tootell, & Livingstone, 2006). Uchida, 2010).
Functional imaging in humans has also revealed patches Piriform cortex, also known as olfactory cortex,
of cortex in the ventral stream that are highly selective receives input from the olfactory bulb and is a candi-
for different object categories, such as faces (Kanwisher, date for a cortical region that is involved in odor dis-
McDermott, & Chun, 1997). Activity in these regions criminations. A recent study recording activity in
may be used during perceptual decisions about complex piriform cortex during an odor-mixture discrimination
object categories like faces and houses (Heekeren, demonstrated that the activity of fewer than 100 neurons
Marrett, Bandettini, & Ungerleider, 2004), but it is cur- was sufficient to accurately predict behavioral perfor-
rently unknown whether individual IT neurons in mance and reaction time. This study also interestingly
humans or monkeys can be used to predict choices in revealed that noise correlations among these neurons
a face-discrimination task in a manner that is directly were extremely low, highlighting a potentially impor-
analogous to the choice-probability studies performed tant difference between olfactory and visual processing
in MT. (Miura, Mainen, & Uchida, 2012).
However, the results of lesion and microstimulation The systems for which the most progress has been
studies strongly implicate IT cells in face perception. made linking sensory neurons to perception tend to
Prosopagnosia, famously documented by Oliver Sacks have at least two things in common. First, experiment-
and others, is a well-described condition in humans ers have designed psychophysical tasks that allow them
thought to result from damage to the temporal lobe in to measure subtle changes in specific perceptual abili-
which patients are completely unable to recognize or ties. Second, the brain areas under study (in the primate
ruff and cohen: relating the activity of sensory neurons to perception 359
to correlated variability as the source of choice-predic- Cohen, M. R., & Maunsell, J. H. R. (2009). Attention
tive signals in individual MT neurons. It is in many ways improves performance primarily by reducing interneuro-
nal correlations. Nat Neurosci, 12(12), 1594–1600.
a high-level model, however, and does not take into
Cohen, M. R., & Maunsell, J. H. R. (2010). A neuronal
account factors like cell type, dynamics, or the pattern population measure of attention predicts behavioral
of connections different neurons make. As data sets get performance on individual trials. J Neurosci, 30(45),
more sophisticated, new theoretical frameworks will be 15241–15253.
necessary to convert data into understanding. Cohen, M. R., & Newsome, W. T. (2008). Context-dependent
Tremendous progress has been made on both theo- changes in functional circuitry in visual area MT. Neuron,
60(1), 162–173.
retical and experimental fronts since Parker and Cohen, M. R., & Newsome, W. T. (2009). Estimates of the
Newsome wrote their landmark review in 1998. As the contribution of single neurons to perception depend on
field continues to progress, their rubric has become timescale and noise correlation. J Neurosci, 29(20),
more important than ever. It is only through a princi- 6635–6648.
pled application of the technological advancements Chowdhury, S. A., & DeAngelis, G. C. (2008). Fine discrimi-
nation training alters the causal contribution of macaque
discussed above that we will improve our understanding area MT to depth perception. Neuron, 60(2), 367–377.
of the relationship between neural activity and Cury, K. M., & Uchida, N. (2010). Robust odor coding via
perception. inhalation-coupled transient activity in the mammalian
olfactory bulb. Neuron, 68(3), 570–585.
ACKNOWLEDGMENTS The authors are supported by NIH grants Desimone, R., Albright, T., Gross, C., & Bruce, C. (1984).
4R00EY020844-03 and R01 EY022930 (MRC), a training grant Stimulus-selective properties of inferior temporal neurons
slot on NIH 5T32NS7391-14 (DAR), a Whitehall Fellowship in the macaque. J Neurosci, 4(8), 2051–2062.
(MRC), and a Klingenstein Fellowship (MRC). We thank Desimone, R., & Duncan, J. (1995). Neural mechanisms of
David Montez, Regina Chang, and Trevor Stoltzfus for helpful selective visual attention. Annu Rev Neurosci, 18, 193–222.
comments on an earlier version of the chapter. Diester, I., Kaufman, M. T., Mogri, M., Pashaie, R.,
Goo, W., Yizhar, O., … Shenoy, K. V. (2011). An optoge-
netic toolbox designed for primates. Nat Neurosci, 14(3),
REFERENCES 387–397.
Espinosa, I., & Gerstein, G. (1988). Cortical auditory neuron
Abbott, L. F., & Dayan, P. (1999). The effect of correlated interactions during presentation of 3-tone sequences: Effec-
variability on the accuracy of a population code. Neural tive connectivity. Brain Res, 450(1), 39–50.
Comput, 11(1), 91–101. Gerdjikov, T. V, Bergner, C. G., Stüttgen, M. C.,
Aertsen, A. M., Gerstein, G. L., Habib, M. K., & Palm, G. Waiblinger, C., & Schwarz, C. (2010). Discrimination of
(1989). Dynamics of neuronal firing correlation: Modula- vibrotactile stimuli in the rat whisker system: Behavior and
tion of “effective connectivity.” J Neurophysiol, 61(5), neurometrics. Neuron, 65(4), 530–540.
900–917. Graziano, M. (2006). The organization of behavioral reper-
Afraz, S.-R., Kiani, R., & Esteky, H. (2006). Microstimula- toire in motor cortex. Annu Rev Neurosci, 29, 105–134.
tion of inferotemporal cortex influences face categoriza- Green, D. M., & Swets, J. A. (1966). Signal detection theory and
tion. Nature, 442(7103), 692–695. psychophysics. New York, NY: Wiley.
Ahissar, E., Vaadia, E., Ahissar, M., & Bergman, H. (1992). Gutnisky, D. A., & Dragoi, V. (2008). Adaptive coding of
Dependence of cortical plasticity on correlated activity of visual information in neural populations. Nature, 452(7184),
single neurons and on behavioral context. Science, 257(5075), 220–224.
1412–1415. Han, X., Qian, X., Bernstein, J. G., Zhou, H.-H., Franzesi,
Bosking, W. H., & Maunsell, J. H. R. (2011). Effects of G. T., Stern, P., … Boyden, E. S. (2009). Millisecond-tim-
stimulus direction on the correlation between behavior and escale optical control of neural dynamics in the nonhuman
single units in area MT during a motion detection task. primate brain. Neuron, 62(2), 191–198.
J Neurosci, 31(22), 8230–8238. Heekeren, H. R., Marrett, S., Bandettini, P. A., & Unger-
Britten, K. H., Newsome, W. T., Shadlen, M. N., Celebrini, leider, L. G. (2004). A general mechanism for perceptual
S., & Movshon, J. A. (1996). A relationship between behav- decision-making in the human brain. Nature, 431(7010),
ioral choice and the visual responses of neurons in macaque 859–862.
MT. Vis Neurosci, 13(1), 87–100. Helmchen, F., & Denk, W. (2005). Deep tissue two-photon
Britten, K. H., Shadlen, M. N., Newsome, W. T., & microscopy. Nat Methods, 2(12), 932–940.
Movshon, J. A. (1992). The analysis of visual motion: A Histed, M., Bonin, V., & Reid, R. (2009). Direct activation of
comparison of neuronal and psychophysical performance. sparse, distributed populations of cortical neurons by elec-
J Neurosci, 12(12), 4745–4765. trical microstimulation. Neuron, 63(4), 508–522.
Brunton, B. W., Botvinick, M. M., & Brody, C. D. (2013). Jazayeri, M., Lindbloom-Brown, Z., & Horwitz, G. D.
Rats and humans can optimally accumulate evidence for (2012). Saccadic eye movements evoked by optogenetic
decision-making. Science, 340(6128), 95–98. activation of primate V1. Nat Neurosci, 15(10), 1368–1370.
Carandini, M., & Churchland, A. K. (2013). Probing per- Kanwisher, N., McDermott, J., & Chun, M. M. (1997).
ceptual decisions in rodents. Nat Neurosci, 16(7), 824–831. The fusiform face area: A module in human extrastriate
Cohen, M. R., & Kohn, A. (2011). Measuring and interpret- cortex specialized for face perception. J Neurosci, 17(11),
ing neuronal correlations. Nat Neurosci, 14(7), 811–819. 4302–4311.
ruff and cohen: relating the activity of sensory neurons to perception 361
32 Characterizing the Effects of
Stimulus and Neural Variability
on Perceptual Performance
WILSON S. GEISLER, JOHANNES BURGE, MELCHI M. MICHEL, AND ANTHONY D. D’ANTONA
ABSTRACT Perceptual performance is limited by both external sensory organs. For example, the quantum nature of
and internal factors. External factors include the physical light causes the number of photopigment molecules
variability of sensory stimuli and the inherent ambiguities that activated in a photoreceptor to vary according to the
exist in the mapping between the properties of the environ-
ment and the properties of stimuli at the sensory organs Poisson probability distribution, even when the stimu-
(natural scene statistics). Internal factors include neural noise lus is nominally the same (de Vries, 1943; Hecht, Shlaer,
and nonrandom computational inefficiencies. External & Pirenne, 1942; Rose, 1948). Although this source of
factors have not received the study they deserve, perhaps noise is ubiquitous, there are only a few situations where
because they are difficult to measure and because methods it is the primary factor limiting performance. These
for characterizing them have not been standardized. This
chapter describes some of the computational tools used to
situations consist primarily of simple detection or dis-
characterize the effects of external and internal variability on crimination tasks where brief, spatially localized targets
perceptual performance. These tools are based on concepts are presented in the visual periphery under dark-
of Bayesian statistical decision theory and are illustrated for adapted conditions when the rod photoreceptors are
several basic natural tasks: grouping of contours across occlu- most sensitive (Hecht et al., 1942).
sions, estimation of binocular disparity, and interpolation of
In some laboratory tasks, it is possible to avoid all
missing pixel-luminance values.
sources of stimulus noise, other than irreducible sources
such as photon noise. In such tasks, performance is
Evolution pushes sensory and perceptual systems to usually dominated by neural variability, and by limita-
perform efficiently in those tasks necessary for the tions in neural computations. Examples of such cases
organism to survive and reproduce. Nonetheless, even would be simple detection or discrimination tasks with
in an organism’s natural tasks, perceptual performance fixed stimuli presented under light-adapted conditions
can never be perfect. Thus, to understand and predict (e.g., detection of a known pattern on a uniform gray
perceptual performance it is crucial to characterize and background). In other laboratory tasks, and in most
understand the many factors that limit performance. natural tasks, additional sources of stimulus variability
These factors include the complexity and variability of are also major factors. Examples of such cases would be
the sensory stimuli, as well as many sources of internal detection of targets in pixel-noise backgrounds (Burgess,
variability, ranging from noise in sensory receptor Wagner, Jennings, & Barlow, 1981), or estimation of
responses, to noise in decision and memory circuits, to physical properties in the environment such as the
noise in motor neuron responses. The aim of this depth, shape, and reflectance of object surfaces (e.g.,
chapter is to describe some of the computational tools see Geisler, 2011; Kersten, Mamassian, & Yuille, 2004).
used to characterize and understand the effects of Most perceptual tasks can be regarded as decision
external (stimulus) and internal sources of variability making in the presence of random variability, and
on perceptual performance. These tools are based on hence an appropriate theoretical framework for analyz-
principles of statistical decision and estimation theory. ing perceptual performance is Bayesian statistical deci-
The computational tools described here are applicable sion theory (e.g., see Geisler, 2011; Kersten et al., 2004;
to many perceptual systems, but the examples are drawn Knill & Richards, 1996). In what follows, we sketch the
from the vision literature. general Bayesian framework and then discuss several
The most basic kinds of stimulus variability are irre- special cases, starting with a discussion of simple detec-
ducible sources of noise that occur in transmission of tion and discrimination tasks and ending with discus-
stimulus information from the environment to the sion of optimal estimation in natural scenes.
geisler et al.: characterizing the effects of stimulus and neural variability 363
It is important to note that Bayesian statistical deci- a positive value u when the response is correct (γ (a, a)
sion theory is used to analyze perceptual performance = γ (b, b) = u), and –u when the response is incorrect
in two different ways. The first is to derive ideal-observer (γ (a, b) = γ (b, a) = −u). As another example, if the goal
models, which are theoretical devices that perform a is to maximize accuracy, while keeping false-positive
perceptual task optimally. An ideal observer usually responses (saying an absent target is present) at some
contains no free parameters and is not meant to be a low rate, then that can be represented by assigning
model of real observers. Rather, its purpose is (1) to a greater cost to false-positive than false-negative
help identify task-relevant stimulus properties, (2) to responses, that is, making γ (b, a) < γ (a, b). In the
describe how those properties should be used to perform depth-estimation task there are many more possible
the task of interest, (3) to provide a rigorous bench- combinations of response and state of the world, and
mark against which to compare real perceptual systems, hence many more possible goals (utility functions). A
and (4) to suggest principled hypotheses and models typical goal would be to minimize the mean squared
for real performance. error, which would be obtained by setting γ (r, ω) = −(r
The second way Bayesian statistical decision theory is − ω)2. In an ideal-observer model, the utility function
used is as a framework for modeling perceptual perfor- (goal) is fully specified. In a perceptual-performance
mance. When used in modeling perception, there are model, the utility function is a part of the model and
generally hypothesized internal (neural or information- may have free parameters.
processing) mechanisms, which have unknown param-
eters that are estimated from perceptual performance Ground Truth and Input Stimuli The second step
data. in using Bayesian statistical decision theory is to specify
the statistical relationship between the states of the
Bayesian statistical decision theory world and the proximal stimulus. In the most common
case, this involves specifying the conditional probability
Specifying the Task The first step in using Bayesian of the different ground-truth states of the world given
statistical decision theory is to specify the task. This the input stimuli; this is the posterior probability distri-
includes specifying the set of possible stimuli, the set of bution, p(ω|s). In practice, it is often convenient to first
possible responses, and the goal of the task. Specifying specify the stimulus likelihood distribution p(s|ω) for
the set of possible stimuli typically requires specifying each possible state of the world and the prior probabil-
(1) the ground-truth (distal) stimuli, which reflect the ity distribution p(ω), and then use Bayes’s rule to
true task-relevant state of the world ω and (2) the proxi- compute the posterior probability distribution.
mal stimuli, which constitute the input data s. For
example, in a simple detection-in-noise task, the true
Input Data In many applications of Bayesian statisti-
state of the world is that a target is either absent (ω =
cal decision theory, there are properties of the percep-
a) or present (ω = b) in a noise pattern, and the proxi-
tual system that are part of the specification of the input
mal stimulus is the specific pattern of pixels that would
to the optimal Bayesian computations. These properties
be imaged on the retina. In a typical depth-estimation
could be either known physical or neural properties
task, the true state of the world is the physical distance
that have no free parameters, or models of these prop-
of one surface patch from another, and the proximal
erties that have free parameters. For example, these
stimulus is the specific pattern of pixels imaged on the
properties might include the optics of the eye, the sam-
two retinas.
pling pattern of the photoreceptors, or the tuning and
The set of possible responses can be quite complex,
noise characteristics of retinal ganglion cells. The prop-
but in most perception experiments it is simple. For
erties can be represented by a function gθ that maps the
example, in the detection-in-noise task it would be one
input stimulus s onto input data,
of two responses that indicate whether the observer
judged the target to be absent (r = a) or present (r = z = g q (s ) , (1)
b). In the depth-estimation task, the response might be
an estimate of the number of centimeters in depth where θ represents any free parameters. In other words,
separating the surface patches. this constraint function incorporates the effects of
Specifying the goal of a task requires specifying the known or assumed properties, and its output is the
costs and benefits (utility) of each possible response for input data to the Bayesian analysis.
each possible state of the world: γ (r, ω). If the goal in
the detection-in-noise task is to be as accurate as pos- Bayes Optimal Response Once the task, input data,
sible, then that can be represented by making the utility and posterior distributions are specified, it is possible
geisler et al.: characterizing the effects of stimulus and neural variability 365
Figure 32.1 Detection and discrimination. (A) Distribution (D) Three parameters describing the possible geometrical
of decision variable values conditioned on two different states relationships between two contour elements. (E) Samples of
of the world (a and b). (B) The receiver operating character- geometrical relationships for pairs of contour elements
istic for the distributions in A. (C) Samples from hypothetical belonging to the same physical contour (blue symbols) and
Gaussian likelihood distributions for the two stimulus catego- belonging to different physical contours (red symbols), for a
ries in a yes-no detection task. The log of the ratio of these particular distance between the elements. The black curve
two likelihood distributions at any point is the decision shows the optimal decision bound given equal prior probabili-
variable in A; that is, Ψ = log[p(z1, z2|b)/p(z1, z2|a)]. The solid ties. (Data from Geisler & Perry, 2009.) (See color plate 28.)
and dashed contours correspond to the two criteria in A.
2AFC task to be √2 larger than in the yes-no task, for corresponding to the solid and dashed criteria, respec-
the same stimuli (Green & Swets, 1966). tively, in figure 32.1A.
The assumptions of signal-detection theory can be
tested in part by inducing the observer to adopt
Receiver Operating Characteristic Analysis The different decision criteria and then seeing if the observ-
effect on performance of changes in the decision crite- er’s hit and false alarm rates fall on the ROC predicted
rion, for a given level of sensitivity, can be represented for the measured value of d′. Typically, different deci-
with the receiver operating characteristic (ROC), sion criteria are induced by varying the relative proba-
which plots the proportion of hits as a function of the bility of stimulus a and b, varying the monetary payoffs
proportion of false alarms (figure 32.1B). Note that in for the different kinds of correct and error responses,
this plot the value of the decision criterion is implicit; varying the instructions to observer, or by asking the
the ROC shows only the locus of hit and false alarm observer to provide a confidence judgment along with
rates for all values of the criterion. The shape of the each response (Green & Swets, 1966). Signal-detection
ROC depends on the shapes of the probability distri- theory predicts that in the 2AFC task the ROC curve
butions; the ROCs plotted in figure 32.1B are for the should be symmetric about the negative diagonal, even
distributions in figure 32.1A. The solid and open when it is not predicted to be symmetric in the yes-no
symbols in figure 32.1B show points on the ROC curve task (e.g., see Green & Swets, 1966).
geisler et al.: characterizing the effects of stimulus and neural variability 367
words, signal-detection theory assumes an arbitrary and ideal performance is nearly identical, implying
rather than the ideal decision variable and criterion. that the human visual system accurately applies the
To obtain quantitative predictions for the ideal decision boundary in figure 32.1E (Geisler & Perry,
observer in an identification task, it is necessary to 2009).
specify the prior probability of the different stimulus We reiterate that ideal observers are not meant to be
categories and the likelihood of the input data for each models of real observers. Rather, they provide a rigor-
of the categories. Specifying the likelihoods and priors ous benchmark against which to compare real observ-
can be very difficult. The most common approach is to ers, and a principled starting point for developing
constrain the stimuli so that it is practical to derive or models for real performance. For example, the classic
compute the likelihoods and priors. One way of con- signal-detection model for interpreting performance in
straining stimuli is to create them by sampling from detection and discrimination experiments is motivated
probability distributions specified by the experimenter. by the computational principles of the ideal observer.
This is what is done in many perception experiments. In addition, there are many examples in the perception
For example, stimulus a might be a sample of Gaussian literature where human performance is found to paral-
noise, and stimulus b a sample of Gaussian noise with a lel that of the ideal observer, showing that modest modi-
fixed added target. In this case, the decision variable fications of ideal observers (or heuristic approximations
and the criterion can be easily computed, making it to the ideal observer) can serve as plausible and testable
straightforward to calculate or simulate ideal perfor- models for real performance (for a review, see Geisler,
mance (e.g., Burgess et al., 1981). 2011).
Working with natural stimuli is more difficult,
because their statistical structure is complex and gen- Estimation Tasks In estimation tasks, there is some
erally unknown. One approach is to restrict what physically ordered dimension along which stimuli fall,
aspects of the natural stimuli are presented in an and the observer is required to estimate the value along
experiment. By considering only certain aspects of that dimension. The distinction between estimation
natural stimuli, it can become practical to measure the and identification is not a sharp one, because one can
relevant probability distributions and compute ideal- regard estimation as identification with a large number
observer performance. For example, consider a task of categories. The primary distinction is captured in the
where the observer is presented with just two contour utility (cost/benefit) function. Generally, in the estima-
elements (short line segments) at the boundary of an tion task, the closer the estimate to the true value, the
occluding surface, and must decide whether the ele- better. On the other hand, in many identification tasks
ments belong to the same or different physical con- all errors are equally costly; for example, if the task is
tours (Geisler & Perry, 2009). For a given occluder to identify a criminal from a police lineup of otherwise
width (distance), two parameters describe the geomet- innocent people, then all errors would be equally bad.
rical relationship between two contour elements: the Estimation tasks are very common under natural condi-
direction of one element from the other, φ, and the tions, but in laboratory settings they are less common
orientation difference between the two elements, θ than identification tasks.
(see figure 32.1D). The blue symbols in figure 32.1E The typical method for measuring estimation perfor-
show samples from the actual distribution in natural mance is similar to that for measuring discrimination
images of direction and orientation difference when performance. On each trial, a variable test stimulus is
the contour elements belong to the same contour; the presented, and the observer is required to respond
red symbols show samples when the elements belong whether it is greater or less than some standard along
to different contours. The solid curve shows the ideal the stimulus dimension of interest (e.g., color, depth,
decision bound when the goal is to maximize accuracy. size, shape, etc.). Often the standard is another stimu-
Specifically, the ideal observer should report that the lus, but in some tasks the standard may be an internal
contour elements are on the same contour if the direc- reference. For example, in a slant-estimation task (e.g.,
tion and orientation difference fall inside the bound- Burge, Girshick, & Banks, 2010), the observer may be
ary; otherwise, the observer should report that the required to respond whether a test stimulus is right-
elements are from different contours. The perfor- side-back from frontoparallel (an internal standard).
mance of this ideal observer can be determined by Data are typically plotted as psychometric functions,
applying this decision rule to test stimuli that contain and the estimate is taken to be the point of subjective
two contour elements (taken from natural scenes) sep- equality (PSE)—the value of the variable stimulus
arated by an occluder. These same test stimuli can be where the observer reports that the test is greater than
presented to human observers. In this case, human the standard with probability 0.5.
( ),
the estimate and the true value. This is the MMSE esti-
mate given by p (c ω ) = gauss c; m ω , Σ ω
(8)
geisler et al.: characterizing the effects of stimulus and neural variability 369
Figure 32.2 Simple nonparametric minimum mean squared error estimates. (A) Parallel conditional means. (B) Recursive
conditional means.
for all possible values of the variables in this second- from the distribution.) On the other hand, the RCM
level context. The result is a second estimation function observer is “discriminative,” in the sense that it provides
that maps second-level context values into optimal esti- optimal estimates, but does not specify the joint distri-
mates: ω̂ 2 = E (ω c 2 ). This process can be repeated to bution of the context and true values (McLachlan,
obtain a series of n estimation functions; the value of n 1992; Vapnik, 1998). The distinction between genera-
is determined by when performance reaches asymptote. tive and discriminative is separate from the distinction
The final estimate is obtained by applying the n estima- between parametric and nonparametric. For example,
tion functions sequentially. direct nonparametric measurements of higher-order
Which version performs best depends on the particu- moments beyond the conditional mean may allow gen-
lar task. Both versions require having enough training eration of random samples from the joint distribution.
data to estimate the mean of ω or each possible pattern Alternatively, parametric models such as multiple linear
of context values. For example, if the values of the regression produce estimates, but cannot generate
context variables range from 0 to 255, then the context random samples from the joint distribution. In percep-
size is limited to three or four variables, because more tual systems, a potential advantage of representing the
variables would require an impractical amount of train- joint distribution is that it may be possible to switch
ing data. However, the context size does grow (in effect) utility/cost functions without needing to learn whole
at each step. In the recursive case, the effective context new estimation functions.
grows because the context for a higher-level estimate We illustrate the two approaches with two examples:
contains estimates that were obtained using the con- (1) disparity estimation, which underlies binocular
texts at lower levels (in figure 32.2B, the context for the depth perception, and (2) missing-pixel estimation,
second estimate can effectively include the light gray which is a simple form of image interpolation (amodal
pixels). Note that it is also possible to apply Gaussian completion).
mixture models recursively.
Another distinction between these two approaches is Disparity estimation To illustrate the first approach,
that the GMM observer is “generative,” in the sense that consider the task of estimating horizontal disparity
the GMM parameters specify the joint distribution of from the images formed in the left and right eye when
the context and true values. (The term “generative” binocularly viewing a small patch of natural scene. In
refers to the fact that if the joint distribution is speci- this example, the context consists of eight variables,
fied, then it is possible to generate random samples where each variable is the dot product of a different,
geisler et al.: characterizing the effects of stimulus and neural variability 371
Figure 32.4 Missing-pixel estimation task. (A) Example test Gaussian. (D) Estimation of error of a parallel conditional
image (calibrated 8-bit gray scale). (B) Distribution of estima- means (PCM) observer trained on natural images plotted as
tion errors (in gray-level) of the point of subject equality for a function of the estimation error of three human observers,
three human observers on 62 test patches. (C) Distribution for 62 test patches. If PCM and human errors were identical,
of psychometric function slopes (standard deviation values) the points would fall on the positive diagonal.
obtained by fitting psychometric data with a cumulative
in this task. However, because of luminance gain control (stimulus) variability must be scaled up for the perfor-
and center-surround mechanisms in the retina, the mance of the ideal observer to match that of the organ-
output of the retina is probably better described as a ism. This scale factor κ is closely related to the definition
contrast image rather than a luminance image (a con- of efficiency, η in signal-detection theory: η = dreal′ 2 dideal
′2
trast image is obtained from a luminance image by (Tanner & Birdsall, 1958). In a detection task with a
subtracting and then dividing by the local mean lumi- fixed signal in Gaussian noise, κ is simply the inverse of
nance at each pixel location). Interestingly, humans the efficiency (κ = 1/η). As a more general example,
match the performance of a PCM observer trained on consider an ideal observer in an identification task,
contrast images. Figure 32.4D shows, for 62 representa- where each category is represented by a Gaussian dis-
tive test patches, the estimation error of the contrast tribution. In this case, κ is the scale factor on the covari-
PCM observer plotted against the human estimation ance matrices that brings the ideal performance down
error. The contrast PCM observer does a good job of to real performance. If the external variability must be
predicting the specific errors made by humans for arbi- scaled by a factor of κ, then the effective internal vari-
trary natural image patches. ability equals κ − 1 times the external variability. In
other words, if the value of κ is near 1.0, then the inter-
nal noise is near zero and external factors dominate
Relative influence of external and internal factors performance; if the value of κ is large, then internal
factors dominate performance.
Efficiency An observer’s performance is generally Whether external or internal factors dominate per-
limited by both external factors (variability and ambigu- formance is highly task-dependent. For detection of
ity of the inputs) and by internal factors (neural, deci- targets in fixed backgrounds, the only external variabil-
sion, and motor noise, as well as nonrandom ity is photon noise (the ideal observer is limited only by
computational inefficiencies). For the purpose of esti- the signal’s energy and photon noise) and the value of
mating the relative influence of external and internal κ is large (typically greater than 10), showing that inter-
factors, the combined effect of all the internal factors nal factors dominate (e.g., Geisler, 1989). For detection
can be regarded as a level of internal noise, which can in high-contrast pixel noise, the values of κ can be quite
be estimated by calculating how much the external a bit smaller (sometimes less than 2), showing that
geisler et al.: characterizing the effects of stimulus and neural variability 373
Burge, J., & Geisler, W. S. (2014). Optimal disparity estima- Kersten, D., Mamassian, P., & Yuille, A. L. (2004). Object
tion in natural stereo images. J Vis, 14(2), 1–18. perception as Bayesian inference. Annu Rev Psychol, 55,
Burge, J., Girshick, A. R., & Banks, M. S. (2010). Visual- 271–304.
haptic recalibration is determined by relative reliability. Knill, D. C., & Richards, W. (Eds.). (1996). Perception as
J Neurosci, 30(22), 7714–7721. Bayesian inference. Cambridge, UK: Cambridge University
Burgess, A. E., Wagner, R. F., Jennings, R. J., & Barlow, Press.
H. B. (1981). Efficiency of human visual signal discrimina- Landers, D. D., & Cormack, L. K. (1997). Asymmetries and
tion. Science, 214, 93–94. errors in perception of depth from disparity suggest a mul-
D’Antona, A. D., Perry, J. S., & Geisler, W. S. (2013). ticomponent model of disparity processing. Percept Psycho-
Humans make efficient use of natural image statistics when phys, 59, 219–231.
performing spatial interpolation. J Vis, 13(14), 1–13. McKee, S. P., Levi, D. M., & Bowne, S. F. (1990). The impreci-
De Vries, H. L. (1943). The quantum character of light and sion of stereopsis. Vis Res, 30, 1763–1779.
its bearing upon threshold of vision, the differential sensi- McLachlan, G. J. (1992). Discriminant analysis and statistical
tivity and visual acuity of the eye. Physica, 10(7), 553–564. pattern recognition. New York, NY: Wiley.
Deriugin, N. (1956). The power spectrum and the correla- Oruc, I., Maloney, L. T., & Landy, M. S. (2003). Weighted
tion function of the television signal. Telecommunications, linear cue combination with possibly correlated error. Vis
1(7), 1–12. Res, 43, 2451–2468.
Field, D. J. (1987). Relations between the statistics of natural Rose, A. (1948). The sensitivity performance of the human
images and the response properties of cortical cells. J Opt eye on an absolute scale. J Opt Soc Am, 38(2), 196–208.
Soc Am A, 4(12), 2379–2394. Swensson, R. G., & Judy, P. F. (1996). Measuring performance
Geisler, W. S. (1989). Sequential ideal-observer analysis of efficiency and consistency in visual discriminations with
visual discrimination. Psychol Rev, 96, 267–314. noise images. J Exp Psych, 22(6), 1393–1415.
Geisler, W. S. (2011). Contributions of ideal observer theory Tanner, W. P., & Birdsall, T. G. (1958). Definitions of d ′ and
to vision research. Vis Res, 51, 771–781. η as psychophysical measures. J Acoust Soc Am, 30(10),
Geisler, W. S., & Perry, J. S. (2009). Contour statistics in 922–928.
natural images: Grouping across occlusions. Vis Neurosci, 26, Tanner, W. P., & Swets, J. A. (1954). A decision-making
109–121. theory of visual detection. Psychol Rev, 61(6), 401–409.
Geisler, W. S., & Perry, J. S. (2011). Statistics for optimal Tolhurst, D. J., Movshon, J. A., & Dean, A. F. (1983). The
point prediction in natural images. J Vis, 11(12), 1–17. statistical reliability of signals in single neurons in the cat
Green, D. M., & Swets, J. A. (1966). Signal detection theory and and monkey visual cortex. Vis Res, 23(8), 775–785.
psychophysics. New York, NY: Wiley. Vapnik, V. N. (1998). Statistical learning theory. New York, NY:
Hecht, S., Shlaer, S., & Pirenne, M. H. (1942). Energy, Wiley.
quanta, and vision. J Gen Physiol, 25, 819–840.
Kersten, D. (1987). Predictability and redundancy of natural
images. J Opt Soc Am A, 4(12), 2395–2400.
ABSTRACT A fundamental problem in neuroscience is to throughout the pathway are linked to sensation and
understand how neural activity is related to behavior. This perception, and (2) how information is transformed
neural coding problem combines psychophysical data with and represented within those pathways.
neurophysiology, and it has been largely applied in two kinds
of psychophysical tasks, detection and discrimination tasks,
In this chapter we discuss two concepts related to
and subjective magnitude estimate (SME) tasks. The aim of sensation and perception. The first is the concept of
the psychophysical study is to quantify the relationship quantifying the neural activity with behavior, which is
between the stimulus and perception, whereas the aim of the called the neural coding problem. The rationale is that
neurophysiological study is to draw a relationship between embedded within the spatial and temporal patterns of
the stimulus and neural responses. The neural code is then
action potentials is a neural representation of the
determined by finding the link that relates the two kinds of
data. The lower-envelope principle is used to determine the sensory stimulus. The aim of neural coding studies is to
neural code for detection and discrimination tasks, while the determine how these representations are related to
principles of consistency and falsification are used to deter- behavior. We first discuss classical neural coding studies
mine the neural code for SME tasks. Throughout the chapter, of the tactile system and provide a theoretical frame-
we discuss neural coding studies of somatosensory function.
work for linking neural activity recorded from nonhu-
Particularly, we will review studies of roughness, orientation,
curvature, and motion, which show that the neural mecha- man primates with human behavior.
nisms for coding local cues are based on extracting two- The second concept is that, under conditions when
dimensional (2D) features of the stimulus from the spatial the form of the input representations are the same
activity across receptors. The mechanisms for spatial process- across sensory modalities, the nervous system appears
ing are highly similar between vision and touch. However, to use similar coding mechanisms to solve similar prob-
the mechanisms for coding global features of object size and
shape are different between vision and touch. lems. To illustrate this second concept, we discuss how
objects are represented in the somatosensory system
and draw parallels with neural mechanisms in the visual
Sensory systems are faced with the common problem of modality. We show that tactile object recognition is
encoding and transforming inputs from the environ- based on processing of the global and local features of
ment into patterns of action potentials that give rise to objects. Results from a number of studies show that
memory and perception. In touch, this process begins neural coding mechanisms for processing local features
with the activation of a large number of receptors in of objects, which rely on 2D arrays of receptors in the
the skin that have specificity for different environmen- skin, are similar between touch and vision. However
tal energies, such as mechanical or thermal inputs to we will show that mechanisms for processing global
the skin, and specificity for forces, positions, and move- features, which depend on integrating inputs across
ments of our limbs. Tactile perception of mechanical fingers, are different.
inputs alone has four different kinds of receptors
located in the skin that are sensitive to different aspects Neural coding
of mechanical energy. Each receptor is associated with
a particular afferent fiber that is thought to encode In the 1960s, Vernon Mountcastle initially suggested
information about different features of the input. This that the neural mechanisms underlying perception can
division of labor among the peripheral nerves results in be studied directly. The approach is to first select an
the central nervous system receiving parallel represen- aspect of perception and perform quantitative psycho-
tations of the peripheral input that are processed in physical experiments in humans to determine how
different regions of cortex. Along each pathway, these perception changes as a function of the stimulus’s
images are transformed by convergent and divergent parameters. Neurophysiological experiments are then
synaptic circuits into central representations of objects performed on nonhuman primates using the same
that are matched against stored memories. The broad stimulus and stimulus conditions to determine how
goals of tactile research are to determine (1) which neurons are modulated by the stimulus. Given the cor-
neuronal population, or populations, at each stage respondence in neural mechanisms between the two
Figure 33.1 Neural coding of discrimination and subjective from Mountcastle et al., 1972, and Freeman & Johnson,
magnitude estimation (SME) studies. (A) The graph shows 1982.) (B) Summary of four coding studies of roughness
the sensitivity (threshold firing) of the most sensitive SA1, RA, perception. Y-axis, SME. X-axis, spatial variation of firing rates
and PC afferents to vibrations. RA afferents account for the of SA1 afferents (see text for details). The correlation is equal
lower limb, and PC afferents the upper limb, of the human or greater than 0.97 in all studies.
psychophysical curve (dashed line. (Adapted with permission
largely depend on information gathered from individ- RA afferents also innervate the skin densely (150 affer-
ual fingers and have been shown to play a role in object ents/cm2), but convey a less acute spatial image of the
recognition (Klatzky & Lederman, 1995; Lederman & stimulus because the receptors they innervate have
Klatzky, 1997). Indeed, psychophysical evidence sug- large, uniform RFs (∼3–5 mm in diameter). The SA2
gests that the local and global features of objects are and PC afferents have large RFs that sparsely innervate
processed independently, with material properties pro- the skin, and unlike the SA1 and RA afferents, play no
cessed separately from shape properties (Klatzky & role in 2D form processing. The SA2 afferents provide
Lederman, 1995). Understanding how objects are information about skin stretch and joint angle, and are
coded in cortex requires that we first understand how thought to play a role in 3D shape perception. The PC
individual features of objects are coded. afferents provide information about vibration (see
Figure 33.2 shows a working hypothesis of how we above) and play an important role in tool use (Johnson,
envision the differences between global and local 2001).
feature processing. We hypothesize that global proper- The neural mechanisms underlying proprioception
ties of size and shape are derived by integrating cutane- are poorly understood. In addition to the SA2 afferents,
ous inputs from mechanoreceptors innervating the skin there are four other types of afferents providing pro-
with proprioceptive receptors in the skin, muscles, and prioceptive input. These are the Golgi tendon organs
joints. Proprioception provides information about (1b afferents), muscle spindles (1a and II), and joint
where the fingers contacting the object lie in space. The afferents. Of these, only the muscle spindles and SA2
local properties of the object are derived from cutane- afferents are thought to provide information about
ous receptors that provide 2D images of the spatial and joint angle. The relative contributions of these afferents
temporal pattern of activity on the skin. to joint-angle perception are not well understood. Psy-
In the next section, we briefly describe the anatomi- chophysical studies show that activation of either affer-
cal pathways underlying tactile perception and then ent can alter perception of joint angle (Collins,
describe neural coding studies of how individual fea- Refshauge, & Gandevia, 2000; Goodwin, McCloskey, &
tures are represented in the brain. Matthews, 1972), and neurophysiological studies show
that the responses of SA2 afferents to different joint-
Anatomical Pathways Tactile object processing angle configurations closely match human psychophysi-
begins with the activation of four types of cutaneous cal reports (Dimitriou & Edin, 2008; Edin & Abbs,
afferents, namely, the SA1, RA, slowly adapting type 2 1991).
(SA2), PC, and proprioceptive afferents (see Hsiao & After leaving the arm, the main cutaneous and pro-
Gomez-Ramirez, 2012, for a review). There are about prioceptive pathway of peripheral afferents project to
100 SA1 afferents/cm2 at the fingertips in both human the dorsal column nuclei in the brain stem, which in
and nonhuman primates (Darian-Smith & Kenins, turn project to neurons in the ventroposterior lateral
1980; Johansson & Vallbo, 1979), with each afferent nucleus (VPL) of the thalamus, which itself projects to
having circular receptive fields (RFs) of about 2–3 mm neurons in primary somatosensory cortex (S1). Neuro-
in diameter that have a Gaussian-like shape (Hsiao, physiological evidence suggests that minimal process-
Fitzgerald, Thakur, Denchev, & Yoshioka, 2006). The ing occurs along these ascending pathways (for a review,
DiCarlo & Johnson, 2000; Hsiao et al., 2002; Sripati, notion that the SA1 system plays a prominent role in
Yoshioka, Denchev, Hsiao, & Johnson, 2006; Thakur, encoding tactile spatial information.
Fitzgerald, & Hsiao, 2012), like neurons in primary Studies reveal that tuning for contour features (e.g.,
visual cortex (Hubel, 1958). This emergent property of curvature) emerges in area 2 and SII (Yau, Pasupathy,
orientation selectivity is mediated by neurons that have Fitzgerald, Hsiao, & Connor, 2009; Yau et al., 2013).
RFs with oriented excitatory centers and flanking inhib- Neurons in both areas respond to curves pointing in a
itory regions that also have oriented shapes (DiCarlo & specific direction, with the representations of curvature
Johnson, 2000; DiCarlo et al., 1998; Sripati et al., 2006; developing and peaking concurrently in both areas
Thakur et al., 2012). The RF structures are similar (Yau et al., 2013). Interestingly, the responses to tactile
to ones observed in primary visual cortex (Hubel & curved stimuli are similar to the responses in area V4
Wiesel, 1959). Orientation-selective neurons are also to visual curves (Yau et al., 2009).
observed in SII cortex, with the majority of neurons
showing position-invariant responses on a single finger 3D curvature Similar results are observed for coding of
(Fitzgerald et al., 2006; Thakur et al., 2006). The orien- 3D curves indented into a single finger pad, with thresh-
tation-discrimination threshold of about 20 degrees old discrimination of about 5 m−1 (Goodwin, Browning,
closely matches the threshold predicted by the corre- & Wheat, 1995) and the threshold being mildly affected
sponding population response of area 3b neurons by indentation force. Neurophysiological observations
(figure 33.3; Bensmaia, Denchev et al., 2008), thus indi- from human peripheral nerves show that only the 3D
cating (using the lower-envelope principle) that the spatial profile of the SA1 population provides a veridi-
neural code for tactile orientation resides in area 3b. cal representation of curvature, and that their activity
Next we investigate the representation of second-order accounts for human psychophysical observations
stimuli (i.e., curves) which can be either 2D and lie (Goodwin et al., 1995; LaMotte & Srinivasan, 1993).
within the plane of the skin (e.g., the letter “C”), or 3D RAs respond poorly to such stimuli and are unrelated
and indent into the skin with different depths (e.g., like to curvature discrimination (Goodwin et al., 1995;
a ball). Khalsa et al., 1998; LaMotte, Friedman, Lu, Khalsa, &
Srinivasan, 1998). The neural coding of 3D curvature
2D curvature Wheat and Goodwin (2001) showed in in cortex has not been studied.
human psychophysics that the discrimination threshold
for curvature in the plane of the skin is about 2D patterns composed of multiple features Further evi-
20 m−1. In corresponding peripheral neurophysiologi- dence that the SA1 system is essential for fine form
cal studies, which were also performed in humans, they processing, analogous to the parvocellular pathway in
showed that the neural representation of curvature is vision, comes from studies that use complex 2D pat-
an isomorphic spatial pattern based on the responses terns such as Braille dots or embossed letters, which are
of SA1s. The responses of the RA afferents did not composed of combinations of curves and oriented bars
match the psychophysical data, further supporting the (for details, see Hsiao & Gomez-Ramirez, 2012). If the
SA1 system is devoted to processing spatial form, then studies of tactile motion that is based on the activation
can complex patterns be recognized as easily in touch of the RA afferents.
as they are in vision? Phillips et al. (1983) showed that
when letters are scaled in height to span the same Motion When interacting with the world, we need to
number of receptors in touch and vision, the patterns represent not only static properties of objects but also
of confusions in an identification task are nearly dynamic features of how and where the objects are
identical. The results showed that complex 2D patterns moving. There have been a number of studies investi-
are perceived similarly in the two systems, suggesting gating tactile motion, with a series of them showing that
that the mechanisms of 2D form processing are neurons in S1 cortex are tuned to tactile motion
similar. (Costanzo & Gardner, 1980; DiCarlo & Johnson, 2000;
Subsequent studies showed that letter-recognition Pei, Hsiao, Craig, & Bensmaia, 2010; Ruiz, Crespo, &
performance in humans is not affected by active or Romo, 1995; Warren, Hämäläinen, & Gardner, 1986).
passive scanning (Vega-Bermudez et al., 1991). In an However, the coding of stimulus motion begins with the
attention study, Hsiao trained nonhuman primates to activation of RA afferents (Gardner & Palmer, 1989).
recognize embossed letters of the alphabet scanned These afferents are sensitive to motion but not to
across their finger pads (Hsiao et al., 1993). They motion direction (Pei et al., 2010). Further, Pei et al.
observed that the ability of nonhuman primates to rec- reported that while a large percentage of SI neurons
ognize letters is nearly identical to human performance, respond to motion signals (∼60% in areas 3b and 1 and
supporting the cross-species assumption that the mech- 30% in area 2), only neurons in area 1 showed motion
anisms of 2D form processing across the two species are tuning independent of the spatial form of the stimulus,
similar. and their population response closely matched humans’
Neurophysiological studies using embossed letters psychophysical data in a motion-discrimination task
show that both the RA and SA1 afferents provide an (figure 33.4). Pei et al. (2011) further examined the
isomorphic representation of the patterns (Phillips et responses of area 1 neurons to tactile, plaid moving
al., 1988), with the representation being more acute for patterns (pairs of gratings moving in different direc-
the SA1 compared to the RA system. The spatial pattern tions) and showed that, similar to vision, a subset of
of the peripheral SA1 afferents accounts well for the neurons encode the individual movement of each
psychophysical data in subjects performing letter-recog- grating composing the plaid stimulus (component
nition tasks (Vega-Bermudez et al., 1991). It is not neurons), while a separate set of neurons encode the
understood how spatial patterns composed of multiple combined motion across all gratings composing the
features of patterns like letters are represented in plaid (pattern neurons). In corresponding neurophysi-
cortex. In the next section, we review neural coding ological studies, Pei and his colleagues found that
ABSTRACT Human visual object decisions are believed to be overlapping fields of perceptual psychology, computer
based on a hierarchical organization of stages through which vision and robotics, and visual neuroscience.
image information is successively transformed from a large In all three fields, theories of representation of visual
number of local feature measurements with a small number
of types (e.g., edges at many locations) to increasingly lower-
knowledge and the processes acting on them are con-
dimensional representations of many types (e.g., dog, car, …). strained by (1) functional behaviors or tasks, and their
Functional utility requires integrating a large number of local priorities; (2) the statistical structure of the visual world,
features to reduce ambiguity, while at the same time selecting and consequently in images received; (3) algorithms
task-relevant information. For example, decisions requiring and knowledge structures for getting from images to
object recognition involve pathways in the hierarchy in which
behaviors; and (4) neurophysiological (or hardware)
representations become increasingly selective for specific
pattern types (e.g., boundaries, textures, shapes, parts, limitations on what can be computed by collections of
objects), together with increased invariance to transforma- neurons (or components and circuits).
tions such as translation, scale, and illumination. Computer There has been considerable growth in (4), our
vision architectures for object recognition and parsing, as well knowledge of the neurophysiology and anatomy of
as models of the primate ventral visual stream, are consistent
the primate visual system at the level of large-scale
with this hierarchical view of visual processing. The hierarchi-
cal model has been extraordinarily fruitful, providing qualita- organization of visual areas and their connections
tive explanations of behavioral and neurophysiological results. (Kanwisher, 2010; Kourtzi & Connor, 2011), and the
However, the computational processes carried out by the finer-scale level of cortical (Callaway, 1998; Lund,
visual hierarchy during object perception and recognition are Angelucci, & Bressloff, 2003; Markov et al., 2013) and
not well understood. This chapter describes how a Bayesian, subcortical neurocircuitry (Guillery & Sherman, 2002).
inferential perspective may help to explain the brain’s hierar-
chical organization of visual knowledge, and its utilization The larger picture is that visual processing involves
through the feedforward and feedback flow of information. processing within a visual area (both laterally and
across laminae), and hierarchical—feedforward and
feedback—processing between areas with various
It takes just one quick glance at the picture in figure feature selectivities (figure 34.2).
34.1A to see the fox, a tree trunk, some grass, and back- However, despite growth in our knowledge of the
ground twigs. This is a remarkable achievement in visual brain, there remains a gap in our understand-
which the visual system turns a massive set of highly ing of how the biology of vision enables common
ambiguous local measurements (figure 34.1B) into behaviors.1 An immediate problem faced when
accurate, reliable identifications. But that is just the beginning such an analysis is that the large-scale
beginning of what vision enables us to do with this systems nature of the problem makes it difficult to
picture. With a few more glances, one can see a whole empirically test theories of behavior at the level of
lot more: the shape of the fox’s legs and head, the neurons. One strategic solution is to temporarily
varying properties of its fur, and so on, as well as guess ignore the details of the neurophysiology and neuro-
what it is doing and whether it is young or old. The circuitry—that is, (4) above—and try to understand a
ability to generate an unbounded set of descriptions narrower problem: what are the representations,
from a virtually limitless number of images illustrates
the extraordinary versatility of human perception.
1
This chapter focuses on the following question: what Even complete knowledge of neural network connectivity
and dynamics would be insufficient to explain visual function.
knowledge representations and computational pro-
For example, a complete description of spatial-temporal
cesses are needed to achieve reliable and versatile switching of the billion-plus transistors in a video game
object vision? Although we are far from complete console would provide little insight into how these patterns
answers, there has been substantial progress in the relate to game goals, algorithms, or behavior.
kersten and yuille: inferential models of the visual cortical hierarchy 389
A B
Figure 34.1 (A) This figure illustrates two problems. spatial scales, such as the various edge and textural properties
(1) How can local measurements made from small patches of local regions B, the shape of parts C, and intermediate- and
(B), using neurons with small receptive fields, be integrated higher-level concepts such as “head” D, respectively. There is
to recognize objects and patterns (e.g., fox, tree trunk, grass)? a bootstrapping problem, in that the accurate interpretation
(2) How does the visual system support a limitless number of of any local patch is ambiguous without knowledge of the rest.
descriptions of a single scene? Answers need to account for (See color plate 30.)
flexible access to information of various types over a range of
Feedforward
Lateral
Feedback
Vn Vn+1
Figure 34.2 (A) Schematic of macaque monkey visual the dorsal and ventral streams, respectively. (B) Feedforward
cortex (figure reprinted from Wallisch & Movshon, 2008, with and feedback connections represent transmission of feedfor-
permission from Elsevier; see also Lennie, 1998). The colored ward and feedback signals between visual areas. Lateral (also
rectangles represent visual areas (see Felleman & Van Essen, called “horizontal”) organization within areas, representing
1991). The gray lines show the connections between areas, features of similar types and level of abstraction. (See color
with the thickness proportional to estimates of the number of plate 31.)
feedforward fibers. Areas in warm and cool tones belong to
kersten and yuille: inferential models of the visual cortical hierarchy 391
A B C D
m m m
s1 s2
s1 s2 s1 s2 next step s1 s2
? I1 I2
I1 I2 I1 I2 I1 I2
Figure 34.3 (A) Simple graph illustrating the generative (C) Bayesian coarse-to-fine inference. Different “models,” m,
constraints on incoming data. (See main text.) (B) More than can be different functions of the parameters s, which in turn
one combination of causes, s, could explain local image mea- lead to different image features. An initial, “quick and dirty”
surement, I1. Optimal perception seeks an explanation, i.e., visual inference may be at the top level (e.g., it is a “fox”),
values of s1 or s2 that give the most probable explanation for ignoring shape details (but using, for example, features from
how the image measurement could have been generated. For the wooded context, fur color, “features of intermediate com-
example, Bayes optimal calculations show that without feature plexity” or “fragments,” that may be sufficient). Fixing the
I2, s1 takes on one value, but with a measurement of I2, it takes hypothesis of “fox” can be followed by reliable inferences at
on a different value. Pearl (1988) calls this “explaining away.” a lower level (e.g., “shape of the head of the fox”).
maps an image pattern to the most probable hypothesis, Computer vision studies have shown discriminative
which in neural terms is not that different from a reflex and generative models can be combined (Tu et al.,
(Purves & Lotto, 2003).4 2005)—an algorithmic strategy similar in spirit to two-
Bayesian algorithms can also be generative. Genera- stage processing accounts of human visual recognition,
tive models rely on knowledge in the likelihood p(I|s), in which an initial, fast decision about the “gist” of a
which specifies how an image results from causes or scene narrows the space of specific objects to match to
explanations s and a prior p(s). These probabilities are the image (Bar, 2003).
related to the posterior through Bayes’s rule: p(s|I) = Bayesian probabilistic methods have been applied in
p(I|s)p(s)/p(I). Generative algorithms make explicit use a number of quantitative studies of human visual behav-
of top-down generative processes, in which high-level ior. There is a long history to studying human percep-
hypotheses are used to simulate the values of lower-level tion (and neural responses) using ideal-observer
nodes, ultimately generating a prediction of I (Mumford, analysis (Gold, Abbey, Tjan, & Kersten, 2009). Here one
1992; Yuille & Kersten, 2006). Generative models makes quantitative comparisons between what an ideal
provide a number of advantages. For example, by elabo- (Bayesian) observer can achieve with humans or
rating the structure of the likelihood, computational neurons (Geisler, 2011; Trenti, Barraza, & Eckstein,
studies have shown that a generative process can 2010). A strategic benefit of ideal-observer analysis in
improve recognition through “explaining away,” which studies of human behavior is that it helps to distinguish
is useful for both learning (Hinton, 2009; Zeiler, Taylor, perceptual limitations inherent to the information-
& Fergus, 2011), and inference applied to image parsing processing problem from limitations of the neural
(Tu, Chen, Yuille, & Zhu, 2005). Generative algorithms mechanisms (cf. Eckstein, Drescher, & Shimozaki, 2006;
predict appearances in time (e.g., Bayes-Kalman; Burgi, Weiss, Simoncelli, & Adelson, 2002).
Yuille, & Grzywacz, 2000), and cope more efficiently Quantitative behavioral experiments have shown
with a wider range of variability, such as the virtually near optimality or ideal-like behavior in a variety of
unlimited ways in which objects can be composed domains, including visual cue integration (Jacobs,
(Chang, Jin, Zhang, Borenstein, & Geman, 2011; Yuille 1999); visual motor control (Orban & Wolpert, 2011;
& Mottaghi, 2013), discussed more below. Wolpert & Landy, 2012); learning (Green, Pouget, &
Bavelier, 2010); and attention (Chikkerur, Serre, Tan,
4
& Poggio, 2010; for reviews, see Geisler, 2011; Kersten
A discriminative algorithm can implement a decision rule & Yuille, 2013; Vilares & Körding, 2011). Findings of
with no explicit use of probabilities. For example, with a large
number of samples, a rule to minimize empirical risk optimal behavior have raised the question of whether
(Schölkopf & Smola, 2002) becomes equivalent to minimizing neural populations within the brain explicitly represent
Bayes risk, as discussed in Kersten et al. (2004). and compute with probabilities, for example, using
kersten and yuille: inferential models of the visual cortical hierarchy 393
frequency tuning in V1 neurons, are consistent with a out slow spatial gradients (presumed due to illumina-
sparse-coding strategy adapted to the statistics of tion), and emphasizing edges (presumed due to surface
natural images (Olshausen, 1996; Hyvärinen, 2010). In changes). However, illumination effects are compli-
addition, neurons in primary visual cortex show cated: slow gradients can also be caused by shape, and
nonlinear divisive-normalization behavior in which simple filtering neither accounts for human perception
responses are inhibited by contrast variation outside of brightness (Kingdom, 2011; Knill & Kersten, 1991),
the classical receptive field. Divisive normalization nor provides accurate reflection estimation in com-
results in a reduction of statistical dependencies puter vision applied to natural images (Tappen,
(Schwartz & Simoncelli, 2001), providing an efficient Freeman, & Adelson, 2005).
representation potentially useful for discovering (addi- This problem is naturally cast in terms of Bayesian
tional) suspicious coincidences. Recently, Freeman, inference, where the generative knowledge is contained
Ziemba, Heeger, Simoncelli, and Movshon (2013) in the image formation model I = f(E, R, S) and spatial
developed a texture model based on high-order statisti- priors on illumination (E), reflectance (R), and shape
cal dependencies in natural images that could account (S)—spatial maps called “intrinsic images” (Barrow &
for selectivities in both macaque and human V2. Purely Tenenbaum, 1978). Conceptually, a Bayesian model
bottom-up, unsupervised feature learning typically would use a posterior proportional to the product of a
ignores task requirements (i.e., what to discount), and likelihood function p(I − f(R, S, E)), and priors that
eventually the behavioral end-goal of a visual pathway characterize the spatial regularities in the natural pat-
needs to be taken into account.5 However, some task terns of reflectance and shape, while discounting illu-
requirements are general, suggesting that certain kinds mination through integration (see Freeman, 1994).
of information can be discounted early on. While computing intrinsic images from natural images
can be done in special cases, it nevertheless remains a
Generic task constraints on early representations It is challenging problem (Barron & Malik, 2012; Grosse,
believed that early vision involves both contour- and Johnson, Adelson, & Freeman, 2009).
region-based linking (Grossberg & Mingolla, 1985; Perceptual evidence for human computation of an
Lamme, Sup, & Spekreijse, 1998; Lee, 2003; Roe, Chen, intrinsic image for reflectance comes from human
& Lu, 2009; Roe et al., 2012). For contour features, lightness judgments, which are more strongly corre-
conditional probabilities, fit with natural image statis- lated with reflectance than image intensity or contrast.
tics, predict aspects of human contour perception, such The classic Craik-O’Brien lightness illusion, shown in
as the Gestalt property of “good continuation”—nearby the upper-middle panel of figure 34.4A, illustrates this.
contour elements tend to have similar orientations Regions with identical physical intensities appear to
(Elder & Goldberg, 2002; Geisler & Perry, 2009). have different lightnesses. The functional interpreta-
Region-based grouping relies on the prior assumption tion is that the illusion is due to a mechanism designed
of piece-wise smoothness in low- and higher-order to produce an estimate of surface reflectance, based on
intensive attributes (i.e., texture; Shi & Malik, 2000). the assumptions that reflectance changes are often
The assumed function of edge- and region-based group- abrupt and illumination changes tend to be gradual
ing is to compute surface representations that are more (figure 34.4B).
reliably associated with object than image properties, Functional MRI evidence for processes involved in
providing a front end to a variety of object-based tasks, computing a lightness map in human V1 and V2 is
including recognition (Marr, 1982). And a first step shown in figure 34.4A (Boyaci, Fang, Murray, & Kersten,
would be to begin the process of discounting causes of 2007). Activity in localized regions of visual cortical
image patterns that are not needed. areas V1 and V2 (distant from the central edge)
The accurate inference of illumination level and responds to a perceived change in lightness in the
direction is low priority for both “what” and “how” tasks, absence of a physical change in intensity (see lower
which care primarily about objects and surfaces. This panels in figure 34.4A). While purely lateral computa-
suggests that at least some components of illumination tions have been invoked to explain this kind of “filling
variation would be discounted early in the visual system. in,” it has also been shown that human V1 response to
This is consistent with retinal lateral inhibition filtering lightness change is also sensitive to perceptual organiza-
tion of occluded surfaces, suggesting that top-down
5 feedback may be involved (Boyaci, Fang, Murray, &
Discounting can be achieved through unsupervised learning.
For example, Cadieu and Olshausen (2012) show unsupervised Kersten, 2010).
learning of invariances of form by factoring out contributions In addition to allowing for illumination variation,
from motion. object recognition has the additional requirement
B sk
li j
sj
si
lj
Ii
Figure 34.4 (A) The upper-middle panel shows a classic The lower graphs show that voxels in both V1 and V2 respond
illusion known as the Craik-O’Brien effect. Away from the to apparent changes in lightness almost as strongly as real
vertical border, the left and right rectangles have the same changes, as compared with a control. (Reprinted from Boyaci
luminance, as indicated by the red line, which shows how light et al., 2007, with permission from Elsevier.) (B) An undirected
intensity varies from left to right. The interesting perceptual graph (Markov random field) can be used to formulate prior
observation is that the left rectangle looks darker than the probabilities representing lateral, spatial statistical dependen-
right. In fact, there is little difference between the appearance cies for contours and surface properties such as reflectance
of a real intensity difference (upper left), and the illusory one. (cf. Kersten, 1991; Marroquin et al., 1987). (See color plate 32.)
that variations due to position and depth need to be highest level to learn high-level features that optimally
discounted. We discuss within-area computations sup- distinguish object classes. At the next level down, the
porting invariant recognition in the later section on principle is again applied to learn lower-level features
feedforward computations. that distinguish the previous features learned, and so
forth (Epshtein, Lifshitz, & Ullman, 2008). The task
Learning hierarchically organized area representations for rec- requirement of what to discount is built into the a
ognition One can use the end goal of object classifica- priori selection of the training classes to be distin-
tion as a constraint on learning feature hierarchies guished. Simulations have shown that once the fea-
through successive, top-down categorization of inter- tures have been learned, accurate object recognition
mediate-level features. Here the invariance require- and localization can be achieved with one forward pass
ments are built into the choice of what distinguishes followed by one backward pass through the hierarchy
the top-level training classes. The basic principle is to (Epshtein et al., 2008).
learn diagnostic features (such as “fragments” or “fea-
tures of intermediate complexity”) that maximize the Learning object compositions to manage image complex-
information for distinguishing object classes (Ullman, ity Compositionality refers to the human ability to
Vidal-Naquet, & Sali, 2002). Humans and nonhuman construct hierarchical representations, whereby fea-
primates seem to learn such features (Harel, Ullman, tures/parts are used and shared to describe a poten-
Epshtein, & Bentin, 2007; Hegdé, Bart, & Kersten, tially unlimited number of relational compositions
2008; Kromrey, Maestri, Hauffen, Bart, & Hegdé, 2010; (Geman, Potter, & Chi, 2002). It is argued that without
Lerner, Epshtein, Ullman, & Malach, 2008). To build a such a generative structure underlying scene and object
feature hierarchy, one applies this principle at the compositions, we could not account for the efficiency
kersten and yuille: inferential models of the visual cortical hierarchy 395
and versatility with which humans can acquire and gen- It has been argued that a hierarchy of multiple areas
eralize visual knowledge. There is also evidence is required to achieve functional invariance, given the
that humans exploit compositionality when learning biological properties of neurons and their connections
new patterns (Barenholtz & Tarr, 2011). One aspect of (Poggio, 2011). In this account, discounting is achieved
compositionality is the ability to represent spatial rela- incrementally through levels of the ventral stream via
tionships between parts, an idea with an early history the operation of AND-like (to detect feature conjunc-
(Biederman, 1987; Hummel & Biederman, 1992; Marr tions) and OR-like operations (to discount variations in
& Nishihara, 1978; Waltz, 1972). A second aspect, con- position and size) over levels (Zhu et al., 2010) via
sistent with current models of primate recognition, is simple- and complex-type cells, respectively (Riesenhu-
the idea of “reusable” features or “shared” parts, where ber & Poggio, 1999).
lower levels have only a few feature types (e.g., edges), During the first feedforward pass, information neces-
but these can be combined in many ways to make com- sarily gets left behind in the race to quickly and accu-
positions of parts with increasing specificity at higher rately draw from a relatively small set of high-priority,
levels. categorical hypotheses. But “no going back” requires
An underlying compositional structure to the visual strong a priori architectural assumptions regarding
world suggests that learning should exploit that assump- what constitutes high-priority end goals, as well as a
tion, and computer vision work has demonstrated unsu- strategic balancing of the trade-off between selectivity
pervised learning of levels of reusable parts from natural and invariance. Invariance is achieved at the cost of loss
image ensembles, which is then applied to multi-class of information—too much loss and categories become
recognition (Zhu, Chen, Lin, & Yuille, 2010; Zhu, Chen, indistinguishable; too little, and there are too many
Torralba, Freeman, & Yuille, 2011; see figure 34.5). object types.
Feedforward Computations Invariant object recog- Efficiency of a compositional hierarchy for recognition Com-
nition by the ventral stream requires discounting spatial positional arguments may help to answer the question
position and size (DiCarlo, Zoccolan, & Rust, 2012; of why a hierarchical visual architecture is desirable.
Fukushima, 1988; Riesenhuber & Poggio, 1999; Wallis, Yuille and Mottaghi (2013) conjecture that the key
Rolls, & Foldiak, 1993). The basic feedforward compu- problem of vision is complexity. The visual system
tations are assumed to be the detection of conjunctions needs to be organized in such a way that it can repre-
of features that belong together as part of an object, sent a very large number of objects and be able to
while at the same time discounting, through disjunction rapidly detect which ones are present in an image.
(which can be viewed as an approximation for “integrat- They demonstrate by mathematical analysis that this
ing out”), sources of variation, including position and can be achieved using compositional models, which
scale. are constructed in terms of hierarchical dictionaries of
parts (see figure 34.5). There are two key issues. First,
this visual architecture exploits part sharing between
different objects, which leads to great efficiency in rep-
resentation and speed of detection. The lower-level
Level 5
parts are small and are shared between many objects.
The high-level parts are larger (composed from lower-
level parts) and are shared less because they are more
Level 4 specific to objects. Second, objects are represented in a
distributed hierarchical manner where the positions,
and other properties, of the high-level parts are speci-
Level 3
fied coarsely, while the low-level parts are specified to
higher precision. This “executive summary principle,”
Level 2
combined with part-sharing, can lead to exponential
Level 1
gains in the number of objects that can be represented,
Figure 34.5 Examples of the mean shapes of visual concepts as well as the speed of recognition. For these types of
automatically learned for multiple objects with part sharing models (based on Zhu et al., 2011) recognition is
between objects. The specificity and the number of types of performed by propagating up hypotheses about which
features increases as one goes up the hierarchy, consistent in
general terms, with the progression of neural selectivities as low-level parts are present to obtain an unambiguous
one moves up the ventral stream. (Figure adapted from Zhu high-level interpretation. And, as discussed in the
et al., 2011.) next section, top-down processing can be used to
kersten and yuille: inferential models of the visual cortical hierarchy 397
for the missing vertices of the diamond (see Kersten, 2013). And suppression measured using fMRI activity
Mamassian, & Yuille, 2004). When the diamond is seen does not necessarily show the spatial specificity sug-
during an adaptation period (figure 34.6C), there was gested by the above adaptation study or by theory (de
an increase in the strength of adaptation to shape (e.g., Wit, Kubilius, Wagemans, & op de Beeck, 2012).
adapting to a skinny diamond results in seeing a stan- In the language of signal-detection theory, the sup-
dard comparison diamond as fatter); at the same time, pression of false and true positives through feedback
there was a decrease in the strength of adaptation to could both be computationally useful. Suppression of
the local orientation of comparison gratings. The con- false positives, enhancement of true positives in one
verse was found when the occlusion cues were inconsis- population of neurons, or both could serve to bind
tent with a diamond (figure 34.6D). The interpretation, object representations with parts and features at lower
consistent with other research, rests on the assumption levels, as in the above compositional model. At the same
that the sites of orientation and shape adaptation are time, increased activity in another neural population
in early and higher-level cortical areas, respectively. could signal false positives (i.e., inconsistent features
There is also evidence from human fMRI studies for that need to be resolved with other hypotheses; Friston,
context-dependent suppression of neural activity in 2005; Rao & Ballard, 1999). Ultra-high field fMRI with
earlier areas in some cases (Alink, Schwiedrzik, Kohler, submillimeter resolution has found stronger fMRI
Singer, & Muckli, 2010; Cardin, Friston, & Zeki, 2011; response in middle cortical layers of V1 during the
Fang, Kersten, & Murray, 2008; Murray, Kersten, Olshau- presentation of scrambled objects as compared with
sen, Schrater, & Woods, 2002; Rauss, Schwartz, & Pour- intact objects (Olman et al., 2012), similar to what one
tois, 2011), but not all (Mannion, Kersten, & Olman, might expect from prediction errors.
many seconds. Variations in perceptual learning and its computation with no current computer vision solution.
transfer may be understood in terms of whether the The visual system has to decide which features form the
learning task requires the “expertise” of a lower versus boundary of the object’s image—that is, a challenging
higher level of processing (Hochstein & Ahissar, 2002). segmentation and grouping problem that could require
In another study, Williams et al. (2008) found that the feedback to retinotopic areas. The locations of these
measured patterns of fMRI activity near foveal retino- features are needed to summarize the average diameter,
topic cortex could discriminate which object category or angular size. Then to estimate physical size from
the observers had been seeing with their peripheral angular size, the system needs to process the larger
vision. It has been known for some time that visual context in order to take the object’s depth into account.
imagery involving fine spatial discrimination, and even Further, size perception often involves comparisons
orientation-specific tactile tasks, may activate represen- with other objects, raising the question of where to
tations in early visual areas (Kosslyn et al., 1993; Kosslyn make those. The complexity of the analysis suggests an
& Thompson, 2003; Lucan, Foxe, Gomez-Ramirez, interplay between high-level representations and early
Sathian, & Molholm, 2010; Zangaladze, Epstein, retinotopic areas, particularly V1 as a result of its high
Grafton, & Sathian, 1999). spatial precision. Studies by Murray et al. (2006) and
Consider the everyday task of inferring an object’s Fang et al. (2008) used a classic depth illusion to show
physical size from its image. This is a nontrivial that the pattern of spatial activity in V1 activity is indeed
kersten and yuille: inferential models of the visual cortical hierarchy 399
modulated by 3D depth context (figure 34.7). When an REFERENCES
object (a ring) appeared bigger, its “neural image” on
Alink, A., Schwiedrzik, C. M., Kohler, A., Singer, W., &
V1 was bigger (i.e., activation shifts to a more eccentric Muckli, L. (2010). Stimulus predictability reduces
representation of the visual field). responses in primary visual cortex. J Neurosci, 30(8),
This effect was significantly stronger when observers 2960–2966.
attended to the object, consistent with feedback from Arnold, D. H., Birt, A., & Wallis, T. S. A. (2008). Perceived
higher-level areas that process depth in the larger size and spatial coding. J Neurosci, 28(23), 5954–5958.
Bar, M. (2003). A cortical mechanism for triggering top-down
context of the scene. Psychophysical data is also consis-
facilitation in visual object recognition. J Cogn Neurosci,
tent with a top-down influence of depth on orientation- 15(4), 600–609.
selective, and putatively early, cortical regions (Arnold, Barenholtz, E., & Tarr, M. J. (2011). Visual learning of
Birt, & Wallis, 2008). statistical relations among nonadjacent features: Evidence
for structural encoding. Vis Cogn, 19(4), 469–482.
Barlow, H. (1981). The Ferrier lecture, 1980. Critical limit-
The longer you look, the more you see ing factors in the design of the eye and visual cortex. Proc
R Soc Lond B Biol Sci, 212(1186), 1–34.
Not many decades ago, “perception” seemed to be not Barlow, H. (1990). Conditions for versatile learning, Helm-
much more than a screen, admittedly with some puz- holtz’s unconscious inference, and the task of perception.
zling distortions, viewed by a high-level executive agent. Vis Res, 30(11), 1561–1571.
Then retinal and cortical studies showed that neurons Barron, J. T., & Malik, J. (2012). Shape, albedo, and illumi-
nation from a single image of an unknown object. Proc. IEEE
were doing much more than transmitting image infor- Conference on Computer Vision (pp. 1–8).
mation: they were emphasizing certain kinds of infor- Barrow, H., & Tenenbaum, J. (1978). Recovering intrinsic
mation, such as edges, at the expense of others (smooth scene characteristics from images. In A. Hanson &
gradients). This led to the idea of the retina and early E. Riseman (Eds.), Computer vision systems (pp. 3–16). New
visual cortical areas as spatiotemporal filter banks. But York, NY: Academic Press.
Beauchamp, M. S., Lee, K. E., Haxby, J. V., & Martin, A.
still, the emphasis was on early perceptual processing as (2002). Parallel visual motion processing streams for
a set of filtering stages, effectively passing decisions manipulable objects and human movements. Neuron, 34(1),
forward from one stage to the next (Lennie, 1998). 149–159.
Computer vision has provided the perspective that Beck, J. M., Latham, P. E., & Pouget, A. (2011). Marginaliza-
in order to produce useful behavioral outcomes, the tion in neural circuits with divisive normalization. J Neurosci,
31(43), 15310–15319.
human visual system is solving a decoding problem
Biederman, I. (1987). Recognition-by-components: A theory
whose understanding requires concepts and a level of of human image understanding. Psychol Rev, 94(2), 115–147.
analysis beyond traditional neural network filtering. Blanchard, G., & Geman, D. (2005). Hierarchical testing
The past decade has seen substantial progress in both designs for pattern recognition. Ann Stat, 33(3),
the computational and neural understanding of how 1155–1202.
Boyaci, H., Fang, F., Murray, S. O., & Kersten, D. (2007).
vision could be solving the problems of object percep-
Responses to lightness variations in early human visual
tion. We have discussed potential limitations on the cortex. Curr Biol, 17(11), 989–993.
robustness and versatility of vision with strictly feed- Boyaci, H., Fang, F., Murray, S. O., & Kersten, D. (2010).
forward processing and have reviewed arguments and Perceptual grouping-dependent lightness processing in
results suggesting that both automatic and executive human early visual cortex. J Vis, 10(9), 1–12.
processes access built-in image knowledge at several Burgi, P. Y., Yuille, A., & Grzywacz, N. M. (2000). Probabi-
listic motion estimation based on temporal coherence.
levels of abstraction. We conjecture that the brain’s Neural Comput, 12(8), 1839–1867.
ability to solve the problems of local uncertainty and Cadieu, C. F., & Olshausen, B. A. (2012). Learning interme-
task versatility rests on deep generative knowledge of diate-level representations of form and motion from natural
the structure of images. A major challenge for the movies. Neural Comput, 24(4), 827–866.
future is to better understand the way the brain repre- Callaway, E. (1998). Local circuits in primary visual cortex
of the macaque monkey. Annu Rev Neurosci, 21, 47–74.
sents and controls the top-down utilization of this Cardin, V., Friston, K. J., & Zeki, S. (2011). Top-down modu-
knowledge (cf. Blanchard & Geman, 2005; Ullman, lations in the visual form pathway revealed with dynamic
1984), eventually explaining how the brain enables us causal modeling. Cereb Cortex, 21(3), 550–562.
to see so much in just one picture of a fox. Chang, L., Jin, Y., Zhang, W., Borenstein, E., & Geman, S.
(2011). Context, computation, and optimal ROC perfor-
ACKNOWLEDGMENTS DK and AL were supported by the WCU mance in hierarchical models. Int J Comput Vis, 93(2),
(World Class University) program funded by the Ministry of 117–140.
Education, Science, and Technology through the National Chikkerur, S., Serre, T., Tan, C., & Poggio, T. (2010). What
Research Foundation of Korea (R31-10008) and by ONR and where: A Bayesian inference theory of attention. Vis
N000141210883. Res, 50(22), 2233–2247.
kersten and yuille: inferential models of the visual cortical hierarchy 401
Kersten, D. (1991). Transparency and the cooperative comp- Lennie, P. (1998). Single units and visual cortical organiza-
utation of scene attributes. In M. S. Landy (Ed.), Computa- tion. Perception, 27, 889–936.
tional models of visual processing (pp. 209–228). Cambridge, Lerner, Y., Epshtein, B., Ullman, S., & Malach, R. (2008).
MA: MIT Press. Class information predicts activation by object fragments in
Kersten, D., Mamassian, P., & Yuille, A. (2004). Object human object areas. J Cogn Neurosci, 20(7), 1189–1206.
perception as Bayesian inference. Annu Rev Psychol, 55, Li, Z. (1997). Primary cortical dynamics for visual grouping.
271–304. In K.-Y. M. Wong & D.-Y. Yeung (Eds.), Theoretical aspects of
Kersten, D. J., & Yuille, A. L. (2013). Vision: Bayesian infer- neural computation. New York, NY: Springer.
ence and beyond. In J. Werner & L. M. Chalupa (Eds.), The Lochmann, T., & Deneve, S. (2011). Neural processing as
new visual neurosciences (pp. 1–16). Cambridge, MA: MIT causal inference. Curr Opin Neurobiol, 21(5), 774–781.
Press. Lucan, J. N., Foxe, J. J., Gomez-Ramirez, M., Sathian, K., &
Kingdom, F. A. A. (2011). Lightness, brightness and transpar- Molholm, S. (2010). Tactile shape discrimination recruits
ency: A quarter century of new ideas, captivating demon- human lateral occipital complex during early perceptual
strations and unrelenting controversy. Vis Res, 51(7), processing. Hum Brain Mapp, 31(11), 1813–1821.
652–673. Lund, J., Angelucci, A., & Bressloff, P. C. (2003). Anatomi-
Knill, D. C. (2003). Mixture models and the probabilistic cal substrates for functional columns in macaque monkey
structure of depth cues. Vis Res, 43(7), 831–854. primary visual cortex. Cereb Cortex, 13(1), 15–24.
Knill, D. C., & Kersten, D. (1991). Apparent surface curva- Ma, W. J. (2010). Signal detection theory, uncertainty, and
ture affects lightness perception. Nature, 351(6323), Poisson-like population codes. Vis Res, 50(22), 2308–2319.
228–230. Ma, W. J. (2012). Organizing probabilistic models of percep-
Knill, D. C., & Pouget, A. (2004). The Bayesian brain: The tion. Trends Cogn Sci, 16(10), 511–518.
role of uncertainty in neural coding and computation. Ma, W. J., Beck, J. M., Latham, P. E., & Pouget, A. (2006).
Trends Neurosci, 27(12), 712–719. Bayesian inference with probabilistic population codes. Nat
Koch, C., Marroquin, J., & Yuille, A. (1986). Analog “neu- Neurosci, 9(11), 1432–1438.
ronal” networks in early vision. Proc Natl Acad Sci USA, Ma, W. J., Beck, J. M., & Pouget, A. (2008). Spiking networks
83(12), 4263–4267. for Bayesian inference and choice. Curr Opin Neurobiol,
Körding, K. P., Beierholm, U., Ma, W. J., Quartz, S., Tenen- 18(2), 217–222.
baum, J. B., & Shams, L. (2007). Casual inference in mul- Maloney, L. T., & Zhang, H. (2010). Decision-theoretic
tisensory perceptions. PloS ONE, 2(9), e943. models of visual perception and action. Vis Res, 50(23),
Kosslyn, S. M., Alpert, N. M., Thompson, W. L., Maljkovic, 2362–2374.
V., Weise, S. B., Chabris, C. F., … Buonanno, F. S. (1993). Mannion, D. J., Kersten, D. J., & Olman, C. A. (2013). Con-
Visual mental imagery activates topographically organized sequences of polar form coherence for fMRI responses in
visual cortex: PET investigations. J Cogn Neurosci, 5(3), human visual cortex. NeuroImage, 78(C), 152–158.
263–287. Markov, N. T., Vezoli, J., Chameau, P., Falchier, A., Quilo-
Kosslyn, S. M., & Thompson, W. L. (2003). When is early dran, R., Huissoud, C., … Kennedy, H. (2013). The
visual cortex activated during visual mental imagery? Psychol anatomy of hierarchy: Feedforward and feedback pathways
Bull, 129(5), 723–746. in macaque visual cortex. J Comp Neurol, 522(1), 225–259.
Kourtzi, Z., & Connor, C. E. (2011). Neural representations Marr, D. (1982). Vision: A computational investigation into the
for object perception: Structure, category, and adaptive human representation and processing of visual information. New
coding. Annu Rev Neurosci, 34(1), 45–67. York, NY: Henry Holt.
Kourtzi, Z., Tolias, A. S., Altmann, C. F., Augath, M., & Marr, D., & Nishihara, H. K. (1978). Representation and
Logothetis, N. K. (2003). Integration of local features recognition of the spatial organization of three-dimensional
into global shapes: Monkey and human fMRI studies. shapes. Proc R Soc Lond B Biol Sci, 200(1140), 269–294.
Neuron, 37(2), 333–346. Marroquin, J., Mitter, S., & Poggio, T. (1987). Probabilistic
Kromrey, S., Maestri, M., Hauffen, K., Bart, E., & solution of ill-posed problems in computational vision. J Am
Hegdé, J. (2010). Fragment-based learning of visual object Stat Assoc, 82(397), 76–89.
categories in non-human primates. PLoS ONE, 5(11), McManus, J. N. J., Li, W., & Gilbert, C. D. (2011). Adaptive
e15444. shape processing in primary visual cortex. Proc Natl Acad Sci
Lamme, V. A., Sup, H., & Spekreijse, H. (1998). Feedforward, USA, 108(24), 9739–9746.
horizontal, and feedback processing cortex. Curr Opin Neu- Mountcastle, V. B. (1997). The columnar organization of
robiol, 8, 529–535. the neocortex. Brain, 120, 701–722.
Lauritzen, S., & Spiegelhalter, D. (1988). Local computa- Mumford, D. (1992). On the computational architecture of
tions with probabilities on graphical structures and their the neocortex. Biol Cybern, 66(3), 241–251.
application to expert systems. J Roy Stat Soc B Met, 50(2), Murray, S. O., Boyaci, H., & Kersten, D. (2006). The rep-
157–224. resentation of perceived angular size in human primary
Lee, T., Mumford, D., Romero, R., & Lamme, V. A. (1998). visual cortex. Nat Neuroscience, 9(3), 429–434.
The role of the primary visual cortex in higher level vision. Murray, S. O., Kersten, D., Olshausen, B. A., Schrater, P.,
Vis Res, 38(15–16), 2429–2454. & Woods, D. L. (2002). Shape perception reduces activity
Lee, T. S. (2003). Computations in the early visual cortex. in human primary visual cortex. Proc Natl Acad Sci USA,
J Physiol (Paris), 97(2–3), 121–139. 99(23), 15164–15169.
Lee, T. S., & Mumford, D. (2003). Hierarchical Bayesian Nassi, J. J., & Callaway, E. M. (2009). Parallel processing
inference in the visual cortex. J Opt Soc Am A Opt Image Sci strategies of the primate visual system. Nat Rev Neurosci,
Vis, 20(7), 1434–1448. 10(5), 360–372.
kersten and yuille: inferential models of the visual cortical hierarchy 403
Yuille, A. L., & Mottaghi, R. (2013). Complexity of repre- Zhang, X., Zhaoping, L., Zhou, T., & Fang, F. (2012). Neural
sentation and inference in compositional models with part activities in V1 create a bottom-up saliency map. Neuron,
sharing. arXiv:1301.3560 73(1), 183–192.
Zangaladze, A., Epstein, C. M., Grafton, S. T., & Sathian, Zhu, L., Chen, Y., Lin, C., & Yuille, A. (2010). Max margin
K. (1999). Involvement of visual cortex in tactile discrimina- learning of hierarchical configural deformable templates
tion of orientation. Nature, 401(6753), 587–590. (HCDTs) for efficient object parsing and pose estimation.
Zeiler, M., Taylor, G., & Fergus, R. (2011). Adaptive decon- Int J Comput Vis, 93(1), 1–21.
volutional networks for mid and high level feature learning. Zhu, L., Chen, Y., Torralba, A., Freeman, W., & Yuille, A.
Proc. IEEE Conference on Computer Vision (pp. 2018–2025). (2011). Part and appearance sharing: Recursive composi-
Zemel, R. S., & Pouget, A. (1998). Probabilistic interpreta- tional models for multi-view multi-object detection. Proc.
tion of population codes. Neural Comput, 10(2), 403–430. IEEE Conference on Computer Vision (pp. 1919–1926).
Zhang, L., Tong, M. H., Marks, T. K., Shan, H., & Cottrell,
G. W. (2008). SUN: A Bayesian framework for saliency using
natural statistics. J Vis, 8(7), 1–20.
ABSTRACT Visual recognition is amazingly rapid and requires Thus, the goal of this chapter is to unveil the “black
a series of computations extending from primary visual cortex box” of VTC and its role in visual perception adapting
to ventral temporal cortex (VTC). Here, we examine the Marr’s framework to examine the inter-relationship
functional architecture of human VTC and its role in recogni-
tion. We detail the computational goals, the information com- among computations, representations, and neural
puted to reach those goals (representations), and the physical implementation (figure 35.1). Our examination of
layout of that information on the cortical sheet (implementa- these three factors builds upon classic cognitive neuro-
tion). Then, we discuss how implementational features of science research, which provided an integrative founda-
neural architecture may impact representations and computa- tion for how anatomical and functional properties of
tions in VTC. Specifically, recent discoveries detail anatomical
the visual system (Van Essen, Anderson, & Felleman,
and topological relationships among a series of fine-scale rep-
resentations nested within large-scale representations in VTC, 1992; Zeki & Shipp, 1988)—namely, its functional
which generate predictable divergences and convergences architecture (Hubel & Wiesel, 1962)—contribute to
across cortical scales. We suggest that this implementation information processing (figure 35.1A). We emphasize
generates a spatial hierarchy of nested functional representa- recent discoveries of implementational features of VTC,
tions that mirrors the hierarchical category information struc-
which reveal superimposition of multiple neural repre-
ture of VTC, where more abstract information is represented
at larger spatial scales. Together these implementational, rep- sentations on the cortical sheet with predictable levels
resentational, and computational features enable a rapid and of anatomical convergence and divergence. This func-
flexible recognition system with access to different levels of tional architecture matriculates at multiple spatial
information abstraction extending from exemplars, to basic scales, generating a spatial hierarchy of nested neural
categories, to broad superordinate categorical distinctions. representations (figure 35.1C). We propose that this
spatial hierarchy generates an information hierarchy
Human recognition is amazingly rapid, enabling us to within VTC (figure 35.1B), which, in turn, enables a fast
determine the gist of a visual scene in just one-tenth of and flexible system supporting recognition at multiple
a second (Thorpe, Fize, & Marlot, 1996). Despite this levels of abstraction ranging from exemplars to broad
remarkable efficiency, visual recognition is nontrivial, categories.
requiring a series of computations arranged in a ventral
stream processing hierarchy (Ungerleider & Haxby,
1994), ascending from primary visual cortex (V1) to Computational theory: What are the computational
high-level regions in ventral temporal cortex (VTC; also goals of ventral temporal cortex?
referred to as inferotemporal cortex, or IT). Human
VTC consists of the posterior half of the ventral aspect Theorists investigate the computational requirements
of the temporal lobe. Lesions to VTC can produce spe- necessary for achieving an efficient and rapid visual
cific deficits, such as the inability to recognize objects, recognition system. This computational research has
faces, or words, while other visual faculties are pre- yielded four important insights:
served, indicating its critical role in recognition. 1. Shape is the critical visual attribute for object
To understand how visual recognition is achieved, it recognition.
is necessary to study the computational goals of the 2. A visual recognition system needs to be robust to
recognition system, the representations and informa- changes in object appearance, such as retinal size, posi-
tion computed to reach those goals, and its physical tion in the visual field, illumination, contrast, and view
layout and implementation in cortex (Marr, 1982). (referred to as tolerance to image transformations)
without losing the ability to discriminate between Representations: How does ventral temporal cortex
similar exemplars with comparable parts and configura- represent information to support visual
tion (such as two different faces). recognition?
3. A visual recognition system should support recog-
nition and perception at several levels of abstraction; Ventral Temporal Cortex Is Tuned to Shape Irre-
for instance, differentiating among exemplars (e.g., spective of Visual Cue A general property of human
Joey vs. Jack), basic categories (e.g., face vs. car), as well VTC is that it responds more strongly to a variety of
as broad superordinate categories (e.g., animate vs. shapes and objects compared to textures, noise, or
inanimate). highly scrambled objects (Malach et al., 1995). Impor-
4. There are classes of stimuli that may require tantly, VTC responses maintain their selectivity to
specialized computations in addition to the domain- shape across a large spectrum of visual cues including
general computations mentioned in (1)–(3). For luminance, color, motion, texture, stereo, and illusory
example, face perception may also require domain- contours (Davindenko, Remus, & Grill-Spector, 2012;
specific computations to determine the gender, age, Grill-Spector, Kushnir, Edelman, Itzchak, & Malach,
expression, or gaze of a person. 1998; Kourtzi & Kanwisher, 2001; Mendola, Dale, Fischl,
Liu, & Tootell, 1999; Vinberg & Grill-Spector, 2008).
Understanding the computational goals of VTC as a Thus, a domain-general property of VTC is that it dem-
recognition system has important implications for both onstrates perceptual constancy to shape across visual
information processing and neural implementation. In cues.
the sections below, we examine the properties of VTC
representations and their relation to the computational Ventral Temporal Cortex Representations Are
goals of a recognition system: Are VTC representations Sensitive to Changes in Exemplar Identity and
tuned to shape? Are they tolerant to changes in appear- Exhibit Some Tolerance to Changes in Exemplar
ance? How do VTC representations and their cortical Appearance Classic theories of recognition suggest
implementation support different levels of information that recognizing exemplars across changes in appear-
abstraction? Empirical research striving to answer these ance requires object-specific, invariant representations
questions has led to the development of computational (Biederman, 1987). However, modern theories have
models that provide important insights into the compu- shown that complete tolerance is unnecessary as long
tations performed by the ventral stream hierarchy as representations for different exemplars are “untan-
(Epshtein, Lifshitz, & Ullman, 2008; Riesenhuber & gled” or linearly separable from each other (DiCarlo &
Poggio, 1999; Serre, Oliva, & Poggio, 2007). Cox, 2007; DiCarlo, Zoccolan, & Rust, 2012). Thus,
Figure 35.2 Focal categorical responses: selective and faces/(FFA-2) fusiform face area-2 averaged across seven sub-
shape-tuned. (A) Example inflated right cortical surface jects show reproducible preference for faces across tasks
zoomed on VTC. Face (black) and body (white) representa- and paradigms. (Adapted from Weiner & Grill-Spector, 2010.)
tions are arranged in focal clusters with predictable topologi- (C) Responses (±SEM) of mFus-faces/FFA-2 and OTS-limbs/
cal characteristics relative to one another and macroanatomical fusiform body area (FBA) averaged across 11 subjects show
landmarks. CoS: collateral sulcus; FG: fusiform gyrus; OTS: differential shape tuning for a face-hand morph continuum.
occipitotemporal sulcus; MFS: mid-fusiform sulcus (white); Selectivity is maintained across format changes. (Adapted
ptCoS: posterior transverse collateral sulcus. (B) Responses from Davidenko et al., 2012.)
(plus or minus standard error of the mean, SEM) of mFus-
Figure 35.3 VTC responses correlate with successful identi- guitar images. Asterisk indicates that neural responses during
fication and detection. Responses (±SEM) averaged across identification (e.g., this is Harrison Ford) are significantly
five subjects from (A) a 5 mm disk on mFus-faces/FFA-2 and larger (P < 0.01) than during image detection. (Adapted from
(B) a 5 mm disk on an adjacent region on the medial FG Grill-Spector et al., 2004.)
during recognition of briefly presented and masked face or
40 BATISTA 477
41 LEMON 491
ABSTRACT The neural connections of the basal ganglia and basal ganglia processing is represented by the internal
cerebellum provide important insights into their function. segment of the globus pallidus (GPi), the pars reticulata
The outputs of the basal ganglia and cerebellum target motor, of the substantia nigra (SNpr), and the ventral palli-
premotor, prefrontal, posterior parietal, and inferotemporal
areas of the cerebral cortex. In addition, the basal ganglia and dum. The comparable structures for cerebellar process-
cerebellum are interconnected. These basal ganglia and cer- ing are the deep cerebellar nuclei: dentate, interpositus,
ebellar circuits form the anatomical substrates for influencing and fastigial. Neurons in the output layers of both cir-
not only the control of movement, but also cognitive behav- cuits send their axons to the thalamus and, by this
iors like planning, working memory, reward-based learning, route, project back upon the cortex. Thus, a major
sequential behavior, visuospatial perception, and attention.
structural feature of basal ganglia and cerebellar cir-
Similarly, abnormal activity in specific basal ganglia and
cerebellar circuits may contribute to a variety of neuropsychi- cuits is that they form loops with the cerebral cortex
atric disorders, including autism, addiction, attention-deficit/ (e.g., Allen & Tsukahara, 1974; Brooks & Thach, 1981;
hyperactivity disorder, obsessive-compulsive disorder, and Kemp & Powell, 1971). These loops were thought to
Tourette syndrome. Thus, the anatomical substrate exists for function largely in the domain of motor control by fun-
the basal ganglia and cerebellum to be involved in a broad
neling information from diverse cortical areas to influ-
range of motor and nonmotor functions.
ence motor output at the level of the primary motor
What are the functions of the basal ganglia and cerebel- cortex (M1). This view has been supported by the
lum? Numerous reports describe the motor deficits obvious motor symptoms that can result from basal
associated with damage to these subcortical structures. ganglia and cerebellar dysfunction (for reviews, see
As a consequence, concepts about basal ganglia and Bhatia & Marsden, 1994; Brooks & Thach, 1981; DeLong
cerebellar function have focused primarily on their con- & Georgopoulos, 1981).
tributions to the generation and control of movement. Over the past 30 years, an accumulation of informa-
Examination of the macro-organization of basal ganglia tion about basal ganglia and cerebellar anatomy has led
and cerebellar connections with the cerebral cortex has a number of investigators to challenge this view (e.g.,
led to a different perspective. In this chapter, we focus Alexander, DeLong, & Strick, 1986; Goldman-Rakic &
on two critical questions. First, which cortical areas are Selemon, 1990; Schell & Strick, 1984). It is now clear
the target of the outputs from the basal ganglia and the that basal ganglia and cerebellar efferents terminate in
cerebellum? Second, do the basal ganglia and cerebel- different subdivisions of the ventrolateral thalamus (for
lum interact at sites other than the cerebral cortex? The a review, see Percheron, François, Talbi, Yelnik, &
answers to these questions lead to some novel and impor- Fénelon, 1996) which, in turn, project to a myriad of
tant insights about basal ganglia and cerebellar function. cortical areas. Thus, the outputs from the basal ganglia
Classically, the macro-organization of basal ganglia and cerebellum influence more widespread regions of
and cerebellar circuitry is described using a relatively the cerebral cortex than previously recognized.
simple hierarchical model. The “input layer” of basal Based on these and other anatomical results, Alexan-
ganglia processing is represented by the striatum der et al. (1986) proposed that the basal ganglia par-
(caudate, putamen, and ventral striatum). The analo- ticipate in at least five separate loops with the cerebral
gous level in cerebellar circuits is represented by spe- cortex. These loops were based in part on their cortical
cific pontine nuclei that send mossy fiber inputs to targets of the output layer of processing and were des-
cerebellar cortex. A major source of afferents to the ignated the skeletomotor, oculomotor, dorsolateral pre-
input layers of both circuits originates from widespread frontal, lateral orbitofrontal, and anterior cingulate
regions of the cerebral cortex, including motor, sensory, circuits. According to this scheme, the output of the
posterior parietal, prefrontal, cingulate, orbitofrontal, basal ganglia has the potential to influence not only the
and temporal cortical areas. The “output layer” of control of movement, but also higher-order cognitive
dum, bostan, and strick: basal ganglia and cerebellar circuits 419
and limbic functions subserved by prefrontal, orbito- separate animals, virus was injected into M1 regions
frontal, and anterior cingulate cortex. where face, arm, or leg movements were evoked by
Similarly, Leiner, Leiner, and Dow (1986, 1991, 1993) intracortical stimulation. The survival time was set to
suggested that cerebellar output is directed to prefron- allow retrograde transport of the virus to “first-order”
tal as well as to motor areas of the cerebral cortex. They neurons in the thalamus and then retrograde transneu-
noted that in the course of hominid evolution, the ronal transport of virus from these first-order neurons
lateral output nucleus of the cerebellum—the dentate— to “second-order” neurons that are the origin of basal
undergoes a marked expansion that parallels the expan- ganglia– and cerebello-thalamocortical inputs to M1
sion of cerebral cortex in the frontal lobe. They argued (Kelly & Strick, 2000; Strick & Card, 1992).
that the increase in the size of the dentate is accompa- These experiments produced three major results.
nied by an increase in the extent of the cortical areas First, M1 is richly innervated by the output nuclei of the
in the frontal lobe that are influenced by dentate basal ganglia and cerebellum. The densest projections
output. As a consequence, Leiner et al. (1986, 1991, originate from GPi (figures 36.1 and 36.2) and the
1993) proposed that cerebellar function in humans has dentate (figures 36.3 and 36.4). Less dense projections
expanded to include involvement in certain language originate from portions of SNpr and interpositus.
and cognitive tasks. Second, both GPi and the dentate are somatotopically
The use of neurotropic viruses (herpes simplex virus organized with separate face, arm, and leg areas that
type 1 [HSV1] and rabies virus) as transneuronal tracers project via the thalamus to the face, arm, and leg areas
has been essential for testing these proposals (for of M1. Third, and perhaps most surprising, the projec-
review, see Kelly & Strick, 2000, 2003; Strick & Card, tions to M1 originate from only 15% of the volume of
1992; Strick, Dum, & Fiez, 2009). This tracing method GPi and about 30% of the volume of the dentate. Fur-
can effectively label a chain of up to three synaptically thermore, the output to M1 originates from restricted
linked neurons in a single experiment (Bostan, Dum, portions of each subcortical nucleus. These results
& Strick, 2010; Kelly & Strick, 2003, 2004). In this imply that the majority of the output from the basal
chapter, we review some of the new observations that ganglia and cerebellum is directed to other cortical
result from using viruses to map the basal ganglia and areas.
cerebellar projections to the cerebral cortex. These
observations have led to important insights about the Premotor areas
cortical targets of these circuits and the functional
domains they influence. Basal ganglia and cerebellar projections to the arm
representations of premotor areas in the frontal lobe
Primary motor cortex were examined following injections of virus into the
ventral premotor area (PMv) or the supplementary
Retrograde transneuronal transport of HSV1 was used motor area (SMA; Akkal, Dum, & Strick, 2007; Hoover
to examine the organization of basal ganglia and cere- & Strick, 1993). These injections consistently labeled
bellar outputs to M1 (Hoover & Strick, 1993, 1999). In neurons in the middle of GPi rostrocaudally. Within
GPe
GPi
D
o i
M
A 15.0 A 14.2 A 13.7 A 14.0 1 mm A 14.2
Figure 36.1 Origin of pallidal projections to M1, PMv, SMA, portion of the internal segment of globus pallidus; I, inner
area 46, and area 9. Labeled neurons (dots) from several portion of the internal segment of globus pallidus; M1 arm,
adjacent sections are displayed on coronal sections through arm area of the primary motor cortex; PMv arm, arm area of
the GPi of animals that received virus injections into different the ventral premotor area; SMA arm, arm area of the supple-
cortical areas. The anterior-posterior location of each section mentary motor area; area 46 and 9 m, cytoarchitectonic
is indicated. Abbreviations: GPe, external segment of globus regions in the prefrontal cortex. (Adapted from Middleton &
pallidus; GPi, internal segment of the globus pallidus; o, outer Strick, 2000, with permission from Elsevier.)
D GPi (o)
Dorsal
4 9L M1
face
46d
Distance (mm)
“Motor”
3 Pre-
SMA M1
SMA arm arm
2
“Motor” PMv
arm
M1 arm
1 “Non-
PMv arm M1
Motor” leg SMA 7b
arm FEF
Ventral
C
0 1 2 3 4 5 6
46d Pre-
“Non- SMA
D
4 Motor” 9L
GPi (i)
9L
Pre-
3 SMA
Distance (mm)
46d
“Motor”
Dorsal
2 D ?
SMA arm
1 C
M1 arm
“Non- 1 mm
Motor” PMv arm
C Figure 36.4 Summary map of dentate output channels. The
0 1 2 3 4 5 dentate is displayed as an unfolded map (for details of unfold-
Distance (mm) ing, see Dum & Strick, 2003). The cortical target of each
output channel is placed at the site of its peak labeling follow-
Figure 36.2 Summary map of basal ganglia output chan- ing retrograde transneuronal transport of virus from that cor-
nels. The outer and inner segments of the GPi are shown as tical area. The dentate can be divided into “motor” and
separate unfolded maps (for details of unfolding, see Akkal “nonmotor” domains based on the grouping of output chan-
et al., 2007). In this planar view, the cortical target of each nels that target functionally similar cortical areas. Abbrevia-
output channel is placed at the site of its densest labeling tions as in figures 36.2 and 36.3. (Adapted from Dum & Strick,
following retrograde transneuronal transport of virus from 2003, and Akkal et al., 2007, with permission.)
that cortical area. The GPi can be divided into “motor” and
“nonmotor” domains based on the grouping of output chan-
nels that target functionally similar cortical areas. Abbrevia-
tions: D, dorsal; C, caudal; see also figure 36.1. (Adapted from
Akkal et al., 2007, with permission.)
D
IP
M
1 mm
DN
Figure 36.3 Origin of cerebellar projections to M1, PMv, according to figure 36.2. Abbreviations: DN, dentate nucleus;
area 46, and area 9. Representative coronal sections through IP, interpositus nucleus; M, medial. (Adapted from Middleton
the dentate and interpositus nuclei of animals that received & Strick, 1998, with permission from Elsevier.)
virus injections into different cortical areas. Conventions are
dum, bostan, and strick: basal ganglia and cerebellar circuits 421
this region, neurons labeled after injections into the channels characterizes skeletomotor output from the
SMA, M1, or PMv formed separate clusters in a dorsal dentate.
to ventral arrangement (figures 36.1 and 36.2). These
observations indicate that pallidal output is not con- Frontal eye field
fined to M1, but projects via the thalamus to multiple
premotor areas in the frontal lobe (see also Inase & The results of studies with conventional tracers sug-
Tanji, 1995; Jinnai, Nambu, Tanibuch, & Yoshida, 1993; gested that the frontal eye field (FEF) receives input via
Saga et al., 2011; Sakai, Inase, & Tanji, 1999). Further- the thalamus from three subcortical nuclei: SNpr, the
more, the arm representation of each motor area superior colliculus (SC), and the cerebellar nuclei. To
receives input from a topographically distinct set of test this proposal, virus was injected into FEF regions,
GPi neurons. We have proposed that this arrangement where eye movements were evoked by intracortical
creates distinct “output channels” in the sensorimotor stimulation (Lynch, Hoover, & Strick, 1994). Virus-
portion of GPi (Akkal et al., 2007; Hoover & Strick, infected neurons were found in SNpr, the optic and
1993). intermediate gray layers of the SC, and the dentate
The output neurons of the dentate have a similar nucleus. Within the dentate, labeled neurons were con-
topographic organization. Injections of virus into the fined to its posterior pole, where some neurons exhibit
arm representations of M1, PMv, and SMA labeled activity correlated with saccadic eye movements (van
clusters of neurons in the middle of the dentate ros- Kan, Houk, & Gibson, 1993). Within the basal ganglia,
trocaudally (figures 36.3 and 36.4; Akkal et al., 2007; labeled neurons were located in posterior and lateral
Middleton & Strick, 1998; see also Hashimoto et al., portions of SNpr (figure 36.6, FEF) where neurons
2010; Lu, Miyachi, Ito, Nambu, & Takada, 2007). The display changes in activity related to saccadic eye move-
“hot spot” of each cluster appeared to be centered in ments (Hikosaka & Wurtz, 1983a, 1983b). The regions
a slightly different region of the dentate. The hot of the basal ganglia and cerebellum that were labeled
spots for the different motor areas are shown on an after injections of virus into the FEF were strikingly dif-
unfolded map of the dentate (figure 36.4). This ferent from those labeled after injections into the skel-
diagram emphasizes two important observations. First, etomotor areas of the frontal lobe. Thus, the oculomotor
the dentate, like GPi, contains distinct output chan- output channels in the basal ganglia and cerebellum
nels that innervate the arm representations of the dif- are distinct from those concerned with skeletomotor
ferent cortical motor areas. Second, the output function.
channels to these different arm representations are
clustered together. This observation suggests that Prefrontal cortex
these output channels are in register within the
nucleus and raises the possibility that they form a Areas 9, 12, and 46 of the prefrontal cortex appear
single map for each body part. to be involved in cognitive operations such as the
Results from single-neuron recording experiments guidance of behavior based on transiently stored infor-
in awake trained monkeys provide physiological mation rather than immediate external cues (“working
support for the existence of distinct output channels in memory”; for review, see Fuster, 1997; Goldman-Rakic,
GPi and dentate (Mushiake & Strick, 1993, 1995; 1996; Passingham, 1993). These regions of prefrontal
Strick, Dum, & Picard, 1995). These studies suggest cortex are known to project to the input stage of basal
that individual output channels are involved in differ- ganglia and cerebellar processing (e.g., Glickstein, May,
ent aspects of motor behavior. Specifically, some output & Mercier, 1985; Haber, Kim, Mailly, & Calzavara, 2006;
channels appear to be especially concerned with move- Kemp & Powell, 1971; Schmahmann & Pandya, 1997).
ments that are internally generated, whereas others We tested whether regions of prefrontal cortex not only
appear to be devoted to movements guided by extero- project to the basal ganglia and cerebellum, but also
ceptive cues. Taken together, these observations indi- are the target of basal ganglia and cerebellar outputs.
cate that the basal ganglia and cerebellum have the Virus injections into areas 9, 12, or 46 labeled many
capacity to influence a broad range of motor behavior neurons in the output nuclei of the basal ganglia
using output channels that project to the premotor (figures 36.1, 36.2, and 36.6; Middleton & Strick, 1994,
areas in the frontal lobe, as well as to M1. Thus, the 2002). Injections into area 12 labeled neurons in a
skeletomotor circuit of Alexander et al. (1986) is more localized portion of SNpr. In contrast, injections
accurately viewed as multiple discrete channels to each into area 46 labeled neurons largely in GPi. Area 9
of the cortical motor areas (figure 36.5; Middleton & injections labeled neurons in both SNpr and GPi. The
Strick, 2001b). A similar arrangement of output topographic nature of basal ganglia projections to
Old Revised
Cortex SMA APA, MC, SC M1 PMv SMA PMd* CMAr* CMAd* CMAv*
F A L
Figure 36.5 The original skeletomotor circuit proposed by pallidus; PUT, putamen; SNr, substantia nigra pars reticulata;
Alexander, DeLong, and Strick (1986) and our revised cl, caudolateral; mid, middle; vl, ventrolateral. Thalamic
scheme. Asterisks indicate loops whose existence is predicted abbreviations: VApc, nucleus ventralis anterior, parvocellular
but not specifically tested using virus transport. Cortical division; VLcc, nucleus ventralis lateralis pars caudalis,
abbreviations: CMAd, dorsal cingulate motor area; CMAr, caudal division; VLcr, nucleus ventralis lateralis pars caudalis,
rostral cingulate motor area; CMAv, ventral cingulate motor rostral division; VLm, nucleus ventralis lateralis pars medialis;
area; M1, primary motor cortex; PMd, dorsal premotor area; VLo, nucleus ventralis lateralis pars oralis. (From Middleton
PMv, ventral premotor area; SMA, supplementary motor area. & Strick, 2001b, with permission from Elsevier.)
Basal ganglia abbreviations; GPi, internal segment of globus
SNpr SNpr
Caudal Rostral
FEF Area TE Area 12 Area 9m Area 9l
pr
pc pc
pc pr pr
pr pr
CC CC
D
pc
CC CC CC
pc M
1 mm
Figure 36.6 Origin of nigral projections to the FEF, area TE, different cortical areas. Abbreviations: CC, crus cerebri; pc,
area 12, area 9 m, and area 9 l. Coronal sections indicating pars compacta; pr, pars reticulata. (Adapted from Middleton
the location of labeled neurons in the caudal and rostral & Strick, 2000, with permission from Elsevier.)
regions of the SNpr following virus injections into the
prefrontal cortex is further emphasized by the finding Virus injections into areas 9 and 46 (but not area
that different regions within rostral SNpr project to 12) labeled neurons in ventral regions of the dentate
medial and lateral portions of area 9 (figure 36.6, areas nucleus (figure 36.3; Middleton & Strick, 1998,
9m and 9l). In all cases, the locations of the neurons 2001a). The neurons labeled after area 9 injections
labeled in GPi and SNpr after injections into prefrontal were found largely medial and caudal to those labeled
areas of cortex were different from the locations of by area 46 injections. The ventral regions of the
neurons labeled after injections into motor areas of dentate that project to these nonmotor areas in the
cortex. frontal lobe clearly differ from the more dorsal regions
dum, bostan, and strick: basal ganglia and cerebellar circuits 423
of this nucleus that innervate motor areas of the 2001, 2005). In addition, area MIP in the anterior bank
cortex (figures 36.3 and 36.4). Thus, both the basal of the IPS and area LIP in the posterior bank of the IPS
ganglia and the cerebellum project via the thalamus to receive output from the cerebellum (Prevosto et al.,
multiple areas of prefrontal cortex. Moreover, the 2010). These results clearly indicate that the sphere of
output channels in the basal ganglia and cerebellum influence of basal ganglia and cerebellar output extends
that influence prefrontal areas of cortex are separate to include portions of the posterior parietal cortex.
from those that influence motor areas of cortex. This Although the implications of basal ganglia and cerebel-
observation suggests that GPi and the dentate can be lar projections to posterior parietal cortex cannot be
divided into distinct motor and nonmotor domains fully addressed here, we will highlight two specific pro-
(figures 36.2 and 36.4; Akkal et al., 2007; Dum & posals about these circuits. The cerebellar projection to
Strick, 2003). posterior parietal cortex may provide signals that con-
The presupplementary motor area (pre-SMA) has tribute to sensory recalibration during some adaptation
traditionally been included with the motor areas of the paradigms (Clower et al., 2001; Prevosto et al., 2010).
frontal lobe. However, a number of recent observa- In addition, we have suggested (Clower et al., 2005) that
tions emphasize the nonmotor nature of this cortical abnormal signals in the basal ganglia projection to the
area (Picard & Strick, 2001). For example, unlike the posterior parietal cortex may contribute to the visuo-
cortical motor areas, the pre-SMA does not project spatial deficits observed in some patients with basal
directly to M1 or to the spinal cord. Instead, the ganglia disorders (Karnath, Himmelbach, & Rorden,
pre-SMA is densely interconnected with regions of pre- 2002).
frontal cortex. Virus tracing was used to test whether
basal ganglia and cerebellar projections to the pre-SMA Inferotemporal cortex
originate from the motor or the nonmotor domains of
GPi and the dentate (Akkal et al., 2007). Virus injected In general, each cortical area that projects to the basal
into the pre-SMA labeled neurons dorsally in the ganglia or cerebellum is known to receive projections
rostral portion of GPi (figure 36.2) and in a ventral back from these subcortical nuclei. Area TE, a region
portion of the dentate (figure 36.4; Akkal et al., 2007). of inferotemporal cortex, is known to project to the
Thus, the output channels to the pre-SMA from GPi input stage of basal ganglia processing (i.e., the tail of
and dentate are grouped with output channels that the caudate and ventral portions of the putamen; Saint-
project to regions of prefrontal cortex rather than Cyr, Ungerleider, & Desimone, 1990), but not to the
output channels to the cortical motor areas (figures input stage of cerebellar processing (Glickstein et al.,
36.2 and 36.4). These observations provide further 1985; Schmahmann & Pandya, 1997). Consistent with
support for the proposal that the pre-SMA is more these observations, virus injections into area TE did not
similar to regions of prefrontal cortex than it is to the result in any labeled neurons in the cerebellar nuclei,
cortical motor areas (Akkal et al., 2007; Picard & Strick, but did label a distinct cluster of neurons in SNpr (Mid-
2001). dleton & Strick, 1996; figure 36.6, area TE). Most of
these neurons were located in a dorsal region of caudal
Posterior parietal cortex SNpr that appears to be separate from the regions that
influence the FEF or subdivisions of prefrontal cortex.
Areas 5 and 7 in posterior parietal cortex also are known Thus, area TE is both a source of input to, and a target
to project to the input stage of basal ganglia and cere- of output from, a distinct portion of the basal ganglia.
bellar processing (e.g., Cavada & Goldman-Rakic, 1991; In contrast, TE neither projects to nor receives input
Glickstein et al., 1985; Kemp & Powell, 1971; Schmah- from the cerebellum.
mann & Pandya, 1997; Yeterian & Pandya, 1993). This TE is known to play a critical role in the visual recog-
raised the possibility that these regions of cortex also nition and discrimination of objects (e.g., Gross, 1972;
are the target of basal ganglia and cerebellar outputs. Miyashita, 1993; Tanaka, Saito, Fukuda, & Moriya,
This possibility has been tested in several experiments 1991). Physiological studies have shown that the region
(Clower, Dum, & Strick, 2005; Clower, West, Lynch, & of SNpr that influences TE contains some neurons that
Strick, 2001; Prevosto, Graf, & Ugolini, 2010). Virus are responsive to the presentation of visual stimuli (e.g.,
tracing demonstrated that both portions of area 7b in Hikosaka & Wurtz, 1983a). These observations, together
the intraparietal sulcus (IPS) and on the cortical surface with our anatomical results, provide evidence that basal
are the target of output from the dentate nucleus, ganglia output is involved in higher-order aspects of
whereas only the portion of area 7b on the cortical visual processing, as well as in motor and cognitive
surface is the target of output from SNpr (Clower et al., function.
dum, bostan, and strick: basal ganglia and cerebellar circuits 425
cerebellum and describe the functional topography in the cerebellum and basal ganglia are components of an
cerebellar cortex (e.g., E, Chen, Ho, & Desmond, 2012; interconnected network concerned with motor and
Ito, 2008, 2011; Ramnani, 2006; Stoodley, 2012; nonmotor aspects of behavior.
Stoodley & Schmahmann, 2009; Stoodley, Valera, &
Schmahmann, 2012; Strick et al., 2009), this subject will Functional implications
not be presented in detail here. Nevertheless, it is clear
that the sites of activation in cognitive and emotion Clearly, the outputs from the basal ganglia and cerebel-
tasks are quite separate from those observed in motor lum gain access to more widespread and diverse areas
tasks, and the results of these studies are fully consistent of cortex than previously imagined. To date, our studies
with the view that cerebellar cortex has distinct motor have shown that the output nuclei of the basal ganglia
and nonmotor domains. and cerebellum project (via the thalamus) to skeleto-
motor, oculomotor, prefrontal, and posterior parietal
The cerebellum is interconnected with the areas of cortex. In addition, a portion of SNpr projects
basal ganglia to inferotemporal cortex. Thus, the anatomical sub-
strate exists for the basal ganglia and cerebellum to
The loops that link the cerebellum with the cerebral influence higher-order aspects of cognition like plan-
cortex have traditionally been considered to be ana- ning, working memory, sequential behavior, visuospa-
tomically and functionally distinct from those that link tial perception, and attention as well as skeletomotor
the basal ganglia with the cerebral cortex (Doya, 2000; and oculomotor function. As a consequence, a sizeable
Graybiel, 2005). The outputs from the cerebellum and component of basal ganglia and cerebellar output oper-
basal ganglia to the cerebral cortex are relayed through ates outside of the domain of motor control.
separate thalamic nuclei (Percheron et al., 1996; Sakai, The new insights gained from virus tracing have
Inase, & Tanji, 1996). Any interactions between cere- important implications for hypotheses about basal
brocerebellar and cerebro–basal ganglia loops were ganglia and cerebellar contributions to normal and
thought to occur primarily at the neocortical level. abnormal behavior. Detailed discussions of this issue
Results from recent anatomical experiments challenge have been presented in our recent reviews (Bostan,
this perspective and provide evidence for disynaptic Dum, & Strick, 2013; Middleton & Strick, 1998; Strick
pathways that directly link the cerebellum with the basal et al., 2009), and therefore only some examples will be
ganglia. presented here. Abnormal activity in basal ganglia and
Transneuronal transport of virus demonstrated that cerebellar loops with cortical motor areas has been
the dentate nucleus projects disynaptically to the stria- shown to contribute to the symptoms of certain motor
tum (caudate and putamen; figure 36.8; Hoshi, Trem- disorders, particularly Parkinson’s disease and dystonia
blay, Féger, Carras, & Strick, 2005). Projections to the (for reviews, see Filip, Lungu, & Bareš, 2013; Sadnicka,
striatum originate from motor and nonmotor domains Hoffland, Bhatia, van de Warrenburg, & Edwards, 2012;
in the dentate and terminate in regions of putamen and Wu & Hallett, 2013). In Parkinson’s disease, the loss of
caudate known to be within the “sensorimotor” and dopaminergic neurons of the substantia nigra pars
“associative” territories of these nuclei (see Parent & compacta results in tremor, rigidity, bradykinesia, and
Hazrati, 1995). These findings indicate that the disyn- akinesia (Wichmann, DeLong, Guridi, & Obeso, 2011).
aptic pathway from the dentate to the striatum enables However, cerebellar activity is also abnormal in Parkin-
the cerebellum to influence both nonmotor and motor son’s patients (Catalan, Ishii, Honda, Samii, & Hallett,
function within the basal ganglia. 1999; Ghaemi et al, 2002; Rascol et al., 1997). In par-
In comparable experiments, virus transport demon- kinsonian patients (Lenz et al., 1988; Ohye, Saito, Fuka-
strated that the subthalamic nucleus (STN) projects machi, & Narabayashi, 1974) and in monkey models of
disynaptically to cerebellar cortex (figures 36.8 and the disease (Guehl et al., 2003), oscillatory activity at
36.9; Bostan et al., 2010). Projections to the cerebellar tremor frequencies has been recorded in regions of the
cortex originate from motor and nonmotor domains thalamus that receive cerebellar, not basal ganglia,
within the STN (see Parent & Hazrati, 1995). Further- efferents. Furthermore, the cerebellar receiving thala-
more, the projections terminate in motor and nonmo- mus is one of the most effective surgical sites for treat-
tor regions of the cerebellar cortex. These findings ing parkinsonian tremor (Narabayashi, Maeda, &
indicate that the disynaptic pathway from the STN to Yokochi, 1987). These results suggest that abnormal
the cerebellar cortex enables the basal ganglia to influ- activity in cerebellar circuits may account for parkinso-
ence both nonmotor and motor function within the nian tremor. Furthermore, deep-brain stimulation of
cerebellum. Taken together, these studies indicate that the STN is not only highly effective in reducing the
STN STN
GPe GPe
Output Output
GPi DN GPi Stage DN
Stage
Thalamus Thalamus
PN PN
Basal Ganglia Output to the Cerebellum Basal Ganglia Output to the Cerebellum
STN STN
GPe GPe
Output Output
GPi DN GPi Stage DN
Stage
Thalamus Thalamus
PN PN
dum, bostan, and strick: basal ganglia and cerebellar circuits 427
A Injection Sites F.ps. Figure 36.9 The subthalamic nucleus (STN) projects to the
a Crus I cerebellar hemisphere. (A) Rabies virus injection sites in the
I p F.in.cr. cerebellar cortex. Left: Flattened map of the cerebellar cortex
II
in a Cebus monkey. Right: Shaded region on the left map is
III a expanded to show an injection site in Crus IIp (AB2, gray)
IV
and another in HVIIB (AB3, black). (B) Combined charts of
VI
Crus II STN neurons labeled by injections of rabies virus into crus IIp
V
AB2 p
(gray dots) or HVIIB (black dots) illustrate the topographic
VI VII
AB2 distribution of STN projections to different cerebellar cortical
VII AB3 lobules. (C) Organization of the STN according to the func-
AB3 F.amp.
VIII VIII tional subdivisions of the basal ganglia. (D) Schematic
summary of projections from the cerebral cortex to the STN.
HVIIB
IX C: caudal; D: dorsal; M: medial. (Adapted from Bostan et al.,
F.ppd. 2013, with permission from Elsevier.)
X
B STN to Cerebellum
Crus IIp Injection
HVIIB Injection Dystonia, another motor disorder often attributed to
the basal ganglia (Neychev, Gross, Lehéricy, Hess, &
Jinnah, 2011), is characterized by involuntary muscle
Caudal contractions, twisting movements and abnormal pos-
D tures (Bhatia & Marsden, 1994). However, dystonia can
also arise from cerebellar dysfunction and may be better
Mid C described as a network disorder involving the basal
M
Rostral 1mm ganglia and cerebellum (LeDoux, 2011; Neychev et al.,
C STN Territories 2011). Human carriers of genetic mutations associated
with dystonia exhibit abnormalities in both the basal
ganglia and the cerebellum (Argyelan et al., 2009;
SMPM
M1
Carbon et al., 2008; Carbon, Argyelan, & Eidelberg,
SM AS 2010; Carbon & Eidelberg, 2009; Eidelberg, 1998;
M1 Caudal Ghilardi et al., 2003; Trost et al., 2002). In addition,
SM
PM SM Sensorimotor abnormal cerebellar activity drives dystonic movements
AS LI
AS Associative both in a pharmacological model of dystonia in normal
AS LI LI Limbic
Mid mice and in mutant tottering mice (Campbell & Hess,
Rostral 1998; Campbell, North, & Hess, 1999; Chen et al., 2009;
D Cortex to STN Pizoli, Jinnah, Billingsley, & Hess, 2002; Neychev, Fan,
M1 Mitev, Hess, & Jinnah, 2008). Overall, these findings
PM support the importance of functional interactions
M1
M1 FEF between the cerebellum and the basal ganglia in the
PF
PF
PM manifestation of motor disorders typically associated
M1
M1
PF
PF
Caudal with the basal ganglia.
PM FEF Several lines of evidence also implicate the intercon-
PM M1 Primary Motor nections between the basal ganglia and cerebellum in
PF Mid PM Premotor
FEF PF Prefrontal nonmotor processes, such as associative reward-related
FEF Frontal Eye Field
Rostral learning, as well as in neuropsychiatric dysfunction.
The cerebellum and basal ganglia are typically viewed
as segregated modules that participate in different
motor symptoms in Parkinson’s disease (Krack, Fraix, aspects of learning. The cerebellum is thought to be
Mendes, Benabid, & Pollak, 2002), but also normalizes involved in adaptive modification of behavior and error-
cerebellar activity and function (Geday, Østergaard, based learning, whereas the basal ganglia is thought to
Johnsen, & Gjedde, 2009; Grafton et al., 2006; Hilker be involved in reward-prediction and reward-based
et al., 2004; Payoux et al., 2004; Trost et al., 2006). The learning (see Doya, 2000; Houk, 2005). Accounts of
disynaptic connection from the STN to the cerebellum reward-related learning have emphasized the role of
may be the anatomical substrate that mediates this the basal ganglia based on the hypothesis that dopa-
effect of STN stimulation (figure 36.8). mine neurons reflect reward-prediction error and
dum, bostan, and strick: basal ganglia and cerebellar circuits 429
attention-deficit disorder, obsessive-compulsive disor- Bostan, A. C., & Strick, P. L. (2010). The cerebellum and
der, and Tourette syndrome are associated with altera- basal ganglia are interconnected. Neuropsychol Rev, 30,
261–270.
tions in basal ganglia or cerebellar function. Thus, it is
Botez, M. I., Botez, T., Elie, R., & Attig, E. (1989). Role of
possible that abnormal activity in specific basal ganglia the cerebellum in complex human behavior. Ital J Neurol
and cerebellar loops with the cerebral cortex results in Sci, 10, 291–300.
identifiable sets of neuropsychiatric symptoms. More- Brogden, W. J., & Gnatt, W. H. (1942). Intraneural condi-
over, we have provided evidence that the cerebellum tioning: Cerebellar conditioned reflexes. Arch Neurol Psychol,
and basal ganglia are interconnected. These intercon- 48, 18.
Brooks, V. B., & Thach, W. T. (1981). Cerebellar control of
nections may allow the two major subcortical nuclei to posture and movement. In V. B. Brooks (Ed.), Handbook of
cooperate in the generation of a broad range of normal physiology, Section 1: The nervous system, Vol. 2, Motor control,
motor and nonmotor functions; the interconnections Part II (pp. 877–946). Bethesda, MD: American Physiologi-
may also enable dysfunction in one circuit to recruit cal Society.
abnormal activity in its partner and, thus, provoke a Buckner, R. L., Krienen, F. M., Castellanos, A., Diaz, J. C.,
& Yeo, B. T. (2011). The organization of the human cere-
diverse set of motor and neuropsychiatric disorders. bellum estimated by intrinsic functional connectivity. J Neu-
rophysiol, 106, 2322–2345.
Campbell, D. B., & Hess, E. J. (1998). Cerebellar circuitry is
ACKNOWLEDGMENTS This work was supported in part by funds activated during convulsive episodes in the tottering (tg/
from the Office of Research and Development, Medical tg) mutant mouse. Neuroscience, 85, 773–783.
Research Service, and Department of Veterans Affairs, and by Campbell, D. B., North, J. B., & Hess, E. J. (1999). Tottering
National Institutes of Health Grants R01 NS24328 (PLS), R01 mouse motor dysfunction is abolished on the Purkinje cell
MH56661 (PLS), P40 OD010996 (PLS), and P30 NS076405 degeneration (pcd) mutant background. Exp Neurol, 160,
(PLS). 268–278.
Carbon, M., & Eidelberg, D. (2009). Abnormal structure-
function relationships in hereditary dystonia. Neuroscience,
164, 220–229.
Carbon, M., Ghilardi, M. F., Argyelan, M., Dhawan, V.,
REFERENCES
Bressman, S. B., & Eidelberg, D. (2008). Increased cere-
Akkal, D., Dum, R. P., & Strick, P. L. (2007). Supplementary bellar activation during sequence learning in DYT1 carriers:
motor area and presupplementary motor area: Targets of An equiperformance study. Brain, 131, 146–154.
basal ganglia and cerebellar output. J Neurosci, 27, Carbon, M., Argyelan, M., & Eidelberg, D. (2010). Func-
10659–10673. tional imaging in hereditary dystonia. Eur J Neurosci, 17,
Akshoomoff, N. A., & Courchesne, E. (1992). A new role 58–64.
for the cerebellum in cognitive function. Behav Neurosci, Catalan, M. J., Ishii, K., Honda, M., Samii, A., & Hallett,
106, 731–738. M. (1999). A PET study of sequential finger movements of
Alexander, G. E., DeLong, M. R., & Strick, P. L. (1986). varying length in patients with Parkinson’s disease. Brain,
Parallel organization of functionally segregated circuits 122, 483–495.
linking basal ganglia and cortex. Annu Rev Neurosci, 9, Cavada, C., & Goldman-Rakic, P. S. (1991). Topographic
357–381. segregation of corticostriatal projections from posterior
Allen, G. I., & Tsukahara, N. (1974). Cerebrocerebellar parietal subdivisions in the macaque monkey. Neuroscience,
communication systems. Physiol Rev, 54, 957–1006. 42, 683–696.
Argyelan, M., Carbon, M., Niethammer, M., Ulug, A. M., Chen, G., Popa, L. S., Wang, X., Gao, W., Barnes, J., Hendrix,
Voss, H. U., Bressman, S. B., … Eidelberg, D. (2009). C. M., … Ebner, T. J. (2009). Low-frequency oscillations in
Cerebellothalamocortical connectivity regulates pene- the cerebellar cortex of the tottering mouse. J Neurophysiol,
trance in dystonia. J Neurosci, 29, 9740–9747. 101, 234–245.
Bellebaum, C., Koch, B., Schwarz, M., & Daum, I. (2008). Clower, D. M., West, R. A., Lynch, J. C., & Strick, P. L.
Focal basal ganglia lesions are associated with impairments (2001). The inferior parietal lobule is the target of output
in reward-based reversal learning. Brain, 131, 829–841. from the superior colliculus, hippocampus and cerebellum.
Bhatia, K. P., & Marsden, C. D. (1994). The behavioural and J Neurosci, 21, 6283–6291.
motor consequences of focal lesions of the basal ganglia in Clower, D. M., Dum, R. P., & Strick, P. L. (2005). Basal
man. Brain, 117, 859–876. ganglia and cerebellar inputs to “AIP.” Cereb Cortex, 15,
Bonson, K. R., Grant, S. J., Contoreggi, C. S., Links, J. M., 913–920.
Metcalfe, J., Weyl, H. L., … London, E. D. (2002). Neural Courchesne, E. (1997). Brainstem, cerebellar and limbic
systems and cue-induced cocaine craving. Neuropsychophar- neuroanatomical abnormalities in autism. Curr Opin Neuro-
macology, 26, 376–386. biol, 7, 269–278.
Bostan, A. C., Dum, R. P., & Strick, P. L. (2010). The basal Courchesne, E., Townsend, J., Akshoomoff, N., Saitoh, O.,
ganglia communicates with the cerebellum. Proc Natl Acad Yeung-Courchesne, R., Lincoln, A., … Lau, L. (1994).
Sci USA, 107, 8452–8456. Impairment in shifting attention in autistic and cerebellar
Bostan, A. C., Dum, R. P., & Strick, P. L. (2013). Cerebellar patients. Behav Neurosci, 108, 848–865.
networks with the cerebral cortex and basal ganglia. Trends Courchesne, E., Yeung-Courchesne, R., Press, G. A.,
Cogn Sci, 17, 241–254. Hesselink, J. R., & Jernigan, T. L. (1988). Hypoplasia of
dum, bostan, and strick: basal ganglia and cerebellar circuits 431
Houk, J. C. (2005). Agents of the mind. Biol Cybern, 92, Lichter, D. G., & Cummings, J. L. (2000). Frontal-subcortical
427–437. circuits in psychiatry and neurology. New York, NY: Guilford.
Inase, M., & Tanji, J. (1995). Thalamic distribution of projec- Liljeholm, M., & O’Doherty, J. P. (2012). Contributions of
tion neurons to the primary motor cortex relative to affer- the striatum to learning, motivation, and performance: An
ent terminal fields from the globus pallidus in the macaque associative account. Trends Cogn Sci, 16, 467–475.
monkey. J Comp Neurol, 353, 415–426. Lu, X., Miyachi, S., Ito, Y., Nambu, A., & Takada, M. (2007).
Ito, M. (2008). Control of mental activities by internal models Topographic distribution of output neurons in cerebellar
in the cerebellum. Nat Rev Neurosci, 9, 304–313. nuclei and cortex to somatotopic map of primary motor
Ito, M. (2011). The cerebellum: Brain for an implicit self. Upper cortex. Eur J Neurosci, 25, 2374–2382.
Saddle River, NJ: FT Press. Lynch, J. C., Hoover, J. E., & Strick, P. L. (1994). Input to
Ivry, R. B., & Keele, S. W. (1989). Timing functions of the the primate frontal eye field from the substantia nigra,
cerebellum. J Cog Neurosci, 1, 136–152. superior colliculus, and dentate nucleus demonstrated by
Jinnai, K., Nambu, A., Tanibuch, I., & Yoshida, S. (1993). transneuronal transport. Exp Brain Res, 100, 181–186.
Cerebello- and pallido-thalamic pathways to areas 6 and 4 McClernon, F. J., Kozink, R. V., Lutz, A. M., & Rose, J. E.
in the monkey. Stereotact Funct Neurosurg, 60, 70–79. (2009). 24-h smoking abstinence potentiates fMRI-BOLD
Karnath, H. O., Himmelbach, M., & Rorden, C. (2002). activation to smoking cues in cerebral cortex and dorsal
The subcortical anatomy of human spatial neglect: Putamen, striatum. Psychopharmacology (Berl), 204(1), 25–35.
caudate nucleus and pulvinar. Brain, 125, 350–360. Manni, E., & Petrosini, L. (2004). A century of cerebellar
Kelly, R. M., & Strick, P. L. (2000). Rabies as a transneuro- somatotopy: A debated representation. Nat Rev Neurosci, 5,
nal tracer of circuits in the central nervous system. J Neurosci 241–249.
Methods, 103, 63–71. Middleton, F. A., & Strick, P. L. (1994). Anatomical evi-
Kelly, R. M., & Strick, P. L. (2003). Cerebellar loops with dence for cerebellar and basal ganglia involvement in
motor cortex and prefrontal cortex of a nonhuman primate. higher cognitive function. Science, 266, 458–461.
J Neurosci, 12, 8432–8444. Middleton, F. A., & Strick, P. L. (1996). The temporal lobe
Kelly, R. M., & Strick, P. L. (2004). Macro-architecture of is a target of output from the basal ganglia. Proc Natl Acad
basal ganglia loops with the cerebral cortex: Use of rabies Sci USA, 93, 8683–8687.
virus to reveal multisynaptic circuits. Prog Brain Res, 143, Middleton, F. A., & Strick, P. L. (1998). Cerebellar output:
449–459. Motor and cognitive channels. Trends Cogn Sci, 2,
Kemp, J. M., & Powell, T. P. S. (1971). The connexions of the 348–354.
striatum and globus pallidus: Synthesis and speculation. Middleton, F. A., & Strick, P. L. (2000). Basal ganglia output
Philos Trans R Soc Lond B Biol Sci, 262, 441–457. and cognition: Evidence from anatomical, behavioral, and
Koob, G. F., & Volkow, N. D. (2010). Neurocircuitry of addic- clinical studies. Brain Cogn, 42, 183–200.
tion. Neuropsychopharmacology, 35, 217–238. Middleton, F. A., & Strick, P. L. (2001a). Cerebellar projec-
Krack, P., Fraix, V., Mendes, A., Benabid, A. L., & Pollak, tions to the prefrontal cortex of the primate. J Neurosci, 21,
P. (2002). Postoperative management of subthalamic 700–712.
nucleus stimulation for Parkinson’s disease. Mov Disord, Middleton, F. A., & Strick, P. L. (2001b). A revised neuro-
17(Suppl. 3), S188–197. anatomy of frontal subcortical circuits. In D. G. Lichter &
Krienen, F. M., & Buckner, R. L. (2009). Segregated fronto- J. L. Cummings (Eds.), Frontal-subcortical circuits in psychiatry
cerebellar circuits revealed by intrinsic functional connec- and neurology (pp. 44–58). New York, NY: Guilford.
tivity. Cereb Cortex, 19, 2485–2497. Middleton, F. A., & Strick, P. L. (2002). Basal ganglia “pro-
Larsell, O. (1970). The comparative anatomy and histology of the jections” to the prefrontal cortex. Cereb Cortex, 12,
cerebellum from monotremes through apes. Minneapolis: Univer- 926–935.
sity of Minnesota. Miquel, M., Toledo, R., García, L. I., Coria-Avila, G. A., &
LeDoux, M. S. (2011). Animal models of dystonia: Lessons Manzo, J. (2009). Why should we keep the cerebellum in
from a mutant rat. Neurobiol Dis, 42, 152–161. mind when thinking about addiction? Curr Drug Abuse Rev,
Leiner, H. C., Leiner, A. L., & Dow, R. S. (1986). Does the 2(1), 26–40.
cerebellum contribute to mental skills? Behav Neurosci, 100, Miyashita, Y. (1993). Inferior temporal cortex: Where visual
443–454. perception meets memory. Annu Rev Neurosci, 16,
Leiner, H. C., Leiner, A. L., & Dow, R. S. (1991). The human 245–263.
cerebro-cerebellar system: Its computing, cognitive, and Mushiake, H., & Strick, P. L. (1993). Preferential activity of
language skills. Behav Brain Res, 44, 113–128. dentate neurons during limb movements. J Neurophysiol, 70,
Leiner, H. C., Leiner, A. L., & Dow, R. S. (1993). Cognitive 2660–2664.
and language functions of the human cerebellum. Trends Mushiake, H., & Strick, P. L. (1995). Pallidal neuron activity
Neurosci, 16, 444–447. during sequential arm movements. J Neurophysiol, 74,
Lenz, F. A., Tasker, R. R., Kwan, H. C., Schnider, S., Kwong, 2754–2758.
R., Murayama, Y., … Murphy, J. T. (1988). Single unit Narabayashi, H., Maeda, T., & Yokochi, F. (1987). Long-
analysis of the human ventral thalamic nuclear group: Cor- term follow-up study of nucleus ventralis intermedius and
relation of thalamic “tremor cells” with the 3–6 Hz compo- ventrolateralis thalamotomy using a microelectrode tech-
nent of Parkinsonian tremor. J Neurosci, 8, 754–764. nique in parkinsonism. Appl Neurophysiol, 50, 330–337.
Lerner, A., Bagic, A., Boudreau, E. A., Hanakawa, T., Neychev, V. K., Fan, X., Mitev, V. I., Hess, E. J., & Jinnah,
Pagan, F., Mari, Z., … Hallett, M. (2007). Neuroimaging H. A. (2008). The basal ganglia and cerebellum interact
of neuronal circuits involved in tic generation in patients in the expression of dystonic movement. Brain, 131,
with Tourette syndrome. Neurology, 68(23), 1979–1987. 2499–2509.
dum, bostan, and strick: basal ganglia and cerebellar circuits 433
D. G. Beiser (Eds.), Models of information processing in the van Kan, P. L. E., Houk, J. C., & Gibson, A. R. (1993). Output
basal ganglia (pp. 117–130). Cambridge, MA: MIT Press. organization of intermediate cerebellum of the monkey.
Swain, R. A., Kerr, A. L., & Thompson, R. F. (2011). The J Neurophysiol, 69, 57–73.
cerebellum: A neural system for the study of reinforcement Wichmann, T., DeLong, M. R., Guridi, J., & Obeso, J. A.
learning. Front Behav Neurosci, 5, 8. (2011). Milestones in research on the pathophysiology of
Tanaka, K., Saito, H.-A., Fukuda, Y., & Moriya, M. (1991). Parkinson’s disease. Mov Disord, 26, 1032–1041.
Coding visual images of objects in the inferotemporal Wiestler, T., McGonigle, D. J., & Diedrichsen, J. (2011).
cortex of the macaque monkey. J Neurophysiol, 66, Integration of sensory and motor representations of single
170–189. fingers in the human cerebellum. J Neurophysiol, 105,
Tanaka, S. C., Doya, K., Okada, G., Ueda, K., Okamoto, Y., 3042–3053.
& Yamawaki, S. (2004). Prediction of immediate and future Wu, T., & Hallett, M. (2013). The cerebellum in Parkinson’s
rewards differentially recruits cortico-basal ganglia loops. disease. Brain, 136, 696–709.
Nat Neurosci, 7, 887–893. Yalachkov, Y., Kaiser, J., & Naumer, M. J. (2009). Brain
Thoma, P., Bellebaum, C., Koch, B., Schwarz, M., & regions related to tool use and action knowledge reflect
Daum, I. (2008). The cerebellum is involved in reward- nicotine dependence. J Neurosci, 29, 4922–4929.
based reversal learning. Cerebellum, 7, 433–443. Yalachkov, Y., Kaiser, J., & Naumer, M. J. (2010). Sensory
Thompson, R. F., Swain, R., Clark, R., & Shinkman, P. and motor aspects of addiction. Behav Brain Res, 207,
(2000). Intracerebellar conditioning—Brogden and Gantt 215–222.
revisited. Behav Brain Res, 110, 3–11. Yang, Z., Xie, J., Shao, Y. C., Xie, C. M., Fu, L. P., Li, D. J.,
Trost, M., Carbon, M., Edwards, C., Ma, Y., Raymond, D., … Li, S. J. (2009). Dynamic neural responses to cue-reac-
Mentis, M. J., … Eidelberg, D. (2002). Primary dystonia: tivity paradigms in heroin-dependent users: An fMRI study.
Is abnormal functional brain architecture linked to geno- Hum Brain Mapp, 30, 766–775.
type? Ann Neurol, 52, 853–856. Yeterian, E. H., & Pandya, D. N. (1993). Striatal connections
Trost, M., Su, S., Su, P., Yen, R. F., Tseng, H. M., Barnes, A., of the parietal association cortices in rhesus monkeys.
… Eidelberg, D. (2006). Network modulation by the sub- J Comp Neurol, 332, 175–197.
thalamic nucleus in the treatment of Parkinson’s disease.
NeuroImage, 31, 301–307.
ABSTRACT The basal ganglia (BG) is composed of several Basal ganglia: A group of interconnected
heavily interconnected nuclei at the base of the cerebrum. A subcortical nuclei
series of anatomically distinct parallel circuits through the BG
receive afferent projections from and project back to cortical
regions that mediate skeletomotor, oculomotor, frontal asso- Four major nuclei make up the BG: the striatum, the
ciative, and limbic functions. Because of this parallel circuit globus pallidus, the subthalamic nucleus (STN), and
organization, it is fair to say that the BG contributes to a the substantia nigra. The approximate location of these
panoply of brain functions ranging from motor control to nuclei in the primate brain is illustrated in figure 37.1A.
cognition and motivation. The specific contribution of the BG The striatum, the largest component of the BG, includes
to these brain functions (i.e., the algorithm or information
transform that it performs) remains a topic of active research.
the caudate nucleus and the putamen. The globus pal-
All circuits share a common intrinsic organization captured lidus is divided into external (GPe) and internal (GPi)
largely by the heuristic model of direct, indirect, and hyper- segments. The substantia nigra, located in the mid-
direct pathways that link BG input stations to outputs. Activa- brain, gets its name from its compact dorsal part (SNc),
tion of the direct pathway may facilitate movement, whereas which is heavily pigmented in primates due to the pres-
the indirect and hyperdirect pathways may suppress move-
ence of brown neuromelanin in dopamine neurons. Its
ment. Neuromodulators such as dopamine have different
effects on activity and synaptic plasticity in the three pathways. ventral part, with a more reticulated appearance, is the
Clinical disorders that involve the BG are often associated pars reticulata (SNr).
with movement disorders. Although the actual contributions Because the connections between the nuclei of the
of the BG to brain function are still actively debated, growing BG are complex, it is most interesting to consider these
evidence suggests that it acts as a reinforcement-driven tutor
circuits from the perspective of the information that
for the learning of automatic behavioral routines.
enters and leaves the BG. Where does it come from,
how is it directed within the BG, what happens to it
within the BG, where does it go when it leaves the BG,
The basal ganglia (BG) is composed of a group of and what influence does it exert on behavior?
nuclei situated at the central base of the forebrain Information reaching the BG comes from two major
(figure 37.1A). Heavily interconnected with the cere- sources: the cerebral cortex and the thalamus. Most
bral cortex, this structural network provides a site at information enters the BG at the level of the striatum,
which segregated functional information from diverse although some key inputs enter at the STN. Signals
cortical areas can interact and be influenced by multi- leave the BG from two structures: the GPi and the SNr.
ple neuromodulators such as dopamine. Outputs from Because there are multiple routes from inputs to
the BG are directed primarily back to the cortex through outputs, information flow through the BG may be
the thalamus (especially toward the frontal lobe), and complex and involve multiple steps of processing
to brainstem structures. Inputs that originate from (figure 37.1).
functionally distinct cortical areas are processed in dif-
ferent parallel loop circuits through the BG (Alexan- Anatomically and functionally distinct
der, Crutcher, & DeLong, 1990). Consistent with this parallel circuits
anatomical organization, the activity of BG output
neurons in these circuits carries signals related to a The BG is organized as a series of anatomically segre-
variety of functions, including voluntary movement of gated loop circuits (Alexander, DeLong, & Strick, 1986;
the limbs, eye movement, decision making, and moti- Middleton & Strick, 2000). This topic is reviewed in
vational control. depth elsewhere in this volume (see chapter 36 by Dum,
How is this information processed, combined, and Bostan, & Strick) and is discussed briefly here. Different
modulated in the BG? How do BG disorders disrupt areas of cortex terminate within specific territories of
stable posture and smooth efficient movement? What the BG that are connected to similarly specific portions
does this tell us about the role of the BG in the normal of the thalamus, which in turn project back to innervate
control of behavior? the areas of cortex from which the circuit originates.
Cortex
STN GPi/SNr
SNc GPi GPe GABA
SNr Glutamate
Dopamine
SNc
Brainstem
Putamen Nucleus
Striatum posterior lat. Caudate body Caudate head
accumbens
Sup.
Brainstem PPN PPN ?
coll.
Figure 37.1 Circuit diagrams of the basal ganglia (BG) and median–parafascicular intralaminar complex (CMPf) of the
associated input-output connections. (A) The positions of key thalamus. CMPf closes another loop by projecting back to the
BG structures involved in skeletomotor control and their basic striatum. GPi also projects to brainstem regions such as the
input-output connectivity are indicated in a parasagittal pedunculopontine nucleus. Dopaminergic (DA) neurons of
section through the macaque brain. The basic loop circuit the substantia nigra pars compacta (SNc) innervate the stria-
includes an excitatory glutamatergic (Glu) projection from tum and, less densely, the GP and STN. (C) Anatomically
the neocortex to the striatum (caudate nucleus and putamen) segregated parallel circuits through the BG subserve skeleto-
and then inhibitory (γ-amino butyric acid-containing; motor, oculomotor, associative, and limbic functions. All cir-
GABAergic) striatal projection (the “direct pathway”) to the cuits obey a common internal organization. Distant yet
internal globus pallidus (GPi). GABAergic neurons in GPi functionally related regions of cortex send converging projec-
project to targets in the thalamus and brainstem. The main tions to a subregion of the striatum. Medium spiny neurons
thalamic target of this basic circuit (VA/VL, ventro-anterior/ in each striatal subregion project to distinct subregions of the
ventrolateral nuclei of the thalamus) projects to the frontal globus pallidus, which in turn project to different regions of
cortex, including parts of the premotor and primary motor thalamus. (Only direct pathway structures are shown.) BG-
cortex. (B) The internal connectivity of the BG (inside the receiving subregions of thalamus project back to a subset of
dotted line). Direct and indirect pathways start in striatal the cortical areas that project into each circuit. Descending
projection neurons that express D1- and D2-type dopamine projections from the GPi terminate in the pedunculopontine
receptors, respectively. D2-type neurons project to the exter- nucleus (PPN) or superior colliculus (sup. coll). The target
nal globus pallidus (GPe). GPe projects to the subthalamic of descending projections from the limbic circuit is not estab-
nucleus (STN) and GPi. STN also receives monosynaptic Glu lished. Abbreviations: DL, dorsolateral; lat., lateral; med.,
input from the motor cortex and projects to GPi and GPe. medial.
GPi sends GABAergic projections to VA/VL and the center
Figure 37.2 Abnormal discharge of neurons in GPi is a criti- Huntington’s disease, D2-type indirect pathway neurons
cal step in the network imbalances associated with Parkinson’s degenerate selectively (dotted outline, left). The net effect on
disease (B) and Huntington’s disease (C). In a normally func- GPi activity is decreased firing rates and rhythmic bursty firing
tioning BG (A), balanced activation of direct and indirect patterns (right). Diagrams on the left indicate the level of
pathways (starting in striatal D1 and D2 neurons, left) leads tonic activity in a pathway by the thickness of lines. Traces on
to intermediate GPi firing rates and random discharge pat- the right show brief (∼2 second) epochs of action potential
terns (right). (B) In Parkinson’s disease, the dopaminergic discharges (i.e., “spike trains”) recorded from three actual
innervation of the striatum, pallidum, and STN degenerates GPi neurons under each clinical condition. The spike train
(dotted outline, left). The best understood consequence of in A was recorded from the GPi of a normal macaque monkey.
this is increased activation of D2-type indirect pathway neurons Single-unit spike trains in B and C were recorded from human
and decreased activation of D1-type direct pathway neurons. subjects undergoing surgery for deep brain stimulation. At
The effects of dopamine denervation on pallidum and STN the time scale used, individual action potentials appear as thin
activity are less well understood. The cascading effects result vertical lines of similar height above the level of background
in abnormal GPi activity marked by increased firing rates and noise (thick horizontal bands).
rhythmic bursty firing patterns (right). C. In early-stage
GPe
STN
100 mV
GPi
1s
~70 spikes/sec resting discharge rate
SNc Random discharge pattern
Brainstem
B Parkinson’s disease
Cortex
GPi neuronal discharge—Parkinson’s disease
GPe
100 mV
STN
GPi
1s
Increased firing rates, burstiness, oscillations,
and synchronized firing
SNc
Brainstem
C Huntington’s disease
Cortex
GPi neuronal discharge—Huntington’s disease
GPe
STN
100 mV
GPi
1s
Decreased firing rates, burstiness and oscillations
SNc
Brainstem
mm
–50
4 1
0
4001 2 3 4
50 3 2
Post-injection
–50
4 1
0
–50 50 0 1 2 3
mm Seconds
Pre Post
C
300 90 Spatial
Colour
9
80
200
70
100 8
60
0
50 7
Sequence Random Sequence Random Sequence Random
Figure 37.3 Inactivation of the BG skeletomotor circuit sequence, even though movement extent and velocity were
does not impair movement initiation or sequencing in a neu- reduced (“post-injection”). (C) Inactivation in the skeletomo-
rologically normal animal, but has a selective effect on move- tor region of GPi had a negligible effect on reaction times
ment speed and extent. (A) Single behavioral trials illustrate (RTs); left, compare pre-injection (open symbols) versus post-
performance of a sequential movement task before and after injection means (filled symbols). This was true irrespective of
an injection of muscimol (a long-acting GABAergic inhibitory whether animals performed overlearned fixed sequences
agent) into the GPi. Animals moved a joystick through a series (“Sequence”) or random sequences (“Random”), or whether
of four out-and-back component movements. Targets were cues provided information by their spatial location (circles)
presented in fixed order (1→4). Spatial trajectories (A) and or color (triangles). In contrast, muscimol injections consis-
tangential velocities (B) of the joystick are plotted. Outward tently reduced movement velocity (middle) and extent (right)
movements to capture a peripheral target often began before under all conditions. Error bars plus or minus standard error
the instruction cue was presented (↓). GPi inactivation did of mean.
not impair the smooth uninterrupted execution of the
choosing the correct response when presented with a Another fruitful approach to disentangling BG func-
novel visual stimulus. In this situation, activity that tions is to determine what aspects of motor behavior
reflects the newly learned association appears earlier in are impaired following lesions or inactivations of differ-
the striatum than in cortical neurons across the course ent circuits. A handful of studies have interrupted GPi
of learning (Brasted & Wise, 2004; Pasupathy & Miller, activity temporarily by injection of a pharmacological
2005; Williams & Eskandar, 2006). Interestingly, task- inhibitory agent (i.e., muscimol, GABAA agonist) into
related activation of the BG appears to become less that structure (Desmurget & Turner, 2008; Inase,
common as a skill becomes well learned (Tang et al., Buford, & Anderson, 1996; Kato & Kimura, 1992). Inac-
2009). These results suggest that the BG contributes tivation of the skeletomotor region induces significant
selectively to early stages of skill learning (Ashby, Turner, impairments in the speed and extent of arm movement
& Horvitz, 2010; Jueptner, Frith, Brooks, Frackowiak, & accompanied by modest postural instability. Remark-
Passingham,1997). ably, acute disconnections of the motor circuit do not
ABSTRACT The cerebellum is a large subcortical brain struc- receives input from the lateral hemispheres of the cer-
ture that influences movement, sensation, and cognitive ebellar cortex and projects via the thalamus back to
behaviors through interactions with the cerebral cortex and contralateral premotor, prefrontal, and parietal neocor-
brainstem. Cerebellar damage does not abolish these behav-
iors, but instead reduces their accuracy and flexibility. It has
tical regions. More medially, the interposed (emboli-
been hypothesized that the cerebellum contributes to its form and globose) nuclei receive input from
various functions by providing predictions about future intermediate cerebellar cortex (paravermis) and project
sensory and motor states, and by implementing a fast error- to the spinal cord via the red nucleus and to primary
driven learning process, which calibrates well-trained behav- motor cortex via the thalamus. The most medial
iors to environmental changes.
nucleus, the fastigial nucleus, receives input from the
midline areas of cerebellar cortex (vermis) and projects
Cell types and physiology to neocortex and brainstem targets, influencing descend-
ing vestibulospinal and reticulospinal pathways.
The cerebellum consists of a highly folded cortical
sheet surrounding white matter and deep cerebellar Functional anatomy
nuclei. A remarkable feature of the cerebellum is its
three-layered cortex, which is homogeneous in both The cerebral cortex can be parcellated into different
cell type and arrangement (figure 38.1). regions based on cytoarchitectonic organization. In
The main source of input to the cerebellum is pro- contrast, the neural circuit of the cerebellar cortex is
vided by mossy fibers, which transmit information from remarkably invariant, even though the human cerebel-
the neocortex via the pontine nuclei, from brainstem lum contains many distinct functional modules. In the
structures including the vestibular and reticular nuclei, medial-lateral direction, it consists of a set of parasag-
or directly from the spinal cord via the spinocerebellar gital compartments, which can be revealed by labeling
tracts. Mossy fibers synapse on granule cells (figure the expression of aldolase C or zebrin (Sugihara &
38.1), which account for over half of the neurons in the Shinoda, 2004). In the anterior-posterior direction, a
human brain (Azevedo et al., 2009). Granule cells send series of horizontally running fissures divide the cere-
axons into the molecular layer, where they branch into bellum into a set of lobules (figure 38.2), labeled I-X
parallel fibers and form excitatory synapses onto Pur- based on Olof Larsell’s careful comparative work
kinje cells. Purkinje cells have flat but elaborate den- (Larsell & Jansen, 1972).
dritic trees, branching orthogonally to the direction of Lobules I–V form the anterior cerebellum and are
the parallel fibers. In this way, each Purkinje cell can mainly concerned with motor control. Lobules IV and
receive input from ∼174,000 granule cells. At rest, Pur- V have reciprocal connections with primary motor
kinje cells fire action potentials, known as simple spikes, cortex (Kelly & Strick, 2003) and a reliable somatotopic
at a rate of 40–70 Hz. Climbing fibers provide more organization (Grodd, Hulsmann, Lotze, Wildgruber, &
sparse input to the cerebellum. Each Purkinje cell Erb, 2001). The feet are represented in superior and
receives input from a single climbing fiber, while each medial aspects of lobule IV, and the arm and hand in
climbing fiber branches to innervate ∼10 Purkinje cells. the hemispheric part of lobule V. The latter also con-
Climbing fibers cause Purkinje cells to fire powerful tains representations of single finger movements (Wies-
action potentials with complex waveforms, termed tler, McGonigle, & Diedrichsen, 2011). Upper-limb
complex spikes, at the low rate of 1–4 Hz. representations are strongly lateralized. Hand move-
Purkinje cells inhibit cells in the deep cerebellar ments activate the ipsilateral anterior lobe almost exclu-
nuclei, which provide the cerebellum’s only output. sively. The representation of orofacial movements
Pauses in the firing of specific sets of Purkinje cells extends from lobule V into the neighboring lobule VI.
release inhibition of the deep cerebellar nuclei cells, Complex limb movements preferentially activate
which in turn emit an excitatory burst of action poten- lobule VI (Schlerf, Verstynen, Ivry, & Spencer, 2010).
tials. The largest of these nuclei, the dentate nucleus, In contrast to lobule V, lobule VI generally shows
Basket
cell (–)
Stellate
cell (–)
Climbing
Purkinje fiber (+)
cell (–)
Granule
cell (+)
Golgi Inferior
cell (–) olive
Mossy
neuron (+)
fiber (+) Deep
cerebellar
nucleus (+)
Figure 38.1 Main anatomical layout of the cerebellar cortex indicates whether the cell gives rise to excitatory (+) or inhibi-
displaying major cell types and connections. The sign tory (–) connections.
bilateral activation for unilateral hand movements. This 2011). Functionally, various language, working memory,
suggests that it acts in concert with cortical secondary and executive function tasks activate lobule VII (Stood-
motor areas such as premotor and supplementary ley & Schmahmann, 2009).
motor area. Indeed, functional connectivity studies Lobules VIIIa and VIIIb are part of a second motor
show high correlations between activity in lobule VI and cortical–cerebellar loop (Kelly & Strick, 2003). As with
activity in the contralateral premotor cortex (Buckner, the anterior motor representation, these lobules have a
Krienen, Castellanos, Diaz, & Yeo, 2011). Language convergent representation of movement and sensory
tasks, including verb generation, also activate right information from the whole body. While there is dis-
lobule VI (and crus I of lobule VII), even when motor cernable somatotopy in these regions, it is weaker than
output-related activity is controlled for. Left lobule VI that in the anterior lobe.
may be associated with spatial tasks (Stoodley & Schmah- The hemispheres of lobule IX connects to the ante-
mann, 2009). The vermis of lobules VI and VII controls rior cingulate and precuneus (Buckner et al., 2011).
eye movements and is known as the oculomotor vermis The vermal aspect of lobule IX appears to be involved
(Prsa & Thier, 2011). in the regulation of the autonomic nervous system, for
Lobule VII is the largest lobule of the human cerebel- example, in cardiovascular control. Finally, the rela-
lum, accounting for roughly half the cerebellar gray tively small lobule X (flocculus + nodulus) is involved
matter volume (Diedrichsen, Balsters, Flavell, Cussans, in vestibular functions and eye movement control.
& Ramnani, 2009). It is subdivided into lobule VIIa Together, the cerebellar cortex comprises a patch-
(consisting of crus I and II) and lobule VIIb. In the work of areas, each with its own pattern of connectivity
monkey, the hemispheres of crus II have reciprocal and functional specialization. The information-process-
connections with Brodman area 46 (Kelly & Strick, ing role that each cerebellar region fulfills within its
2003), and human functional connectivity data indi- respective loop remains unclear. However, given the
cate that lobule VII participates in at least three subnet- homogenous functional architecture of the cerebellar
works including dorsolateral prefrontal, inferior cortex, many researchers agree that a single computa-
parietal, and lateral temporal areas (Buckner et al., tional function should characterize the role of the
I-IV Foot
Motor
V
functions
VI
V Hand
Crus I
Mouth
VI
Language
Crus II
IX VIII
VIIb Spatial Language
functions
Eye Working
VIIa crus I memory
Executive
VIIa crus II functions
Executive
VIIb VIII functions
IX
VIIIa Autonomic X Motor
nervous system Vestibular+ functions
eye movement
VIIIb
IX
X
Hemisphere Vermis Hemisphere
Figure 38.2 Functional anatomy of the human cerebellum numerals indicate lobules following Larsell’s notation; italics
shown on a flattened representation of the cerebellar cortex indicate likely involvement in behavioral functions. Gray
(based on Van Essen, 2002). Lobule surface area is roughly shading indicates sensory-motor involvement and connectiv-
proportional to real size. The upper inset shows dorsal and ity to motor regions.
ventral views of a cerebellar 3D reconstruction. Roman
cerebellum across motor and nonmotor domains (Ivry, ataxic reaching movements, hand paths are highly
Spencer, Zelaznik, & Diedrichsen, 2002; Schmahmann, curved and tend to be quite variable from reach to
2004). However, what constitutes this “universal cere- reach (figure 38.3A). An underlying inability to
bellar transform” is a much-debated matter. account for complex limb mechanics may cause this
abnormal curvature (Bastian, Martin, Keating, &
Thach, 1996). Consistent with this idea, patients show
Clinical aspects greater deficits when they reach quickly and move
many joints or body parts simultaneously. Patients com-
Neurological diseases, tumors, malformations, and pensate by moving more slowly and breaking move-
stroke can all damage the cerebellum. Loss of move- ments down into simpler components, a phenomenon
ment coordination or “ataxia” is the most common and originally described by Gordon Holmes as decomposi-
obvious deficit. Because the cerebellum projects to con- tion of movement (Holmes, 1939).
tralateral neocortical areas, right cerebellar damage One common element of ataxia is dysmetria, poor
affects right-sided movements. Recovery of motor func- control over the extent of movement. People with
tion often occurs, but is generally worse if deep cerebel- cerebellar damage often overshoot (hypermetria) or
lar nuclei are affected. In a recent study, children and undershoot (hypometria) targets. As a result, their
adults recovered better from surgical tumor removal movements show oscillations when approaching the
when deep cerebellar nuclei were spared (Konczak, target, a phenomenon known as intention tremor.
Schoch, Dimitrova, Gizewski, & Timmann, 2005). Intention tremor likely represents a series of corrective
Although ataxia is often used to describe specific movements, and may stem from the use of time-delayed
deficits (e.g., arm ataxia during reaching, or gait ataxia sensory feedback (see below). Consistent with this idea,
during walking), it is a global term that refers to spe- intention tremor is reduced or absent when patients
cific features of dyscoordination. For example, during point to a target with their eyes closed, reducing visually
A B C Control
Goal
Velocity Dentate
Controller cooling
(inverse model)
Target
State
estimate
Firing rate during flexion
Efference
copy
Forward
model
Body
0 ms 500 ms
Figure 38.3 The cerebellum as a predictive forward model. command’s influence on body state, thereby avoiding prob-
(A) Hand paths made by a cerebellar patient reaching toward lems associated with delayed sensory feedback. (C) Velocity
a target. Note features of ataxia, including dysmetria and end- and firing rate of a motor cortical neuron during typical
point oscillations. (B) A controller generates a motor movement (control; dashed line) and when cerebellar input
command based on a behavioral goal and an estimate of is blocked through dentate cooling (solid line). (Data redrawn
the body’s state. It sends an efference copy of the command from Hore and Flament, 1988.)
to the forward model (cerebellum), which predicts the
A B
Perturbation
Leftwards
Error
Aftereffect
Error
Rightwards
Trial Trial
C D
20 Catch trials
Initial
Movement error [deg]
Force error
10
After- 0
Controls
effect
–10
Cerebellar
patients
–20
Figure 38.4 The role of cerebellum in error-based learning. direction of movement. After an initial error, the movement
(A) Traditional adaptation experiment in which a perturba- adapts, resuming a nearly straight trajectory (gray line). When
tion suddenly occurs, leading to an exponential learning the force is suddenly removed, an aftereffect in the opposite
curve. Upon perturbation removal, an aftereffect is observed. direction occurs. (D) Deficit of cerebellar patients in adapting
(B) A key feature of error-based learning is that it occurs even to force fields. The force field is switched on (gray box),
during random perturbations, which induce small but mea- causing a large initial error. Healthy individuals show increas-
surable changes in behavior. (C) During a force-field reaching ing aftereffects on trials with no force field, but patients with
task, a robotic device exerts a force perpendicular to the cerebellar degeneration do not (Smith & Shadmehr, 2005).
Figure 38.5 Cellular mechanisms underlying cerebellar gray triangle shows the occurrence of a complex spike (CS)
learning. (A) In the task, a monkey fixated a moving on trial n. (C) Relationship between CS probability and
target that changed direction 250 ms after movement onset. change in SS rate. (Based on Medina and Lisberger, 2008.)
(B) Trial-to-trial change in the SS rate of a Purkinje cell. The
this SS decrease preceded the CS and hence the moment elsewhere. Clearly, some well-learned predictive motor
of directional change. Thus, the Purkinje cell is predict- mechanisms are preserved in cerebellar patients. Fur-
ing the error. The decrease in firing rate related to an thermore, recent evidence using transcranial direct
anticipatory rightward movement of the eye and current stimulation (Galea, Vazquez, Pasricha, Orban
increased tracking performance accuracy. To prevent SS de Xivry, & Celnik, 2011) suggests that the cerebellum
firing rates from decreasing indefinitely, this system also may produce short-term modifications of forward
requires a counteracting (i.e., potentiating) learning models, whereas the neocortex may store longer-lasting
mechanism. This can be seen in the potentiating effect motor memories.
of trials in which CS activity does not change from base- At the whole-brain level, it has become clear that the
line (figure 38.5C). These results provide strong evi- cerebellum is not simply a motor control device, but
dence in support of the Marr and Albus hypothesis that has functional involvement in a wide array of mental
the climbing fiber provides an error signal. However, the abilities. How the cerebellum contributes to these non-
results also point to the operation of other learning motor functions remains unclear. We hope that the
mechanisms within the cerebellar circuitry. next generation will fulfill the dream of a universal
theory that explains cerebellar function from cellular
to behavioral levels, as envisioned by David Marr more
Concluding remarks and open questions
than 40 years ago.
Many open questions about cerebellar function remain. ACKNOWLEDGMENTS The authors thank Dr. Erin Heerey and
Although the Marr-Albus model of cerebellar learning Prof. Richard Ivry for comments on an earlier draft. Thanks
provides an elegant account of many experimental find- to Prof. Daniel Wolpert for assistance with figure 38.1.
ings, there is also substantial evidence opposing this
theory. For example, the low-frequency CS activity
(1–5 Hz) may not be high enough to provide the tem- REFERENCES
poral resolution necessary for error-based adaptation of Albus, J. S. (1971). A theory of cerebellar function. Math
finely controlled motor behaviors (Horn, Pong, & Biosci, 10, 25–61.
Gibson, 2004; but see Kitazawa, Kimura, & Yin, 1998). Azevedo, F. A., Carvalho, L. R., Grinberg, L. T., Farfel, J.
M., Ferretti, R. E., Leite, R. E., … Herculano-Houzel,
Furthermore, during saccade adaptation the pattern of
S. (2009). Equal numbers of neuronal and nonneuronal
CS activity increases with learning, consistent with a role cells make the human brain an isometrically scaled-up
in adaptive output production, rather than decreasing primate brain. J Comp Neurol, 513(5), 532–541.
as would be predicted if it encoded an error signal Bastian, A. J., Martin, T. A., Keating, J. G., & Thach, W. T.
(Catz, Dicke, & Thier, 2005; but see Soetedjo, Kojima, (1996). Cerebellar ataxia: Abnormal control of interaction
& Fuchs, 2008). torques across multiple joints. J Neurophysiol, 76(1),
492–509.
On a systems level, it remains unclear whether the Bhanpuri, N. H., Okamura, A. M., & Bastian, A. J. (2012).
cerebellum implements predictive forward models or Active force perception depends on cerebellar function.
whether it merely adapts forward models stored J Neurophysiol, 107(6), 1612–1620.
ABSTRACT The apparent ease of skilled motor behavior masks particular because they may provide a window into the
the complex neural processes involved in the control of voluntary control of movement. In this chapter, we
movement. To handle the complexity of most motor tasks, introduce the basic notions underlying optimal control
optimal feedback control suggests that the brain continu-
ously processes sensory feedback and selectively compensates theory and discuss how this approach may help us iden-
for noise disturbances as well as external perturbations. Moti- tify problems that the brain must solve to perform even
vated by this powerful prediction, recent studies have used the simplest movements. Then, we review recent find-
perturbation paradigms to investigate the neural control of ings that emphasize the similarity between biological
movement. After introducing the basic mathematical con- control and the sophistication of optimal control
cepts of optimal feedback control, we review evidence that
the brain generates flexible feedback strategies following
models. In particular, we focus on perturbation para-
perturbations perceived from visual feedback (e.g., a sudden digms showing that many aspects of optimal control
change in the target location), as well as mechanical pertur- models are observed in corrective responses generated
bations applied to the limb. Importantly, we highlight evi- by humans: (1) continuous processing of sensory feed-
dence that the motor system can generate goal-directed back underlies the control of movement, and (2) feed-
responses in as little as 50–60 ms following a mechanical per-
back control is tailored to the constraints imposed by
turbation. A transcortical feedback pathway through primary
motor cortex appears to play an important role in these the task at hand. We finish our chapter by briefly dis-
rapid corrective responses. cussing how sophisticated motor behavior may be
linked to processing in distributed brain circuits, high-
Elite athletes push the limits of the sensorimotor system, lighting recent evidence that neural processing in
integrating sensory information to generate rapid yet primary motor cortex (M1) may possess some of the
remarkably precise movements. A good example is a attributes required for flexible feedback control.
hockey player breaking away from his defenders. In a
split second, the player has to read the goaltender’s
Optimal control: Definitions and applications
movement and decide whether to shoot the puck or
in neuroscience
fake out the goaltender and wait for an opening. Even
the simplest movements that we perform in daily life, Flash and Hogan (1985) introduced optimal control
such as reaching for a cup of coffee, also involve principles to movement neuroscience almost 30 years
complex sensorimotor coordination. The ability to use ago. Intuitively, this approach is based on the assump-
sensory information to flexibly guide, correct, or modify tion that the brain selects motor commands that maxi-
our actions is the hallmark of skilled biological control. mize or minimize a performance criterion. In the
Recently, optimal feedback control (OFC) has been context of reaching movements, Flash and Hogan sug-
used as a model of how the brain processes sensory gested that the brain selects motor plans that minimize
information to control movement. A powerful feature the derivative of the hand’s acceleration (or the jerk).
of this model is that it describes how the motor system This hypothesis was justified by the fact that point-to-
should handle performance errors caused by neural point movements tend to be smooth with a bell-shaped
variability or environmental disturbances. This feature velocity profile, which is reproduced nicely by minimiz-
has renewed interest in feedback response strategies, in ing the derivative of the hand’s acceleration during
reaching movements. Following Flash and Hogan’s
work, a wealth of studies have proposed biological cost
* These authors contributed equally to this work. functions that incorporate kinetic parameters (e.g.,
crevecoeur, cluff and scott: goal-directed movement planning and execution 461
minimum torque change; Nakano et al., 1999; Uno, Deterministic Optimal Control In general, the
Kawato, & Suzuki, 1989) and energy consumption solution of a particular control problem is not unique.
(Biess, Liebermann, & Flash, 2007; Berret, Chiovetto, This also applies to biological motor control, where
Nori, & Pozzo, 2011) to identify movement variables reaching movements may follow distinct paths to the
that the motor system may control and optimize. same target or even have different velocities along the
This approach is grounded in the theory of optimal same movement path. Given that each of these move-
control, a formalism that applies optimization princi- ments satisfies the goal of reaching the target (i.e.,
ples to describe what the motor system should do motor equivalence), extensive research has been con-
according to how our limbs move, as well as the ducted to identify how the brain selects one control
performance criteria (e.g., energy consumption) and solution among infinitely many alternatives. One way to
constraints (e.g., time constraint) associated with move- reduce the set of possible movement solutions is to
ment. The following section outlines the basic notions constrain the problem using a cost function. For
of control theory and defines (optimal) control prob- instance, we may be interested in applying a sequence
lems that are often encountered in movement of joint torques that allows us to reach a target while
neuroscience. minimizing the intensity of muscle activity to avoid
fatigue. In this example, the cost is directly related to
Dynamical Systems and Control Problems At the the motor command, and the problem is to find the
basis of control engineering is the notion of a dynamical reaching path that minimizes muscle energy expendi-
system: a set of variables evolving as a function of time. ture. This approach uses optimization principles to
For instance, the angular motion of a body segment can determine the best way to reach the target among all
be seen as a dynamical system described by state vari- possible movement solutions.
ables, such as the joint angle and velocity. In general, A typical cost function contains a final cost, g(x), and
the evolving state of a dynamical system can be described a running cost, L(x,u), that accumulates along the tra-
by a differential equation of the form: jectory followed by the state variables:
x = f (x , u ), (1) tf
{ }
With these definitions, we can define a control
∂J (x , u ) ∂J (x , u )
problem as follows: find a time-varying control vector − = min L(x , u ) + f (x , u ) . (4)
that steers the state variables to a desired location. ∂t u ∈U ∂x
Assuming that the initial state x0 is known, the
The function J(x,u) is the cost to accumulate from the
problem is to find a control function u(t), t0 <= t <= tf,
present time until the end of the problem horizon, tf.
such that the solution of Eq. 1,
This quantity is called the cost-to-go and plays a central
t
role in the derivation of numerical solutions (Todorov,
x(t ) = x 0 + ∫ f (x(s ), u(s ))ds , (2) 2006). An intuitive interpretation of Eq. 4 is that the
t0
control vector can vary the orientation of the instanta-
meets the constraints of the control problem. A reach- neous direction of the state trajectory (f(x,u)), which
ing movement, for example, may be described as a should ideally follow the direction that is opposite to
control problem where the brain must steer the hand the gradient of the cost-to-go (∂J/∂x). However, the opti-
to the spatial location of a goal target. Expressing a mization also takes into account the instantaneous cost
simple task such as reaching as a control problem is a (L(x,u)), and as a result, Eq. 4 achieves the best com-
powerful approach to identify the challenges that motor promise between the gradient of the cost-to-go and the
control presents for the brain. As we will see, even the instantaneous cost (L(x,u)). This compromise deter-
simplest movements impose complex sensorimotor mines the instantaneous variation of the cost-to-go
transformations. (∂J/∂t).
Sensory
Optimal feedback
state
estimation Musculoskeletal system
Figure 39.1 Illustration of basic processes expected under musculoskeletal system). An efference copy of the descending
the OFC framework. The selection of the behavioral task motor command is used internally to predict the conse-
determines the feedback control policy (task selection). The quences of motor actions. These internal predictions are com-
purpose of the feedback control policy is to continuously bined with feedback from sensory receptors to compute the
process and convert sensory data into motor commands that posterior estimate of the state of the body (optimal state esti-
best satisfy the task demand (optimal feedback control policy). mator). This state estimate is used to adjust the motor com-
Once the sensory feedback is processed, motor commands are mands during the ongoing motor action.
sent to the peripheral motor system (biomechanical plant or
The many studies that have used this approach have disturbances alter performance. The control of stochas-
agreed on the general conclusion that healthy motor tic processes was introduced to address this limitation.
systems favor smooth and efficient movements. The In this framework, feedback is essential to update motor
shortcoming of this approach, however, is that because commands and compensate for neural variability.
the laws of physics relate all movement parameters, Online monitoring and control are often described in
virtually every meaningful cost function in the form of terms of a state estimator combined with a controller
Eq. 3 partially captures the smoothness and efficiency (figure 39.1). The state estimator typically combines
of biological motor control. Also, this approach does sensory and motor signals to compute the present state
not systematically account for the continuous update of of the body (figure 39.1; optimal state estimator), and
motor commands that is necessary to correct for motor the controller uses the estimated state of the body to
errors. select control actions that best reflect the goal and con-
straints of the task (figure 39.1; optimal feedback
Stochastic Optimal Control In general, factors control policy). This section presents the basic formal-
that can induce motor errors and trial-to-trial variability ism of the problems of estimation and control of sto-
fall into two broad categories. First, the variable activa- chastic processes.
tion of neural circuits induces variable motor behavior. Because random (Brownian) motion does not have
Neural noise can be found in sensory systems, move- finite instantaneous variations, the control problem is
ment preparatory activity, and the activation of muscles formulated in discrete rather than continuous time
in the motor periphery (Churchland, Afshar, & Shenoy, (Arnold, 1974). To begin, the analog of Eq. 1 becomes
2006; Faisal, Selen, & Wolpert, 2008; Osborne, Lis-
dX = F (X , u )dt + G(X , u )dW . (5)
berger, & Bialek, 2005; Scott & Loeb, 1994; van Beers,
Haggard, & Wolpert, 2004). Additionally, motor errors In Eq. 5, the capital X signifies that the state vector is
can be produced by external disturbances resulting now a stochastic variable. Eq. 5 expresses that small
from our interaction with the environment. Both neural changes in the state (dX) follow a deterministic law
variability and external disturbances require that the described by the function F that captures the system
brain continuously update motor commands based on the dynamics as in Eq. 1, and a stochastic term that captures
available sensory data to produce successful behavior. random disturbances in the process (dW). We now
Harris and Wolpert (1998) considered the influence convert this equation to discrete time by considering
of neural variability on movement planning and sug- changes in the process over a time step of δt. From the
gested that motor commands are selected to minimize definition of Brownian motion (Arnold, 1974), the
the variance of movement end points, assuming the accumulation of random noise over δt follows a Gauss-
intensity of motor noise scales with the size of the motor ian distribution with zero mean and variance equal to
command. While this model explicitly considers the δt. We use ξ(t) to designate these Gaussian disturbances
effect of motor noise, it does not address the online at each time step. The formulation in discrete time
adjustment of motor commands required when becomes
crevecoeur, cluff and scott: goal-directed movement planning and execution 463
X (t + δt ) = M (X (t ), u(t )) + N (X (t ), u(t ))ξ(t ), (6) optimal control and filtering problems is available. The
main assumptions are that the system follows linear
where M(.) and N(.) are the discrete versions of the dynamics, coupled with a quadratic cost function and
functions F and G introduced in Eq. 5. The analog of subject to additive Gaussian noise (linear quadratic
Eq. 4 also becomes a discrete equation. The optimiza- Gaussian, or LQG). The function M(.) introduced in
tion can no longer be computed over the true state Eq. 6 becomes a linear function of the state and control
variables, because independent simulations of the same vectors (represented by the state-space matrices A and
process will lead to distinct sample paths. Instead, for B), and the noise disturbance is captured by additive
stochastic control problems, the controller should opti- Gaussian noise (ξt). With these definitions, the system
mize the expected outcome, which ensures that perfor- dynamics become
mance will be optimal in the most likely scenario. The
analogue of Eq. 4 becomes X t +1 = AX t + But + ξt . (8)
The sensory feedback available at time t is of the form
J t (X t , ut ) = min {L(X t , ut ) + E [ J t +1(X t +1, ut +1 )| X t , ut ]} , (7)
u
Yt = HX t + ωt , (9)
where E[.] denotes the expected value of the argument
where ωt is Gaussian random variable and H is the
and the explicit dependency on time was replaced by
mapping between the system state and sensory data.
the subscript t. Eq. 7 means that the best control action
This matrix may express that information about some
minimizes the present running cost, as well as the
state variables is not available, such as an external per-
expected value of the future cost-to-go given the selected
turbation that can only be indirectly measured through
control action (figure 39.1; optimal feedback control
its effect on the motion of the body. The matrix H may
policy).
also express that some sensory signals provide informa-
A clear difficulty encountered with stochastic pro-
tion about a combination of state variables, as for
cesses is that the state of the system (Xt) may not be
instance muscle spindles provide sensory feedback that
known exactly, because sensory feedback is also cor-
is a mixture of the muscle’s length, velocity, and higher
rupted by noise. Filtering techniques can be used to
derivatives. The cost function is a quadratic form in the
estimate the state of a stochastic process (figure 39.1;
state and control variables, defined by weight matrices
optimal state estimator). These techniques combine
Qt and Rt, respectively:
imperfect sensory data with internal assumptions about
the state of the system (or internal priors). Prior assump- L(X t , ut ) = X tT Q t X t + utT Rt ut . (10)
tions are often formulated as the output of a forward The optimal control problem consists in finding a
model, making a prediction of the present state of the sequence of control variables that minimize the total
body given the available information, including the expected cost, that is,
motor command sent to the muscles (Wolpert & Flana-
gan, 2001). Let Yt denote the information available at ⎡N ⎤
J = E ⎢ ∑ L(X t , ut )⎥ . (11)
time t. We assume that the conditional distribution of ⎣ t =1 ⎦
Yt given the state variable Xt is known. First, we can
predict the distribution of Xt given the distribution of For this class of control problems, the optimal control
Xt-1 by using Eq. 6 (forward model). This prediction step policy turns out to be a linear function of the estimated
gives a prior belief about the system state. Second, the state, denoted x̂t :
feedback data, Yt, can be used to correct the prior dis- ut = Ct xˆt . (12)
tribution by applying Bayes’s theorem, which gives the The sequence of optimal feedback gains, Ct, is deter-
posterior distribution of the state (Xt) given the sensory mined by the cost matrices (Qt and Rt in Eq. 10) and by
feedback (Yt). In other words, the best estimate of the the system dynamics (A and B in Eq. 8). The result is a
system state is obtained by performing Bayesian integra- time-varying feedback gain that tells us how the motor
tion of sensory signals and forward predictions at each system should transform the estimated state of the
time step. The resulting estimate is optimal in the sense system into motor commands.
that the variance of the posterior distribution is mini- The optimal estimate of the state is calculated in two
mized. In the particular case of linear dynamics affected steps (Kalman filter). First, the prediction of the next
by additive Gaussian disturbances, this procedure is system state is obtained by simulating the dynamics over
known as a Kalman filter (Kalman, 1960). one time step given the current motor command, ut,
and taking the expected value of the outcome:
Linear-Quadratic-Gaussian Regulator Let us examine
the simplest case where an analytic solution of the xˆtp = Axˆt −1 + But −1. (13)
crevecoeur, cluff and scott: goal-directed movement planning and execution 465
internal representation of the body and environmental environmental dynamics can never be known exactly,
dynamics is available, while uncertainty is handled by robustness may be an important consideration in motor
considering the presence of random noise in the system. neuroscience.
However, uncertainty in the internal model of dynamics
can also alter the control of a movement. For instance, Application to biological control: Flexible
the inertia of the limb differs slightly according to the sensorimotor control strategies
weight of a watch, clothing, or hand-held objects. In
addition, muscle dynamics may vary depending on the The motor system has a remarkable ability to perform
level of background activity, biochemical factors, or successfully while never reproducing exactly the same
fatigue (Zahalak, 1981). This class of model distur- movement. This consistent success in the presence of
bances does not fall under those disturbances modeled variability suggests the central nervous system is well
by random noise, as they potentially introduce system- aware of the constraints of the task at hand and is less
atic biases during movement. concerned about errors that do not affect performance.
In general, researchers have approached this problem This tendency is often referred to as the minimum inter-
with learning or adaptation studies (Shadmehr, Smith, vention principle (Todorov & Jordan, 2002) and is cap-
& Krakauer, 2010; Wolpert, Diedrichsen, & Flanagan, tured by the ability to ignore limb deviations that do
2011). In this framework, changes in motor commands not interfere with task completion. The same idea is
reflect adaptive adjustments of how the brain repre- reflected in the notion of structured variability or an
sents environmental dynamics. While tremendous uncontrolled manifold, where limb and whole-body motion
progress has been made with this approach, a clear are more variable along dimensions that are irrelevant
shortcoming is that the body and environment can for the task (Balasubramaniam, Riley, & Turvey, 2000;
change more rapidly than the adaptation processes typi- Cluff et al., 2011; Scholz & Schoner, 1999; Valero-
cally investigated in motor learning studies (e.g., learn- Cuevas, Venkadesan, & Todorov, 2009).
ing curves varying over tens to hundreds of trials). For Optimal feedback control provides a framework for
instance, muscle dynamics rapidly change during effort us to understand task-related error corrections. Because
without giving us the chance to practice tens of trials to motor commands have a cost, there is no need to
adapt to those changes. control movement errors that do not interfere with the
Engineers have developed an approach based on the intended goal. This trade-off between behavioral per-
concept of robustness to deal with these internal model formance and motor costs, expressed in a straightfor-
uncertainties. The idea is to make the control design as ward way by the quadratic cost function in Eq. 10, is
insensitive to model errors as possible (Bhattacharyya, only possible if the brain continuously processes sensory
Chapellat, & Keel, 1995; Doyle, Francis, & Tannen- data to select control actions that are appropriate for
baum, 1992). This approach typically focuses on prop- the goal and constraints of the task. In agreement with
erties of the controller rather than on the actual system this principle, we review several studies emphasizing
trajectories emphasized in the classical optimal control that this type of flexible, task-dependent feedback
approach. An important theoretical result is that the control underlies both voluntary motor behavior and
controllers that are the most robust against model responses to external perturbations.
errors do not always correspond to the controllers that
are the most efficient (Boulet & Duan, 2007; Michiels Visuomotor Feedback Responses Liu and Todorov
& Niculescu, 2007). In other words, improving the (2007) provided compelling evidence for flexible bio-
robustness of control may degrade performance, logical control strategies by demonstrating that feed-
whereas optimizing a performance criterion can make back responses depend on the hand’s position when a
the control design more fragile to model errors. Com- goal target changes location during reaching. In this
patible with these principles, previous studies have sug- experiment, visual target perturbations were intro-
gested that motor performance is altered to maintain duced at the start (early) or near the end of a reaching
performance or preserve movement smoothness in movement (late; figure 39.1A) in a task where subjects
conditions of higher uncertainty (Crevecoeur, McIntyre, were instructed to stop at a peripheral target. When the
Thonnard, & Lefèvre, 2010; Ronsse, Thonnard, Lefèvre, target location was perturbed early in the movement,
& Sepulchre, 2008). However, to our knowledge, the the participants corrected their hand path smoothly
trade-off between the efficiency and robustness of (figure 39.1B). In contrast, hand path corrections were
biological motor control and its influence on motor incomplete when the same perturbation was introduced
planning and execution has not been thoroughly late in the movement (figure 39.1B). Liu and Todorov
investigated. Given that internal models of body and suggested that the dependency of the correction upon
crevecoeur, cluff and scott: goal-directed movement planning and execution 467
D
Task-relevant Task-irrelevant
A Normal feedback feedback
25
20
Position [cm]
15
35 cm
10
0
20
m -4 0 4 -4 0 4 -4 0 4
40 c
cm
Position [cm]
E
3 Pre-normal Task-relevant Post-normal
B feedback
10 Target
Lateral position [cm]
Hit 0
0
0
0 200 400 600 800 Early Late 1 14 1 14 1 14
Time [ms] Perturbation Trial blocks
F 3 Pre-normal Task-irrelevant Post-normal
C
feedback
10 Target
Lateral position [cm]
2
Undershoot [cm]
1
5 Stop 1
Hit 0
0
0
0 200 400 600 800 Early Late 1 14 1 14 1 14
Time [ms] Perturbation Trial blocks
Figure 39.2 Visuomotor responses account for the goal of are relevant to the ongoing task. In the normal condition, the
the ongoing task. (A) Subjects were instructed to reach and hand feedback cursor reproduced the hand trajectory. In the
stop at a target (“stop” condition) that either stayed in the task-relevant feedback condition (orange traces), the visual
central location or jumped in the lateral direction after the cursor moved away from the hand trajectory and remained at
start of the movement. Data are the population average tra- this point for the rest of the movement. In contrast, in the
jectories when participants were instructed to stop at the task-irrelevant feedback condition (blue traces), the hand
target. Color code: black, baseline; red, early perturbation; feedback cursor moved away from the hand trajectory but
blue, late perturbation. (B) Hand path deviation in the direc- returned to the true hand position by the end of the move-
tion of the displaced target. Note that subjects were unable to ment. (E) Time course of adaptation to task-relevant feedback
compensate when the target jumped late in the movement, perturbations. Data are the mean (solid line) and standard
demonstrating that feedback responses depend on the task deviation (shaded region) force difference across subjects
constraints. Color scheme is the same as in A. Dashed lines, between right and left visual perturbations (180–230 ms after
“hit” condition; solid lines, “stop” condition. (C) Results from cursor jump). Note that subjects produced larger corrections
target intercept experiment. In this experiment, the target when the cursor displacements were relevant to the ongoing
jumped laterally and then moved downward at a fixed rate. task. (F) Time course of adaptation to task-irrelevant feedback
The subjects were instructed to “stop” or “hit” the target perturbations. Note that subjects did not adapt when the
before it stopped moving. (Adapted from Liu & Todorov, perturbations were irrelevant to the ongoing task. (Adapted
2007.) (D) Visuomotor responses only correct for errors that from Franklin & Wolpert, 2008.) (See color plate 35.)
ΔEMG [au]
Actual Cursor
position
y
Unperturbed 0 Hand X position (cm) Mirror
x reaching –50 0 50 100 150 D E 10 R1 R2 R3
2 cm Matching
Left hand Right hand
Perturbed Time [ms] 8
–100 –10 –5 0 5 10
reaching
EMG [au]
6
B 0 4
3 2
R1 R2 R3
100 0
ΔEMG [au]
Time [ms]
−50 0 50 100 150 200 250
200
3
ΔEMG [au]
2
300
0 1
–50 0 50 100 150
Time [ms] 400 0
−1
500
−50 0 50 100 150 200 250
Time [ms]
Figure 39.3 Corrective responses to mechanical perturba- perturbation. (Adapted from Nashed et al., 2012.) (C) Cor-
tions applied during reaching and postural control. (A) Dif- rective responses in a bimanual postural control task. In the
ferences in corrective responses while reaching to a circle mirror perturbation condition (left panel), the hands are
target (left panel) or rectangular bar (middle panel). Black perturbed in opposite directions (black arrows). In the match-
dotted lines are unperturbed reaching movements to each ing perturbations condition (right panel), the hands are
target shape. Extensor perturbations were applied at the perturbed in the same direction (gray arrows), causing dis-
shoulder and elbow on random trials to elicit elbow motion placement of the feedback cursor that is displayed at the
just after movement onset. Corrective responses were rapidly spatial average position of the two hands. (D) Lateral hand
directed back to the circular target (black solid lines), but in path kinematics. Thin lines correspond to individual subject
contrast, were directed to new locations on the rectangular data; thick lines denote the population average for each
bar (gray lines). Note that differences between corrections condition. Color scheme is the same as in C. Shaded box is
can be observed in brachioradialis stretch responses (popula- the acceptable target region. (E) Pectoralis major stretch
tion data, elbow flexor) within ∼60 ms of the perturbation (R2 responses in the mirror (black lines) and matching conditions
and R3; right panel). Vertical lines denote perturbation onset, (gray lines). Data are the population average response, and
and dashed lines denote separation of the different phases of shaded region corresponds to the SEM. Bottom panel is the
the stretch response. (B) Differences in corrective responses difference between stretch responses in the matching and
while reaching to a rectangular bar occluded by environmen- mirror conditions (Matching – Mirror). Dotted line denotes
tal objects (left panel) and an unoccluded rectangular bar average difference across participants, shaded region is the
(middle panel). Corrective responses directed the hand SEM. Note that differences in muscle stretch responses
between the objects to the target (black solid lines), but emerge ∼75 ms after the perturbation, demonstrating that
directed the hand to new locations on the unoccluded rect- right limb responses depend on feedback about the left arm’s
angular bar (gray lines). Note that the stretch response motion. (Adapted from Omrani et al., 2013.)
begins to reflect the target shape ∼60 ms after the
colleagues (2012) recently investigated whether long- redirected their hand to new locations on the bar.
latency responses selectively correct for task-relevant Similar context-dependent responses were evoked
errors while subjects made reaching movements to a when obstacles in the environment required that the
circular target or rectangular bar oriented perpendicu- participants navigate to the target through a narrow
lar to the reach (figure 39.3A). On certain trials, a channel (figure 39.3B). These behavioral results were
mechanical perturbation was applied to displace the reproduced by an optimal feedback control model with
hand in the lateral direction. When the perturbation differing sensitivity to lateral hand errors. Further, the
pushed the hand away from the circular target, the model predicted that the shape of the goal target should
participants performed rapid corrective responses to influence the feedback response as early as sensory
direct their hand back to the target. In contrast, when feedback about the perturbation became available. In
the same perturbation was applied while subjects agreement with these model predictions, differences in
reached to the rectangular bar, the participants muscle responses between tasks were observed in as
crevecoeur, cluff and scott: goal-directed movement planning and execution 469
little as 70 ms following the perturbation, establishing Studies have since confirmed that sensory informa-
that rapid motor corrections (i.e., long-latency responses) tion from one limb in both reaching (Mutha & Sain-
integrate muscle stretch information with knowledge burg, 2009) and posture (Dimitriou, Franklin, &
about the behavioral goal and spatial features of the Wolpert, 2012; Omrani, Diedrichsen, & Scott, 2013)
environment (Nashed et al., 2012). rapidly modifies corrective responses in the other limb
during the long-latency time window. When the two
Flexible Bimanual Feedback Control In the hands control independent cursors and are perturbed
context of sudden mechanical perturbations, Marsden in opposite directions, robust long-latency responses
and colleagues (Marsden, Merton, & Morton, 1981) are observed in both arms to counter the perturbation
first outlined the task dependency of interlimb responses, (Omrani et al., 2013). In contrast, long-latency stretch
noting that long-latency muscle stretch responses in the responses are substantially smaller when the two hands
right arm after left arm perturbations reflected the task are perturbed in opposite directions while controlling
the right arm was performing. If the right arm held a a single cursor displayed at the spatial average position
table for support, the extensors were activated to stabi- of the two hands. In this context, corrective responses
lize the participant following the left arm perturbation. are unnecessary because the position of the single feed-
Remarkably, subjects even reversed their corrective back cursor is not disturbed by the perturbation. When
responses and activated the flexor muscles of the right the direction of the left arm perturbation was not pre-
arm if they had to stabilize a cup of tea. Perhaps the dictable, stretch responses in the right arm depended
most powerful example of how the task modulates on proprioceptive input from the left arm in the one-
interlimb feedback responses is that muscle responses cursor condition, and were larger if both arms were
were absent if the subject grasped a loose handle and perturbed in the same direction (figure 39.3C, D, and
benefitted little from right arm responses. The coordi- E). These differential feedback responses were not
nated responses observed in the muscles of the unper- observed when each hand controlled its own indepen-
turbed arm clearly emphasize that online feedback dent cursor. Why should the brain distribute corrective
control is not hard-wired but can be engaged at will, responses when several effectors are involved in the
depending on the context and on the intended task? Optimal feedback control predicts this behavioral
behavior. pattern, since dividing the response across effectors
Bimanual control therefore provides a remarkable reduces the effort and variability of motor corrections
tool to address how sophisticated feedback control can (Diedrichsen & Dowling, 2009).
be distributed across different body parts. Motivated by
the capacity of OFC models to exploit many different Internal Models of Multijoint Dynamics We
ways to attain the same movement goal (i.e., task redun- have so far focused on evidence that the motor system
dancy), recent studies have examined feedback correc- continuously processes sensory data to generate task-
tions in bimanual tasks by comparing motor responses dependent feedback responses. It is important to
when the two arms act independently or are coupled recognize, however, that task-dependent feedback
by the task demand. A compelling example of this flex- responses can only be achieved if the motor system has
ibility is when one hand is gradually perturbed in a task knowledge of how the body should move in response
that requires bimanual reaching movements (Diedrich- to external forces that arise from our environmental
sen, 2007). If each hand controls its own cursor while interactions or forces generated by muscles. An impor-
reaching to separate targets (two-cursor task), only the tant feature of body dynamics is the presence of interac-
perturbed hand shows a corrective response. However, tion torques between joints that require coordinated
when the two arms control a single cursor (one-cursor multijoint responses to control the motion at each
task), displayed as the spatial average position of the joint. Extensive evidence has shown that the voluntary
two hands, perturbations applied to one hand elicit motor system compensates for these interaction torques
bilateral responses to correct the cursor’s trajectory to produce straight reaching movements (Gribble &
(Diedrichsen, 2007). These flexible responses were Ostry, 1999; Hollerbach & Flash, 1982), but an interest-
reproduced by expressing the task constraints (Qt in Eq. ing question is whether corrective responses also reflect
7) for each hand independently, or as the average of knowledge of limb mechanics.
the two hands. The flexibility of bilateral corrections is Previous work emphasized that mechanical perturba-
not restricted to changes in the size of the response, as tions evoked rapid responses in muscles that are not
even the direction of these coordinated responses can directly stretched by the perturbation, suggesting that
be reversed if required by the task (Diedrichsen & coordinated motor responses occur in the long-latency
Gush, 2009). time window (Gielen, Ramaekers, & van Zuylen, 1988;
4 50
Figure 39.4 Long-latency responses express knowledge of although there was no change in the length of the posterior
multijoint limb mechanics. (A) Subjects were instructed to deltoid muscle, there is still a robust long-latency response
maintain their hand at a central target, and step-torque per- (excitatory and inhibitory). Data are the population-level
turbations were applied to the shoulder and elbow joints muscle response, and shaded region corresponds to SEM.
(flexor torques at both joints, dark gray shading and denoted Same color scheme as in A and B. (Adapted from Kurtzer et
by (F), extensor torques at both joints, light gray shading and al., 2008.) (D) Population-level response of shoulder-like M1
denoted by (E)). Stretch responses were recorded from the neurons. Shoulder-like neurons respond to the underlying
posterior deltoid muscle (PD) (B) Limb configuration and shoulder torque even though local information from the
applied multijoint torques were selected to cause substantial shoulder is ambiguous about the underlying torque. Note that
elbow motion (dashed lines) but minimal shoulder motion the response to the underlying torque begins about 50 ms
(solid lines). (C) Posterior deltoid (shoulder extensor) after the onset of the shoulder and elbow perturbations.
muscle activity aligned on perturbation onset. Note that (Adapted from Pruszynski et al., 2012.)
Soechting & Lacquaniti, 1988). The question of whether shoulder and elbow joints onto a response that is appro-
these responses relate to limb dynamics was recently priate for the actual underlying torque rather than the
addressed by applying different combinations of pertur- observed motion pattern.
bations to the shoulder and elbow. In one experiment, It is important to emphasize that OFC as a theory of
multijoint loads applied to the shoulder and elbow did motor behavior can only tell us what the optimal motor
not produce motion at the shoulder but led to either solution should look like. The studies presented above
flexion or extension motion at the elbow (Kurtzer, highlight feedback responses that possess an impressive
Pruszynski, & Scott, 2008, 2009; figure 39.4A, B, and C). degree of flexibility and can be modified to suit the
The authors found that there was no short-latency needs of many behavioral tasks. It has been demon-
muscle stretch response in the posterior deltoid, a strated that long-latency stretch responses are modified
shoulder extensor, highlighting that this spinal reflex is in different dynamic environments (Ahmadi-Pajouh,
not elicited without overt motion at the joint (figure Towhidkhah, & Shadmehr, 2012; Kimura & Gomi, 2009;
39.4B, C). In contrast, the long-latency response Krutky, Ravichandran, Trumbower, & Perreault, 2010),
(50–105 ms post-perturbation) integrated motion infor- and a direct prediction is that corrective responses
mation from both joints and generated a large response should be modulated by the novel dynamical context.
in the posterior deltoid to counter the shoulder flexor Changes in corrective responses to a gradual perturba-
torque, even though there was no motion at the joint tion have been observed over longer time scales
(figure 39.4C). These results clearly showed that (Wagner & Smith, 2008). A recent study shows that
rapid motor corrections map the sensed motion of the long-latency responses also express knowledge of
crevecoeur, cluff and scott: goal-directed movement planning and execution 471
internal models acquired during motor learning (Cluff responses contribute to volitional control of the limb
& Scott, 2013). This result emphasizes that adaptive during the long-latency time window. This idea was
changes to novel dynamics alter voluntary behavior and recently tested in a study examining whether the trans-
rapid feedback responses. cortical feedback pathway through M1 exhibits knowl-
In summary, the studies outlined above emphasized edge of the limb’s biomechanical properties (Pruszynski
the following principles: (1) upper-limb postural control et al., 2011) using the paradigm developed by Kurtzer
and reaching involve continuous sensory processing; et al. (2008). The authors applied different combina-
(2) feedback control processes selectively compensate tions of shoulder and elbow torques evoking flexor or
for errors that interfere with the ongoing task, produc- extensor motion at the elbow and no motion at the
ing motor strategies appropriate for the movement shoulder. As a result of these multijoint perturbation
goal; (3) these principles describe voluntary motor loads, shoulder motion (none, in this case) is ambigu-
behavior as well as responses to visual and mechanical ous about the applied perturbation, and the motor
perturbations; and (4) sophisticated feedback responses system can only appropriately counter the underlying
emerge ∼50–60 ms following a mechanical perturba- torque by taking elbow motion into consideration. The
tion, coinciding with the long-latency stretch response. responses of shoulder-related neurons in MI were found
to appropriately respond to the applied load at ∼50 ms,
M1 as part of a flexible feedback controller about 15 ms before long-latency responses were
recorded in shoulder muscles (figure 39.4D).
A robust observation across the studies outlined above How this transcortical feedback pathway resolves this
is that short-latency responses (∼20–50 ms) are pre- multijoint integration problem is unclear. One clue is
dominantly sensitive to muscle stretch (Pruszynski & that the earliest activity in MI, from 20 to 50 ms after a
Scott, 2012), whereas task-dependent responses consis- perturbation, does not reflect specific features of the
tently emerge in the long-latency time window (∼50– perturbation (figure 39.4D). That is, regardless of per-
100 ms). Long-latency responses coincide with the turbation direction, all neurons sensitive to shoulder or
contribution of long-loop mechanisms, including a elbow motion display similar responses until 50 ms after
transcortical pathway through M1 (Cheney & Fetz, the perturbation is applied. This may suggest that M1
1984; Desmedt, 1978; Matthews, 1991). Indeed, single- requires ∼30 ms to identify the appropriate response, or
unit recordings in monkeys indicate that M1 receives that knowledge of limb mechanics is computed else-
somatosensory feedback (Evarts & Fromm, 1977; Scott where in sensorimotor circuits. Several brain regions
& Kalaska, 1997) or mechanical perturbations (Herter, that receive sensory feedback from the limb project to
Korbel, & Scott, 2009; Picard & Smith, 1992). More- M1, including primary somatosensory cortex, parietal
over, human studies using transcranial magnetic stimu- area 5, and cerebellum (Fromm & Evarts, 1982; Martin,
lation emphasize a causal link between M1 processing Cooper, Hacking, & Ghez, 2000; Mason, Miller, Baker,
and long-latency stretch responses, since motor cortex & Houk, 1998). There was considerable interest in
stimulation disrupts the task-dependent features of examining how brain regions including M1 responded
long-latency responses (Capaday, Forget, Fraser, & to sensory feedback in the 1970s (for a review, see
Lamarre, 1991; Day, Riescher, Struppler, Rothwell, & Desmedt, 1978), but since then the role of sensory feed-
Marsden, 1991; Kimura, Haggard, & Gomi, 2006, but back in M1 processing has received little attention from
see also Shemmell, An, & Perreault, 2009). Given the the scientific community. Given the tight link between
involvement of M1 in the generation of voluntary voluntary control and sensory feedback processing, the
behavior (Porter & Lemon, 1993) and rapid feedback use of OFC as a framework to understand voluntary
pathways, this brain region is a clear candidate to motor control has led to a renewed interest in how dif-
implement flexible feedback control strategies (Scott, ferent cortical and subcortical circuits participate in
2004). feedback processing for motor control (Scott, 2012).
A number of neurophysiological studies in nonhu-
man primates have highlighted that a transcortical feed- Conclusion
back through M1 provides important task-dependent
processing following mechanical perturbations. In a We have argued that OFC is a powerful tool for under-
seminal study, Evarts and Tanji (1976) found that within standing biological motor control and that it has shed
40 ms of an upper limb perturbation, the responses of light on many of the complexities the brain must con-
pyramidal tract neurons differ depending on whether sider to move successfully in the presence of neural
the monkey was instructed to push or pull a handle. variability or environmental disturbances. Perhaps the
The authors suggested that these task-dependent neural most important contribution of OFC in movement
crevecoeur, cluff and scott: goal-directed movement planning and execution 473
Gielen, C. C., Ramaekers, L., & van Zuylen, E. J. (1988). on reaching and adaptive control. J Neurophysiol, 83(4),
Long-latency stretch reflexes as co-ordinated functional 1886–1899.
responses in man. J Physiol, 407(1), 275–292. Mason, C. R., Miller, L. E., Baker, J. F., & Houk, J. C.
Gribble, P. L., & Ostry, D. J. (1999). Compensation for (1998). Organization of reaching and grasping movements
interaction torques during single- and multijoint limb in the primate cerebellar nuclei as revealed by focal musci-
movement. J Neurophysiol, 82(5), 2310–2326. mol inactivations. J Neurophysiol, 79(2), 537–554.
Hammond, P. H. (1956). The influence of prior instruction Matthews, P. B. (1984). Evidence from the use of vibration
to the subject on an apparently involuntary neuro-muscular that the human long-latency stretch reflex depends upon
response. J Physiol, 132(1), 17–18P. spindle secondary afferents. J Physiol, 348(1), 383–415.
Harris, C. M., & Wolpert, D. M. (1998). Signal-dependent Matthews, P. B. C. (1991). The human stretch reflex and the
noise determines motor planning. Nature, 394(6695), motor cortex. Trends Neurosci, 14(3), 87–91.
780–784. Mehta, B., & Schaal, S. (2002). Forward models in visuomo-
Hasan, Z. (2005). The human motor control system’s tor control. J Neurophysiol, 88(2), 942–953.
response to mechanical perturbation: Should it, can it and Michiels, W., & Niculescu, S.-I. (2007). Stability and stabi-
does it ensure stability? J Motor Behav, 37(6), 484–493. lization of time-delay systems. An eigenvalue-based
Herter, T. M., Korbel, T., & Scott, S. H. (2009). Compari- approach. In Advances in design and control, 12. Philadelphia,
son of neural responses in primary motor cortex to tran- PA: Society for Industrial and Applied Mathematics (SIAM)
sient and continuous loads during posture. J Neurophysiol, Publications.
101(1), 150–163. Mutha, P. K., & Sainburg, R. L. (2009). Shared bimanual
Hollerbach, J., & Flash, T. (1982). Dynamic interactions tasks elicit bimanual reflexes during movement. J Neuro-
between limb segments during planar arm movement. Biol physiol, 102(6), 3142–3155.
Cybern, 44, 67–77. Nakano, E., Imamizu, H., Osu, R., Uno, Y., Gomi, H.,
Izawa, J., & Shadmehr, R. (2008). On-line processing of Yoshioka, T., & Kawato, M. (1999). Quantitative exami-
uncertain information in visuomotor control. J Neurosci, nations of internal representations for arm trajectory plan-
28(44), 11360–11368. ning: Minimum commanded torque change model.
Kalman, R. (1960). A new approach to linear filtering and J Neurophysiol, 81(5), 2140–2155.
prediction problems. J Basic Eng-T ASME, 82, 35–45. Nashed, J. Y., Crevecoeur, F., & Scott, S. H. (2012). Influ-
Kawato, M. (1999). Internal models for motor control and ence of the behavioral goal and environmental obstacles
trajectory planning. Curr Opin Neurobiol, 9(6), 718–727. on rapid feedback responses. J Neurophysiol, 108(4),
Kimura, T., & Gomi, H. (2009). Temporal development of 999–1009.
anticipatory reflex modulation to dynamical interactions Omrani, M., Diedrichsen, J., & Scott, S. H. (2013). Rapid
during arm movement. J Neurophysiol, 102(4), 2220–2231. feedback corrections during a bimanual postural task.
Kimura, T., Haggard, P., & Gomi, H. (2006). Transcranial J Neurophysiol, 109(1), 147–161.
magnetic stimulation over sensorimotor cortex disrupts Osborne, L. C., Lisberger, S. G., & Bialek, W. (2005).
anticipatory reflex gain modulation for skilled action. A sensory source for motor variation. Nature, 437(7057),
J Neurosci, 26(36), 9272–9281. 412–416.
Knill, D. C., Bondada, A., & Chhabra, M. (2011). Flexible, Phillips, C. G. (1969). The 1968 Ferrier lecture: Motor appa-
task-dependent use of sensory feedback to control hand ratus of the baboon’s hand. Proc R Soc Lond B Biol Sci,
movements. J Neurosci, 31(4), 1219–1237. 173(1031), 141–174.
Körding, K. P., & Wolpert, D. M. (2004). Bayesian integra- Picard, N., & Smith, A. M. (1992). Primary motor cortical
tion in sensorimotor learning. Nature, 427(6971), responses to perturbations of prehension in the monkey.
244–247. J Neurophysiol, 68(5), 1882–1894
Krutky, M. A., Ravichandran, V. J., Trumbower, R. D., & Porter, R., & Lemon, R. N. (1993). Corticospinal function and
Perreault, E. J. (2010). Interactions between limb and voluntary movement. New York, NY: Clarendon Press.
environmental mechanics influence stretch reflex sensitiv- Pruszynski, J. A., Kurtzer, I., Nashed, J. Y., Omrani, M.,
ity in the human arm. J Neurophysiol, 103(1), 429–440. Brouwer, B., & Scott, S. H. (2011). Primary motor cortex
Kurtzer, I. L., Pruszynski, J. A., & Scott, S. H. (2008). underlies multi-joint integration for fast feedback control.
Long-latency reflexes of the human arm reflect an internal Nature, 478(7369), 387–390.
model of limb dynamics. Curr Biol, 18(6), 449–453. Pruszynski, J. A., Kurtzer, I., & Scott, S. H. (2008). Rapid
Kurtzer, I., Pruszynski, J. A., & Scott, S. H. (2009). Long- motor responses are appropriately tuned to the metrics of
latency responses during reaching account for the mechan- a visuospatial task. J Neurophysiol, 100(1), 224–238.
ical interaction between the shoulder and elbow joints. Pruszynski, J. A., Kurtzer, I., & Scott, S. H. (2011). The
J Neurophysiol, 102(5), 3004–3015. long-latency reflex is composed of at least two functionally
Li, W., & Todorov, E. (2007). Iterative linearization methods independent processes. J Neurophysiol, 106(1), 449–459.
for approximately optimal control and estimation of non- Pruszynski, J. A., & Scott, S. H. (2012). Optimal feedback
linear stochastic system. Int J Control, 80(9), 1439–1453. control and the long-latency stretch response. Exp Brain Res,
Liu, D., & Todorov, E. (2007). Evidence for the flexible 218, 341–359.
sensorimotor strategies predicted by optimal feedback Qian, N., Jiang, Y., Jiang, J.-P., & Mazzoni, P. (2013). Move-
control. J Neurosci, 27(35), 9354–9368. ment duration, Fitts’s law, and an infinite-horizon optimal
Marsden, C. D., Merton, P. A., & Morton, H. B. (1981). feedback control model for biological motor systems. Neural
Human postural responses. Brain, 104(3), 513–534. Comput, 25, 697–724.
Martin, J. H., Cooper, S. E., Hacking, A., & Ghez, C. (2000). Ronsse, R., Thonnard, J.-L., Lefèvre, P., & Sepulchre, R.
Differential effects of deep cerebellar nuclei inactivation (2008). Control of bimanual rhythmic movements: Trading
crevecoeur, cluff and scott: goal-directed movement planning and execution 475
40 Multineuronal Views of
Information Processing
AARON BATISTA
ABSTRACT Cognitive neuroscientists have only recently gained the interactions between pairs of neurons. Their tech-
the ability to record from many neurons at once while animals nique, cross-correlation (now often called synchrony),
behave. The multineuronal view of information processing
involved measuring the probability of neuron 2 firing
provides new insights into the collective action of neurons.
Some of the principles emerging from these studies are (1) an action potential with respect to the time when
there is flexibility in the exchange of information between neuron 1 fires (figure 40.1). Connectivity schemes such
brain areas; (2) populations of neurons together embody as direct excitation, direct inhibition, shared excitation,
underlying computational structures; (3) neurons are active and shared inhibition could be inferred from the shape
in response to specific patterns of synaptic input, and these of the cross-correlation histogram. The central opera-
input patterns can provide a more accurate description of
neural “tuning” than do external covariates. A fuller under- tion is
standing of how neural networks transform and manipulate C(τ) = ∑ S A (t )S B (t + τ), (1)
information is emerging from the perspectives provided by t
multineuronal recording techniques and the computational where C is the cross-correlation histogram (or correlo-
tools being developed to interpret multineuronal data.
gram) and S is the time of action potentials from one
neuron. The time lag between action potentials from
Neuroscience has made tremendous strides in discover- the two cells is τ. Note that this equation is nearly identi-
ing how the sensory environment and ensuing actions cal to a convolution (differing only by a sign). The
are represented by individual neurons. The question of cross-correlation measures how similar two signals are
how information is transformed as it flows through the over a range of temporal offsets. C is constructed for
brain is just as important, but we know far less about it. each individual trial, then averaged over trials and cor-
Understanding how neural information is altered, elab- rected for chance to yield correlograms like those in
orated, and refined—essentially, neural computation— figure 40.1. The appearance of a peak in a correlogram
requires observing more than one neuron at a time. can indicate an interaction between two neurons, but
This chapter surveys the history of multineuronal Carlos Brody (1999) pointed out that such plots must
investigations of the brain. We begin with paired record- be interpreted with caution. Peaks can appear spuri-
ings and the work of George Gerstein and his colleagues ously, for example, if two neurons that are not actually
in the 1960s. They introduced the first tools to probe connected respond to the same stimulus, or are oscil-
information flow between pairs of neurons. Next we lating in unison. Statistical techniques and a dose of
survey other paired-recording techniques and what they good judgment can help resolve whether a peak in a
have contributed to our understanding of the neural correlogram actually indicates neural connectivity.
mechanisms of sensory, motor, and cognitive processes.
Then we move beyond pairwise interactions to consider Some Applications of Cross-Correlation Reid
recently introduced multineuronal recording and anal- and Alonso (1995) used the cross-correlation technique
ysis techniques, and the new perspectives on neuronal to examine connectivity between the lateral geniculate
information processing that they are enabling. The nucleus and the primary visual cortex. They discovered
chapter concludes with a perspective on the types of that the connections between these areas were highly
questions whose answers might be near at hand using specific: they were tightly constrained to overlapping
multielectrode techniques. receptive field locations and matching light-dark polar-
ity. Even when a correlation was present, it was fairly
Paired recording techniques weak: the probability of a V1 spike appearing in response
to the spike of a particular lateral geniculate nucleus
Cross-Correlation Perkel, Gerstein, and Moore cell was less than 8%. It is reasonable to expect the
(1967) introduced a statistical framework to analyze lateral geniculate nucleus–to-V1 projection to be one of
A B A B C A
B
Figure 40.1 Covariogram shapes and the connectivity they The autapse in the schematic symbolizes any influence (direct
imply. (A) Direct excitation; (B) direct inhibition; (C) shared or indirect) that A has on itself. The dashed lines show the
excitation from an unobserved third neuron; (D) an autocor- null hypothesis of independence between the cells, con-
relation reveals oscillations. The large peak at 0 is suppressed. structed by shuffling data across trials.
the stronger feedforward pathways in the nervous confirmed and extended earlier observations (Riehle,
system. Although the specificity of connectivity between Grun, Diesmann, & Aertsen, 1997), using a related
the areas is perhaps not surprising, the weakness of the technique called unitary event analysis, of coincident
connection strength between particular neurons is a spiking in M1 that conveyed information about the cog-
surprise. nitive aspects of movement.
At the opposite pole of the nervous system—motor For many years, synchronized oscillations have been pro-
output—a technique conceptually related to cross- posed to serve a special role in cortical function. The
correlation has been used to illustrate connectivity in demonstration of synchronized oscillations between a
the motor system. In the spike-triggered average technique, pair of cells relies on the cross-correlation technique.
electromyographic (EMG) activity is recorded along Synchrony is the tendency for two cells to fire together—
with the activity of a neuron in primary motor cortex that is, to bear a significant correlation at a time lag
(M1). When EMG signals are aligned on the time of near zero. An oscillation, of course, is the tendency for
spiking of an M1 cell, a distinct peak sometimes emerges a single neuron to fire at a particular frequency. If the
after activity has been averaged over thousands of autocorrelation of a cell is plotted (figure 40.1D), where
spikes. M1 neurons that can be shown in this manner SB in Eq. 1 is replaced with SA, oscillations appear as
to cause EMG activity are termed corticospinal (CS) cells. periodic side bands. When the cross-correlation between
They project to the spinal cord, and as such they provide two neurons reveals periodic side bands, it indicates a
more direct control of muscles and movements than do synchronized oscillation between those cells. Charlie
other cortical neurons. A special subclass of CS cells Gray and colleagues (Gray, König, Engel, & Singer,
consists of the corticomotoneuronal (CM) cells. These 1989) found that visual cortex neurons exhibit synchro-
project directly to the motor neurons in the spinal cord. nized oscillations that might function in the integration
They are best identified anatomically, but physiological of separate visual features into a cohesive image. The
evidence for them comes from short-latency peaks in electroencephalographic (EEG) signal is a consequence
the spike-triggered averaged EMG signal. CM cells may of large-scale synchronized oscillations among popula-
be essential for the dextrous control of the fingers tions of neurons.
(Bortoff & Strick, 1993).
An important open question is whether correlations Coherence Methods Cross-correlation studies gen-
among neurons carry information beyond what is con- erally reveal fairly weak interactions between pairs of
veyed by the activity of the neurons taken individually. neurons, even when they are near each other. Typically
To this end, investigators have related the presence of very few neuron pairs exhibit synchronous spiking,
enhanced correlation to the neural encoding of sensory whether they are examined in sensory (Smith et al.,
(Panzeri, Schultz, Treves, & Rolls, 1999) and motor 2013) or motor (Jackson, Gee, Baker, & Lemon, 2003)
events. For example, Nicho Hatsopoulos and colleagues cortex. The scarcity of synchronized neurons is surely
(Hatsopoulos, Ojakangas, Paninski, & Donoghue, 1998) exacerbated when the two neurons are not in the same
applied cross-correlation to neurons recorded in M1. brain area. Coherence analysis improves on the resolving
They observed synchrony between pairs of neurons that power of cross-correlation by examining the correla-
carried information about the timing of movement, tion within specific frequency bands separately. For
beyond what was evident in individual cells. This rhythmic neural phenomena, such as the local field
Latent dimension 2
was moved away from the region of common drive to
both neurons, the cross-correlation decreased. The
noise correlation was unaffected by changing orienta- Target
onset
tion, but it was influenced by the contrast of the visual Movement
stimulus: noise correlation increased when visual con- onset
Latent dimension 1
trast decreased. The authors reason that the increase in
noise correlation with decreasing stimulus contrast
indicates the V1 network shifts from a stimulus-driven D
mode to a mode in which intrinsic network dynamics
dominate its activity.
A reflection before we leave the topic of pairwise
techniques. The noise correlation technique hinges on
the notion that the correlation between neurons is not
driven by a sensory stimulus, but rather by some under-
lying (unobserved) process that affects both cells. This
process need not be “noise” (that is, random and
E
Projection onto jPC 2 (a.u.)
“Voltaire said of Sir Isaac Newton that with all his science he completed and the goal achieved. The command signal
knew not how his hand moved.” should also contain high-level information appropriate
—John Napier, 1971 for the context of the movement, which may vary from
trial to trial.
Skilled grasp is a distinctive component of human
ABSTRACT This chapter focuses on action vocabularies and
motor control. Visual guidance of skilled prehension
motor commands with reference to object grasp and manipu-
lation. First, I highlight the language used in the motor and manipulation of objects is essential for human
control literature related to motor “commands,” which creative art, sculpture, and music, as well as for tech-
emphasizes the concept of supraspinal structures generating nological development, tool use, and manufacture.
movement by passing a command to the spinal cord. Ideas Therefore, in this chapter, I focus on the status of
about commands for movement have been strongly influ-
research on commands for grasp control. I begin by
enced by the fact that electrical stimulation of the motor
cortex actually causes movement. I highlight critical differ- looking at evidence for activity in motor and premo-
ences between evoked and natural movements, and especially tor cortex that is selective for particular types of grasp
differences in timing and patterning of neural activity emanat- carried out under visuomotor control. I then consider
ing from the primary motor cortex. I discuss the rich versatil- how activity in the visuomotor grasping circuit results
ity of skilled grasp and suggest that this versatility cannot be in activity in corticospinal neurons with projections to
explained by combination of just a few muscle synergies. I
emphasize that the visuomotor grasping circuit primes the the cervical spinal cord, and targeting motor nuclei
motor network for grasp of objects; other (prefrontal) circuits innervating the upper limb muscles recruited during
are involved in how the object should be used. I describe some grasp.
of the work on grasp selectivity in both ventral premotor and Although some features of this corticospinal output
primary motor cortex, and on the interactions between them. from primary motor cortex (M1) fit well with a role as
I discuss the evidence for motor cortex pyramidal tract
neurons acting as potential “command neurons.” These pyra-
command neurons, others certainly do not. In addition,
midal neurons exhibit some features that fit well with such a we now know that some of the same neurons recruited
role, including some neurons intercalated in “smart” transcor- when movements are generated are also recruited
tical reflexes and with fast, direct access to motoneurons. during other states such as action observation, so we
These features are discussed in the light of theories of motor have to understand what precisely it is that distinguishes
control, such as optimal feedback control and active infer-
the level and pattern of activity that is specifically associ-
ence. Activation of pyramidal tract neurons is not restricted
to their role as command neurons: their discharge can also ated with movement generation. These approaches may
show mirror properties, being modulated during observation help define exactly what we mean by motor commands
of others’ actions, without any sign of concomitant electro- for grasp.
myographic activity.
Much of the literature on motor control theory makes The language of motor commands: The impact of
free use of the term “motor command.” By definition, stimulation studies
a command must be both delivered and received.
Although neural activity in a number of different struc- It is generally assumed that the brain’s motor system
tures has been identified as having the properties of generates motor commands that cause movements to
such a command signal, far less is known about how occur (Franklin & Wolpert, 2011; Frey et al., 2011). The
these signals are received by spinal interneurons and term “command” is closely related to that of agency in
motoneurons and transformed into muscle activity and the motor system, since by implication the motor
movement. A further point is that commands for move- system, by issuing command signals for movement, rein-
ments should not only initiate the movement, but also forces our sense that these commands cause things to
continue to guide and update it until the action is happen.
ABSTRACT Actions are chosen on the basis of expectations Pessiglione, 2009) and it now seems likely that the
about the benefits they will yield. Areas in the brain’s frontal vmPFC/mOFC is involved in the actual making of the
lobes are important for making decisions about which choice decision itself. When people make a decision between
it is best to take. In functional magnetic resonance imaging
experiments, the ventromedial prefrontal cortex exhibits
two potential choices, the vmPFC/mOFC exhibits a
signals related to the values of potential choices, and the activ- signal that suggests it is comparing the values of the
ity pattern is consistent with the emergence of a decision. choices. For example, Boorman, Behrens, Woolrich,
Ventromedial prefrontal cortex lesions disrupt value-guided and Rushworth (2009) used functional magnetic reso-
decision making. Another frontal lobe area, the anterior cin- nance imaging (fMRI) to measure blood oxygen level–
gulate cortex, is also important for value-guided decision
dependent (BOLD) indices of brain activity in human
making. There is some evidence suggesting that anterior cin-
gulate cortex is especially involved in the selection of an subjects while they made a series of decisions between
action in the light of the rewards that will follow and the effort two different choices associated with different numbers
that will be invested. More fundamentally, it may be con- of reward points. The points won by the subjects were
cerned with the computation and comparison of values translated into a monetary payment at the end of the
needed for decision making during foraging. During forag-
experiment. The decisions were difficult in the sense
ing, the key decision is whether to engage and pursue the
default choice, or whether the foraging opportunities avail- that the reward associations contained two elements: a
able elsewhere in the environment mean it is better to certain number of points (reward magnitude), which
switch away from the current behavior and pursue other changed randomly from trial to trial, and a certain
alternatives. probability of the points being delivered (reward proba-
bility), which slowly changed over the course of a series
When we make a movement, we need to make sure that of decisions. Therefore, in order to choose effectively,
it is executed correctly. We have to ensure that muscles subjects had to consider both of the elements that com-
and joints move in such a way that the limb is directed posed each option.
to the desired target location and the hand is config- Boorman and colleagues found vmPFC/mOFC
ured correctly. Rather than looking at the neural mech- encoded the difference in value between the option
anisms that determine how an action is executed, this that was chosen and the option that was rejected in each
chapter instead focuses on the cortical mechanisms that decision (figure 42.2). They referred to this as a relative
underlie why one action is chosen instead of another. It chosen value signal. At the time that a choice is made,
focuses on the cortical mechanisms (figure 42.1) that the vmPFC/mOFC BOLD signal becomes positively
underlie selection of a choice on the basis of the value correlated with the value of the option that is chosen
of its consequences. and negatively correlated with the value of the option
that is being rejected (figure 42.2). One interpretation
of this pattern is that vmPFC/mOFC activity increas-
Value signals in prefrontal cortex for guiding ingly reflects the value of the option that is to be chosen,
decisions and at the same time it is less and less activated by the
value of the option that is to be rejected. When this
The ventromedial prefrontal cortex (vmPFC) and adja- process is complete, the decision is made.
cent medial orbitofrontal cortex (mOFC) play a central Several other fMRI studies have confirmed the exis-
role in choice selection. For some time, it has been clear tence of relative chosen value signals in vmPFC/mOFC
that activity in the vmPFC/mOFC region is correlated (de Martino, Fleming, Garrett, & Dolan, 2013; FitzGer-
with the values of stimuli and choices (Kable & ald, Seymour, & Dolan, 2009; Philiastides, Biele, &
Glimcher, 2007; Lebreton, Jorge, Michel, Thirion, & Heekeren, 2010; Wunderlich, Dayan, & Dolan, 2012).
rushworth et al.: choice values: the frontal cortex and decision making 501
Figure 42.1 Frontal brain regions, in the macaque, involved orbitofrontal cortex; lOFC, lateral orbitofrontal cortex; ACCs,
in reward-guided learning and decision making. Abbrevia- anterior cingulate cortex sulcus; ACCg, anterior cingulate
tions: vmPFC/mOFC, ventromedial prefrontal cortex/medial cortex gyrus. (Adapted from Rushworth et al., 2011.)
Figure 42.3 (A) The biophysical model Hunt and col- values (on frequency range 3–9 Hz; black lines) and choice
leagues (2012) used to predict a transition from activity that value difference (on frequency range 2–4.5 Hz; gray lines)
is proportional to the sum of choice values to the value of the on biophysical model activity; solid lines are correct trials,
chosen value. It contains two pools of excitatory pyramidal dashed lines, incorrect trials. (C) vmPFC activity shows
neurons (PA, PB) corresponding to choices of either option A several value-related hallmarks of the biophysical network
or option B. There is recurrent excitation between neurons model. (D) Effect of overall value (3–9 Hz; black) and value
within pools, but inhibitory interneurons (Pi) mediate com- difference (2–4.5 Hz; gray) on correct/error trials (solid/
petition between pools. Activity in each pool is initially dashed lines, respectively) during first half of experiment in
affected by an input proportional (IA, IB) to the value of each vmPFC. The analysis used here was performed on human
option, but inhibition between pools leaves only a single pool vmPFC but was equivalent to that performed in panel
in a high-firing attractor state. The corresponding option is 42.3B on the biophysical model. (Adapted from Hunt et al.,
then chosen. (B) Z-scored effect of overall sum of choice 2012.)
rushworth et al.: choice values: the frontal cortex and decision making 503
signals—one was in the parietal cortex, but the other probability of reward), but on other days its reward
was in vmPFC/mOFC. In summary, the results are con- probability began to approach that of the best stimulus
sistent with the idea that a value-comparison process (either 0.2 or 0.375 probability of reward). Discriminat-
takes place in vmPFC/mOFC, and that during the ing the best option from the second-best option there-
course of the comparison vmPFC/mOFC is dominated fore became more and more difficult as the second-best
by different types of value signal. option’s value approached that of the best option. In
the control state macaques were able to make the deci-
Disrupting vmPFC/mOFC impairs value-guided sion in all three cases, but they learned the identity of
decision making the best option more slowly when they were performing
testing sessions in which the best option and second-
If it is true that vmPFC/mOFC discriminates between best option were close in value (figure 42.4). Macaques
choices on the basis of their value, then we might with vmPFC/mOFC lesions were significantly worse
expect that vmPFC/mOFC lesions would impair value- when the second-best option was close in value to the
guided decision making. We might expect that choice best option. In other words, vmPFC/mOFC lesions
discrimination would become more and more impaired made macaques worse at taking the most difficult value-
as the difference in choice values decreased. This pre- guided decisions. Moreover, the disruptive impact on
diction is based on what happens when mechanisms for difficult decisions was specific to vmPFC/mOFC lesions
color discrimination are impaired by brain lesions in and did not occur after lesions in other frontal brain
visual association cortex; discrimination performance areas that contain neurons with reward-related activity
becomes worse and worse as differences in the color of patterns, such as anterior cingulate cortex (ACC) or
two stimuli are made smaller and smaller (Buckley, lateral orbital frontal cortex (lOFC) (Noonan et al.,
Gaffan, & Murray, 1997). 2010; Rudebeck et al., 2008; Walton, Behrens, Buckley,
The prediction was borne out when lesions were Rudebeck, & Rushworth, 2010).
made in vmPFC/mOFC in macaques (Noonan et al., Another way to assess value-guided decision making
2010). Noonan and colleagues trained their macaques is to examine the consistency and transitivity of choices.
to choose between three stimuli with different probabi- Normally, if a person or a monkey prefers option X over
listic associations with reward. Each day, the macaques Y and Y over Z, then their preferences are said to be
learned about three new stimuli, but on each day one transitive if they also prefer X over Z. It is just such
of the stimuli was a high-value stimulus because it was transititivity that is lost after vmPFC/mOFC lesions
associated with a high probability of reward (0.6), and (Rudebeck & Murray, 2011). Once again, the effect is
one stimulus was a low-value stimulus because it had no specific to vmPFC/mOFC lesions and is not seen after
association with reward (0 probability of reward). The lOFC lesions. Similar patterns of impairment in value-
third stimulus was important because on some testing guided decision making have also been reported in
days it was just as poor in value as the worst stimulus (0 human patients with vmPFC/mOFC lesions (Camille,
Figure 42.4 Effect of choice option value proximity for show number of trials to reach 70% V1 choices. Lesions of
macaques with mOFC lesions. Proportion of choices which mOFCs caused impairments when the best and second-best
were of the best value option when the difference in value value differences were small (A). The mOFC lesion locations
between the best and second-best option was small (A, 0.2), are represented on an unoperated control brain, with dark-
medium (B, 0.4) and large (C, 0.6). Control pre-lesion (light ness indicating lesion overlap (overlap in one to four animals).
gray), post-mOFC lesion (dark gray) performance. Insets (Adapted from Noonan et al., 2010.)
rushworth et al.: choice values: the frontal cortex and decision making 505
(Luk & Wallis, 2013). ACC lesions impaired action- lead to one piece of fruit and only later perceive the
reward association learning but not stimulus-reward opportunity to pursue another piece of fruit by taking
association learning, while the opposite was true after another course of action. The critical choice for the
lOFC lesions (Rudebeck et al., 2008). A similar dissocia- foraging animal is, therefore, whether to engage with a
tion between the effects of ACC and OFC lesion deficits potential food option or whether to carry on searching
has been reported in human patients (Camille, Tsu- for better opportunities elsewhere.
chida, & Fellows, 2011). In contrast to cognitive neuroscientists, behavioral
Such patterns of neural activity and lesion effects ecologists have long been interested in such choices.
suggest the existence of different mechanisms for deci- Whether or not an animal engages with the potential
sion making in ACC versus vmPFC/mOFC and lOFC. option it encounters is determined by the value of the
It is possible that these different mechanisms work in a encountered option, the effort that will be expended in
completely parallel manner, with one set of brain engaging with it, whether the environment is, on
regions learning associations between actions and average, sufficiently rich that better options are likely
rewards and another set learning associations between to be encountered frequently if the animal carries on
stimuli and rewards. However, it seems unlikely that this searching, and how effortful it will be to continue
will be the case. While values may adhere to particular searching (Charnov, 1976; Stephens & Krebs, 1986). In
stimuli and objects in the world in a relatively constant summary, foraging decisions should be governed by the
fashion, it is not clear that this is always likely to be true average value of continuing to search (search value), the
of actions. For example, the value of an action directed effort entailed by searching (search costs), and the value
to the left or the right might, in many real-world sce- of each option encountered (encounter value). Despite
narios, change depending on which way a person is the ecological importance of such choices, surprisingly
facing. What might, however, be pertinent for a system little has been known about their neural mechanisms.
that can bring together information about reward and Kolling, Behrens, Mars, and Rushworth (2012)
actions is to decide whether it is worth persisting with attempted to compare the neural processes underlying
a given action or whether it might be better to try an engage/search foraging choices with the neural mecha-
alternative one. These are the types of decisions that nisms underlying binary decision making. On each
animals often have to make when there are no specific trial, human subjects could alternate between two dif-
stimulus-reward associations available to guide their ferent styles of decision making while fMRI data were
behavior. In addition, it might be useful for such a collected. Before the subjects entered the scanner, they
mechanism to compute whether the effort entailed by had learned about a set of visual stimuli in which each
an action outweighs the benefit that it bestows. The stimulus was associated with a different number of
next section considers such possibilities in the context points that were translated into a monetary payment at
of the foraging choices that are important for many the end of the trial. At the beginning of each trial in
animals. the fMRI experiment, subjects saw two of these stimuli
(figure 42.5). The options encountered in this way were
Distinct mechanisms for decision making in ACC referred to as the encounter options. They were intended
and vmPFC/mOFC to simulate the opportunity that a foraging animal
might engage with. The investigators refered to their
The last section discussed how vmPFC/mOFC might be values as encounter values. A box at the top of the screen
a mechanism for making decisions between potential included a number of alternative stimuli. These were
choices. Although such decisions are commonly studied intended to represent the richness of the subject’s
by psychologists, cognitive neuroscientists, and econo- current foraging environment and to indicate the search
mists, they may be surprisingly rare outside of the value; if the subject did not engage with the encoun-
laboratory. This is simply because in many natural situ- tered option, then he or she opted to search for better
ations, when an animal is foraging, potential food items options that would be drawn at random from the stimuli
are encountered sequentially rather simultaneously in the box and become the encounter options on the
(Freidin & Kacelnik, 2011). It might be only on some next trial. The first decision that subjects made on each
occasions that a macaque foraging in the wild is given trial was therefore a foraging-style choice to either
the opportunity to make the type of choice that the engage with an encountered option or to search for
vmPFC/mOFC appears to make—for example, a choice better alternatives. To simulate search cost, subjects lost
between an apple and an orange. Instead, what is more points if they chose to search (the cost on each trial was
likely to happen is that the macaque might first see an indicated by the color of the box surrounding the
opportunity to pursue a course of action that might search options). Subjects could keep opting to search
for better alternatives as many times as they wanted. values, and search costs. By contrast, vmPFC/mOFC
However, as soon as they opted to engage, they were only carried an encounter value signal but no represen-
then able to make the second type of decision—a binary tations of search value or search costs. Other fMRI
comparitive decision between the two component experiments have also reported integrated reward/
stimuli that constituted the encounter option. This effort signals in ACC as opposed to vmPFC (Croxson,
second decision was therefore of the same type as those Walton, O’Reilly, Behrens, & Rushworth, 2009).
that are known to be related to the vmPFC/mOFC. Another reason for thinking the two regions, vmPFC/
Several aspects of the results suggested binary choices mOFC and ACC, were mediating binary comparative
and foraging choices were being mediated by the decisions and foraging choices, respectively, could be
vmPFC/mOFC and the ACC, respectively. First, several found in examination of individual differences in activ-
brain areas, including ACC, were more active when ity patterns in the two regions. Individual differences in
subjects made foraging-style engage/search choices, the strength of ACC signals, but not vmPFC signals,
while other brain regions, including vmPFC/mOFC, were correlated with individual differences in foraging
were more active when subjects made binary compari- behavior. Individual differences in the strength of
son decisions (figure 42.5). Perhaps even more impor- vmPFC/mOFC signals, but not ACC signals, were cor-
tantly, ACC activity reflected all three key factors that related with individual differences in the way binary
should determine foraging: encounter values, search comparative choices were made.
rushworth et al.: choice values: the frontal cortex and decision making 507
The way in which ACC encoded values for foraging If ACC has a mechanism for valuation and promotion
choices was quite distinct from the manner in which of behavioral change and search, then this may suggest
vmPFC/mOFC encoded values during binary decision a reinterpretation of some other findings made during
making. As earlier sections have described, vmPFC/ earlier investigations of ACC. The action-reward learn-
mOFC encodes a relative value difference signal, with ing tasks that are impaired by ACC lesions (Camille,
the BOLD signal increasing as a function of the value Tsuchida et al., 2011; Kennerley, Walton, Behrens,
of the chosen option and decreasing as a function of Buckley, & Rushworth, 2006; Rudebeck et al., 2008)
the value of the rejected option. By contrast, rather typically involve alternation between actions but no
than encoding decision values in a framework that is informative stimuli. Repetitive selection of an action
relative to the choice that is made, the ACC encodes interleaved with periods of exploration of alternative
foraging choice values in a fixed manner; ACC activity actions may be just the sort of behavior normally under
always increases as a function of the search value and the control of a foraging system. It may be that the ACC
decreases as a function of the encounter value. The lesion did not disrupt a mechanism for linking specific
search value signal, however, ramps up more quickly actions to outcomes but that, instead, it disrupted a
when subjects are going to make the search choice mechanism with which animals decide whether to
rather than the engage choice (figure 42.5). persist with one choice versus switching to an
In the foraging task it was noticeable that by default alternative.
subjects often seemed simply to engage with the option The human ACC region that carries search/engage
placed in front of them; subjects made more of these signals for foraging is well placed to influence whether
choices and did not move away from them unless the or not a successful action will be repeated or whether
value of searching considerably outweighed the value the value of an alternative action will be explored. It lies
of engaging. One way of thinking about the ACC’s just adjacent and anterior to the human rostral cingu-
choice signals is that they are configured in such a way late motor area (Amiez & Petrides, 2014; Beckmann,
as to indicate the value of changing away from the Johansen-Berg, & Rushworth, 2009; Picard & Strick,
current default behavior and of searching for better 1996), and as we have already seen a similar region is
alternatives. interconnected with the rostral cingulate motor area
(van Hoesen et al., 1993), which has connections to a
Reinterpreting the role of ACC number of motor regions.
The hypothesis that ACC encodes value signals that are ACC neurons during searching and foraging
needed for foraging-style choices can explain some of
the other signals that have been reported in ACC in the A longstanding problem that has dogged attempts to
past. For example, it has been noted that during binary understand ACC has been the difficulty of finding sin-
choices the ACC often carries a signal that is inversely gle-neuron activity patterns that are consistent with
proportional to the difference in value between the some of the most influential theories of ACC func-
choices available, rather than proportional to the dif- tion. For example, it has proven difficult to identify
ferences in value between the choices as in vmPFC/ ACC neurons that detect conflict (Nakamura, Roesch,
mOFC (Hare et al., 2011). According to the foraging & Olson, 2005). By contrast, there are several reports
hypothesis, the inverse value signal recorded during a of ACC neuron activity that suggest that it is con-
binary comparison task can be interpreted as a signal cerned with searching the environment for the best
reporting how beneficial it would be to not take the choices to take and with foraging-style search/engage
choice that is about to be taken but, instead, to switch decisions.
to the other choice. Quilodran, Rothé, and Procyk (2008) taught
Another important strand of research has linked macaques a task in which they had to work out which
ACC to detecting when there is conflict between pos- one of four options was associated with reward. The
sible responses that might be made (Botvinick, 2007). macaques solved the task by searching through each of
Response conflict occurs when there is a similar the four possible choices. When they identified the
amount of evidence favoring more than one response, choice that was associated with reward, they were
and once again the ACC may be signaling the value of allowed to repeat it again for several trials and received
taking a course of action that is alternative to the one rewards on each occasion. After several trials, however,
that is being taken. It is not clear, however, that a the reward was reassigned to another option and the
theory that focuses just on conflict monitoring could cycle of searching for the best option, and then repeat-
explain some of the results found in the foraging task. edly choosing that option, began again. The animals,
rushworth et al.: choice values: the frontal cortex and decision making 509
Figure 42.7 The activity of neurons in ACC in monkeys gain was a function of the search costs that were to be paid;
during a foraging task. Outcome-related activity increases here, rate of gain is indicated by a regression slope (beta
with each outcome that is received for foraging in a given weight) relating time spent foraging in a patch with firing
patch in an example neuron (A) and, on average, the gain of rate, and it can be seen that it decreases with the search cost
the outcome-related response with time in the patch co-varied that will have to be paid in order to travel to a new patch.
with the speed of departure from the patch (B). The rate of (Adapted from Hayden et al., 2011.)
but the reward-related firing rates gradually increased reward benefit/effort cost decisions that have been
as the time approached for the patch-leaving decision. associated with ACC activity (Kolling et al., 2012).
After the reward-related firing rate reached a certain Lesion studies also suggest ACC plays a critical role
level, the monkeys tended to switch to a new patch on in integrating information about reward and effort
the next trial. The steepest rises in firing rates were expectations during decision making. Rats can be
seen on those trials where the animal left the patch trained to make choices between two arms of a T-maze
soonest and when the travel time to the next patch was that are associated with different reward benefits (for
shortest. example, two versus four reward pellets) and different
A more direct demonstration that ACC neurons effort costs (such as climbing over a barrier versus no
might integrate information, not just about reward barrier). Normally rats are prepared to pay some cost
expectations, but also about the effort that is going to in order to receive a bigger reward benefit, but this
be invested in pursuing a course of action comes from changes after lesions are made in the Cg1/Cg2 fields
a multidimensional choice task that involved macaques of the ACC (Rudebeck, Walton, Smyth, Bannerman, &
making choices between different pictures that were Rushworth, 2006; Schweimer & Hauber, 2005; Sch-
each associated with different outcomes (Kennerley, weimer, Saft, & Hauber, 2005; Walton, Bannerman,
Dahmubed, Lara, & Wallis, 2009). The three critical Alterescu, & Rushworth, 2003; Walton, Bannerman, &
dimensions that defined each outcome were the size of Rushworth, 2002; Walton et al., 2009). By contrast,
reward payoff (volume of juice delivered), effort lesions of orbitofrontal cortex or prelimbic cortex, in
(number of lever presses necessary to earn the reward), the rat, do not have the same effect.
and the probability (probability that reward would be
delivered). Kennerley and colleagues found that Conclusion
neurons in all three areas they investigated, ACC, lOFC,
and lateral prefrontal cortex, encoded the three differ- We have emphasized the different functional contribu-
ent factors that determined choice values. However, tions of frontal lobe regions to value-guided decision
ACC neurons more frequently encoded each of the making. We have emphasized a flexible vmPFC/mOFC
three value-determining factors than did lOFC or lateral system that makes comparisons between choice values
prefrontal neurons. Individual ACC neurons were also by focusing on the most relevant aspects of value (for
especially likely to multiplex information across all example, reward probability or reward magnitude). In
three value dimensions so that information about, for some cases such a system may work in series with brain
example, both the reward benefits and the effort costs areas such as the ACC, which might select an action
of a choice was integrated within the firing rate of single compatible with the reward goal that is the focus of
ACC neurons. Such activity patterns might underlie the attention. In some situations, however, the ACC might
integration of reward/effort expectations in ACC activ- operate in a quite independent manner to take simple
ity reported in fMRI studies (Croxson et al., 2009) and stay-switch decisions on the basis of different types of
would, of course, enable ACC to make the type of value information such as average search values and
rushworth et al.: choice values: the frontal cortex and decision making 511
performance of tasks involving conflict. J Neurophysiol, Rushworth, M. F., Noonan, M. P., Boorman, E. D.,
93(2), 884–908. Walton, M. E., & Behrens, T. E. (2011). Frontal cortex
Noonan, M. P., Mars, R. B., & Rushworth, M. F. (2011). and reward-guided learning and decision-making. Neuron,
Distinct roles of three frontal cortical areas in reward- 70(6), 1054–1069.
guided behavior. J Neurosci, 31(40), 14399–14412. Schweimer, J., & Hauber, W. (2005). Involvement of the rat
Noonan, M. P., Walton, M. E., Behrens, T. E., Sallet, J., anterior cingulate cortex in control of instrumental
Buckley, M. J., & Rushworth, M. F. (2010). Separate value responses guided by reward expectancy. Learn Mem, 12(3),
comparison and learning mechanisms in macaque medial 334–342.
and lateral orbitofrontal cortex. Proc Natl Acad Sci USA, Schweimer, J., Saft, S., & Hauber, W. (2005). Involvement
107(47), 20547–20552. of catecholamine neurotransmission in the rat anterior cin-
Padoa-Schioppa, C. (2009). Range-adapting representation gulate in effort-related decision making. Behav Neurosci,
of economic value in the orbitofrontal cortex. J Neurosci, 119(6), 1687–1692.
29(44), 14004–14014. Stephens, D. W., & Krebs, J. R. (1986). Foraging theory. Princ-
Padoa-Schioppa, C., & Assad, J. A. (2006). Neurons in the eton, NJ: Princeton University Press.
orbitofrontal cortex encode economic value. Nature, van Hoesen, G. W., Morecraft, R. J., & Vogt, B. A. (1993).
441(7090), 223–226. Connections of the monkey cingulate cortex. In B. A. Vogt
Philiastides, M. G., Biele, G., & Heekeren, H. R. (2010). A & M. Gabriel (Eds.), Neurobiology of cingulate cortex and limbic
mechanistic account of value computation in the human thalamus. Boston, MA: Birkhauser.
brain. Proc Natl Acad Sci USA, 107(20), 9430–9435. Walton, M. E., Bannerman, D. M., Alterescu, K., &
Picard, N., & Strick, P. L. (1996). Motor areas of the medial Rushworth, M. F. S. (2003). Functional specialization
wall: A review of their location and functional activation. within medial frontal cortex of the anterior cingulate
Cereb Cortex, 6, 342–353. for evaluating effort-related decisions. J Neurosci, 23,
Quilodran, R., RothÉ, M., & Procyk, E. (2008). Behavioral 6475–6479.
shifts and action valuation in the anterior cingulate cortex. Walton, M. E., Bannerman, D. M., & Rushworth, M. F. S.
Neuron, 57(2), 314–325. (2002). The role of rat medial frontal cortex in effort-based
Rothé, M., Quilodran, R., Sallet, J., & Procyk, E. (2011). decision making. J Neurosci, 22(24), 10996–11003.
Coordination of high gamma activity in anterior cingulate Walton, M. E., Behrens, T. E., Buckley, M. J., Rudebeck, P.
and lateral prefrontal cortical areas during adaptation. H., & Rushworth, M. F. (2010). Separable learning systems
J Neurosci, 31(31), 11110–11117. in the macaque brain and the role of orbitofrontal cortex
Rudebeck, P. H., Behrens, T. E., Kennerley, S. W., in contingent learning. Neuron, 65(6), 927–939.
Baxter, M. G., Buckley, M. J., Walton, M. E., & Walton, M. E., Groves, J., Jennings, K. A., Croxson, P. L.,
Rushworth, M. F. (2008). Frontal cortex subregions play Sharp, T., Rushworth, M. F., & Bannerman, D. M. (2009).
distinct roles in choices between actions and stimuli. J Neu- Comparing the role of the anterior cingulate cortex and
rosci, 28(51), 13775–13785. 6-hydroxydopamine nucleus accumbens lesions on operant
Rudebeck, P. H., & Murray, E. A. (2011). Dissociable effects effort-based decision making. Eur J Neurosci, 29(8),
of subtotal lesions within the macaque orbital prefrontal 1678–1691.
cortex on reward-guided behavior. J Neurosci, 31(29), Wang, X. J. (2002). Probabilistic decision making by slow
10569–10578. reverberation in cortical circuits. Neuron, 36(5), 955–968.
Rudebeck, P. H., Walton, M. E., Smyth, A. N., Bannerman, Wunderlich, K., Dayan, P., & Dolan, R. J. (2012). Mapping
D. M., & Rushworth, M. F. (2006). Separate neural value based planning and extensively trained choice in the
pathways process different decision costs. Nat Neurosci, 9(9), human brain. Nat Neurosci, 15(5), 786–791.
1161–1168.
ABSTRACT Actions are distinguished from movements on the capacity (e.g., those related to growth, development,
basis of their being goal-directed. At any given moment, the skill acquisition, senescence, and injury or illness). To
possibilities available for goal-directed action are defined by address this shortcoming, we introduce the concept of
a complex dynamic between perceived attributes of the envi-
ronment and the actor’s unique goals and behavioral capaci- reciprocal affordance, which places the strengths of J. J.
ties. While much attention has been given to the perceptual Gibson’s insights (Gibson, 1979) within a modern cog-
side, cognitive neuroscience has all but overlooked the essen- nitive neuroscience framework. Our starting point will
tial role of the actor. For instance, a keyboard may enable an be a brief introduction to several influential contempo-
individual to type a letter, but not if he or she is wearing rary theories of action, each of which has moved the
mittens, has no experience with electronics, or suffers from
paralysis. Theories of action must accommodate the contex-
field forward significantly while also failing to address
tual nature of the situation. Here, we introduce the concept the dynamic interplay between environment and actor
of reciprocal affordance, which explicitly acknowledges that adequately. We then review evidence supporting recip-
actors actively shape, and are shaped by, their environments. rocal affordance as a new and important conceptual
This offers a way to integrate a diversity of factors that influ- framework, and identify a number of areas that may
ence our actions, including development, disability, and the
prove especially amenable to this perspective.
use of technology.
The characteristic that distinguishes action from move- Contemporary theories of action
ment is its goal-directedness, and how we represent and
select actions is the fundamental problem in under- The emergence of action as an area of inquiry, distinct
standing goal-directed behavior. There is consensus from perception and motor control, has yielded a
that solving this puzzle involves understanding percep- number of significant advances. Here, we briefly con-
tual processing of objects and surfaces, on the one sider treatment of the actor in shaping goal-directed
hand, and volitional motor planning and control on the behavior by several of the more influential theoretical
other. However, this dichotomy, while experimentally perspectives.
convenient, is suspect. Evidence is accumulating in For more than two decades, Goodale and Milner’s
both the behavioral sciences and neurosciences that (1991) dual systems theory has been the dominant
perceptual, motor, and, indeed, a broad set of action- theoretical framework. The key idea is that two func-
related cognitive functions are intertwined to the point tionally and anatomically separable pathways are
of being largely inseparable. As a consequence, action involved in the representation of visual information
science must grapple with challenging issues that span for perceptual judgments (occipital-temporal ventral
all three of these research domains. stream) versus goal-directed actions (occipital-parietal
Several decades of action research have yielded a dorsal stream). There is extensive evidence for these
deep understanding of perceptual processes that two pathways and their distinct functional properties.
mediate motor behavior. Very little attention, however, Most notable is patient D. F., whose remarkable preser-
has been given to the fact that actions are determined vation of visually guided action in spite of severe apper-
by the relationship between perceptual attributes of the envi- ceptive agnosia was the impetus behind the insight that
ronment and the unique cognitive and motor capabilities of perception for action is supported by mechanisms dis-
the actor. The fact that actors, like their environments, tinct from those involved in perceptual judgments.
are highly dynamic has been almost entirely ignored. Despite its many virtues, this approach is heavily focused
This dynamism is due to rapid changes in goal or moti- on perception of the environment and gives short shrift
vational states and access to tools or other technologies, to the role of the actor in shaping actions. Little atten-
as well as to more gradual alterations in behavioral tion is devoted, for instance, to the role of the actor’s
47 MANNING, KAHANA,
AND NORMAN 557
48 UNCAPHER, GORDON,
AND WAGNER 567
524 memory
and functional imaging evidence for the role of pre- are increasingly considering memory as decision
frontal cortex and medial temporal lobe structures in making, wherein perception ultimately guides action
representing context. through decision processes that weigh retrieved mne-
Given the established dependence of episodic monic evidence in relation to decision criteria.
memory on medial temporal lobe binding and reactiva- From the multiple memory systems perspective, dif-
tion computations, one of the more surprising develop- ferent forms of memory are supported by distinct com-
ments in recent years is the observation that fMRI putations and neural substrates. One central distinction
activity in multiple regions of lateral parietal cortex is between declarative memory, which consists of epi-
consistently varies with episodic retrieval behavior. It is sodic and semantic memory, and procedural memory,
perhaps unsurprising that the medial temporal lobe is which underlies habits and skilled behavior. Further
not the only region to exhibit activity that covaries with illustrating the intersection between theorizing about
retrieval processing, given that binding and reactiva- memory and decision making, Shohamy and Daw con-
tion, in isolation, are insufficient to support memory- sider the evidence for these distinct memory systems in
guided behavior. Indeed, to remember and to act on parallel with the literature on decision making and
the products of remembering requires multiple addi- learning from reward. These authors highlight two
tional processes, from attending to the cues that trigger exciting theoretical developments. First, they discuss
memory retrieval, to maintaining reactivated informa- how decision research on reward learning has speci-
tion that may serve as the basis for memory-guided fied a neural mechanism for learning about actions,
decisions, to integrating reactivated evidence with deci- and they consider how this mechanism may partly
sion criteria, and ultimately to selecting an appropriate underlie the habits and skills (procedural memory)
action. Two chapters in this section consider the role of that have been the focus of much of memory research.
parietal cortex in episodic retrieval through the broader Second, they discuss how important aspects of reward-
lens of remembering as decision making. Uncapher, mediated decision behavior are not well accounted for
Gordon, and Wagner introduce a number of promi- by this “habit-like” mechanism, sparking interest in
nent hypotheses recently advanced to account for understanding the mechanisms governing goal repre-
lateral parietal contributions to episodic retrieval and sentation and goal-directed action learning. Highlight-
briefly review the literature in relation to these hypoth- ing the reciprocal influences between decision and
eses. These authors argue that any model of lateral memory research, Shohamy and Daw consider how
parietal contributions to retrieval must take into account declarative memory mechanisms may support goal-
the rich, fine-grained functional heterogeneity evident directed action learning. The explanatory power of
in lateral parietal cortex, and they offer a working viewing memory and decision behavior as linked prom-
model of lateral parietal function that posits that mul- ises continued integration of these fields, and the
tiple parietal processes—including processes of atten- emergence of models in which core mechanisms of
tion, multifeatural binding and maintenance, decision memory and decision making are isomorphic.
making, and action intention—as well as interactions Memory as decision making directly stems from the
between these processes ultimately support memory- demands of life, in which people and other organisms
guided behavior. Miller and Dobbins similarly argue constantly face the challenge of representing the
that memory inherently entails decision making, and as current state of the world and determining how to act
such many of the neural correlates of episodic retrieval to achieve goals and optimize outcomes. Learning
evidenced by functional imaging likely reflect various enables organisms to draw on predictive models of how
decision and control processes, rather than memory past states of the world mapped to outcomes, such that
reactivation processes per se. These authors review an memory-based predictions shape current interpreta-
emerging literature that suggests that lateral prefrontal tions of the world and favor selection of appropriate
and parietal cortex may contribute to memory-guided actions. While the conceptualization of memory as pre-
behavior, in part through control processes that tune diction has a long history, this view is increasingly influ-
and bias memory judgments. Miller and Dobbins posit ential in theorizing about memory. Not only is it central
a biasing and orienting framework wherein frontopari- to theories of procedural memory (habits and skills),
etal processes bias memory-guided judgments for more priming, and other nondeclarative forms of memory,
efficient processing when biases are confirmed and, the idea has also emerged as central to theories of epi-
alternatively, support attentional orienting and more sodic memory, wherein the recollection of details of a
thorough exploration of memory contents when biases past event may be viewed as an episodic-based predic-
are disconfirmed. Both chapters illustrate how efforts tion about what the present may hold. Strikingly, a
to understand the cognitive neuroscience of memory recent theme in memory research extends this idea to
526 memory
44 Short-Term Memory: Neural
Mechanisms, Brain Systems, and
Cognitive Processes
CHARAN RANGANATH, MICHAEL E. HASSELMO, AND CHANTAL E. STERN
ABSTRACT The idea that memory across delays of a few seconds, “long term” to refer to different concepts and time
or short-term memory (STM), can be distinguished from scales. For example, in the synaptic plasticity literature,
memory across longer delays, or long-term memory (LTM),
the terms short term and long term are used to refer to
has a long history but remains highly controversial. The con-
flict has come about, in part, because of the challenges in longer time scales.
integrating findings from studies of STM at different levels of It is probably fair to say that most cognitive neurosci-
analysis, using different task paradigms, and in different entists have been exposed to the general idea that LTM
species. A review of this literature reveals that there is no (i.e., the ability to retain information across longer
single neural mechanism, brain system, or cognitive process delays on the order of minutes or longer) is supported
that supports performance on STM tasks. Furthermore, at the
neural level, STM and LTM may be intricately linked. by lasting, experience-dependent changes in synaptic
strength in the medial temporal lobe (MTL) memory
“Memory is a convenient chapter heading designating certain system, and that this system is not necessary for reten-
kinds of problems that scientists study. Methods of science tion across very short delays. Instead, the early view held
have been brought to bear on the problems of memory for by some was that STM (defined here, operationally, as
over a hundred years, in many different organisms, and at the ability to retain information across delays of several
many different levels of analysis, extending from molecular
mechanisms to the phenomena of conscious awareness.”
seconds) was supported by the prefrontal cortex (PFC).
At present, few researchers who study STM and LTM
—Endel Tulving, introduction to “Memory” section of the
first Cognitive Neurosciences (1995) would endorse this view, as it presents an overly simplis-
tic, and largely incorrect, mapping between neural
The study of short-term memory serves as a beautiful mechanisms (persistent activity for STM vs. synaptic
example of both the stunning progress in cognitive plasticity for LTM1), brain systems (PFC for STM vs.
neuroscience over the past 50 years, and the number of MTL for LTM), cognitive processes (active maintenance
open questions that remain. Researchers have made vs. episodic memory encoding and retrieval), and con-
great strides by relating work from elegant behavioral scious experiences (the attention-demanding process
paradigms in animal models, studies on humans with of having something remain present in consciousness
brain damage, and neuroimaging studies of activity in vs. the experience of becoming aware of information
the healthy human brain. These findings have yielded that is associated with a sense of a different time and
a series of answers to very specific questions posed at place).
different levels of analysis. Our current challenge is to In this chapter, we will summarize available evidence
bridge these different levels of analysis. regarding the neural mechanisms and brain systems
Here, we use the terms “short-term memory” (STM) that support STM. We start off by reviewing the basic
and “long-term memory” (LTM) as operational terms
to describe memory performance at various retention 1
Note that we are not using the term “working memory”
intervals, with the understanding that multiple pro- (WM) in this chapter, though many researchers equate the
cesses and mechanisms (to be considered in this term WM with STM. This is because we are describing memory
chapter) could differentially support performance on mechanisms on short time scales independent of their use for
cognitive processes. In contrast, we use WM in a manner that
STM and LTM measures. We use STM to refer to time
is consistent with its use by Baddeley and others—that is, to
scales up to tens of seconds, and LTM to refer to describe various cognitive processes that might support STM
memory spanning minutes or longer. It is important to performance. The definitions of STM and WM are reviewed
note that different literatures use “short term” and in more detail in Aben, Stapert, and Blokland (2012).
528 memory
perirhinal cortex (Navaroli, Zhao, Boguszewski, & Brown, called “short-term potentiation” or “post-tetanic poten-
2012), and hippocampus (Knauer, Jochems, Valero- tiation” and can be induced by calcium influx in pre-
Aracama, & Yoshida, 2013). A similar plateau-potential synaptic terminals (Bliss & Lomo, 1973; McNaughton,
phenomenon has also been described in layer V of Douglas, & Goddard, 1978). Even if the rapidly decay-
prefrontal cortical slices (Haj-Dahmane & Andrade, 1998) ing synaptic modification does not depend upon both
and in the cingulate cortex (Zhang & Seguela, 2010). presynaptic and postsynaptic activity, it can be used to
In the entorhinal cortex, plateau potentials appear to store memory (Mongillo, Barak, & Tsodyks, 2008).
arise from a calcium-sensitive nonspecific cation current Alternately, any short-term change in excitability could
(CAN current) activated by muscarinic acetylcholine allow selective reactivation of the most recently pre-
receptors (Egorov et al., 2002; Fransén, Alonso, & sented stimulus, including the CAN current described
Hasselmo, 2002; Fransén et al., 2006; Klink & Alonso, above that can decay over a period of seconds when it
1997) that strongly depolarizes neurons. In the pres- does not cause persistent spiking (Klink & Alonso,
ence of acetylcholine, the generation of spiking by 1997; Yoshida & Hasselmo, 2009).
intracellular current injection or synaptic stimulation Neuronal oscillations have been proposed to contrib-
causes the neuron to enter an internal regenerative ute to another potential STM mechanism. For instance,
cycle of sustained spiking. Each new spike activates Lisman and colleagues proposed a model in which indi-
voltage-sensitive calcium channels, and the new influx vidual items could be transiently maintained by activa-
of calcium activates the CAN current (Fransén et al., tion of specific ensembles of neurons within a single
2002). This current causes additional depolarization, cycle of a gamma (30–80 Hz) oscillation (Jensen &
leading to another spike, which again activates voltage- Lisman, 2005; Lisman & Idiart, 1995; Lisman & Jensen,
sensitive calcium channels that further perpetuate the 2013). In the model, activation of multiple item repre-
cycle. The cellular processes in the entorhinal and peri- sentations is regulated by low-frequency theta oscilla-
rhinal cortex provide a mechanism of persistent spiking tions (4–8 Hz in the human), such that each item
suitable for retaining representations of novel stimuli representation is sequentially activated during a differ-
in an active state (Hasselmo & Stern, 2006), which ent theta phase (Lisman & Idiart, 1995). Afterdepolar-
could also underlie some of the data from functional ization due to mechanisms such as the CAN current
MRI (fMRI) and unit-recording studies in parahippo- allow the reactivation of the representation on the next
campal structures described below. theta cycle, with first-in first-out replacement of items
The cellular mechanisms for generation of persistent that can model the precession of place cell spiking activ-
activity could not only support the temporary retention ity relative to theta (Jensen & Lisman, 1996; Koene &
of novel information, but also drive synaptic changes Hasselmo, 2007; Lisman & Idiart, 1995; Navratilova,
that can support retention over long delays (Hasselmo Giocomo, Fellous, Hasselmo, & McNaughton, 2012). In
& Stern, 2006; Jensen, Idiart, & Lisman, 1996; Jensen & this framework, the afterdepolarization corresponds to
Lisman, 2005). Specifically, persistent spiking could a mechanism of short-term memory for maintaining
generate strengthening of synaptic connections based representations for spatial location (Navratilova et al.,
on presynaptic and postsynaptic activity (Jansen et al., 2012; de Almeida, Idiart, Villavicencio, & Lisman,
1996; Jensen & Lisman, 2005), thereby forming pat- 2012).
terns of synaptic connectivity that could allow a similar Jensen and Lisman (2005) argued that the sequential
presynaptic cue to evoke the associated postsynaptic activation of neocortical item representations on each
activity. The strengthening of long-term synaptic con- theta cycle could support both temporary maintenance
nections can mediate LTM performance by allowing of multiple items and also the long-term retention of
later activity to cue the retrieval of an associated item temporal sequence information. The temporal distance
(Hasselmo et al., 1995; Hasselmo & Wyble, 1997; Jensen between activation of successive item representations in
et al., 1996; McNaughton & Morris, 1987; Treves & their model is substantially compressed to ∼20 ms,
Rolls, 1992). Once retrieval of a long-term memory has which corresponds roughly to the duration of one cycle
been cued, the pattern of recurrent excitation could of gamma oscillation and is within the time window
allow the retrieved memory to be held in active main- required for long-term potentiation (LTP; Bi & Poo,
tenance for an arbitrary period of time (Hasselmo & 1998; Markram, Lubke, Frotscher, & Sakmann, 1997).
Wyble, 1997; Zilli & Hasselmo, 2008). Thus, the sequential activation of items in a theta cycle
As an alternative to the persistent activity mechanisms could allow for the item representations to become
described above, STM could also be supported by tran- directly linked through Hebbian plasticity mechanisms
sient changes in synaptic efficacy that decay over a and, as a result, support long-term retention of multi-
period of seconds. This form of synaptic change is item memories. The idea that single-unit activity and
530 memory
is that areas in the MTL (like other cortical areas) shown that the hippocampus is critical for STM under
play a role in STM processes. This idea was controver- certain circumstances. Several fMRI studies, for instance,
sial because initial studies (Corkin, 1984; Scoville & have shown a role for the hippocampus in the active
Milner, 1957) showed that patients with MTL damage maintenance of novel, complex stimuli. The first of
exhibit a normal capacity for short-term retention of these studies, by Stern and colleagues (2001), investi-
simple stimuli or recall of sequences of simple digits gated activity while participants performed a “two-back”
or spatial locations (i.e., “immediate serial recall”). task with novel, complex scene stimuli. On each trial,
These studies, along with more recent findings, participants were required to decide whether each
suggest that patients with MTL damage can show scene was the same as one presented two trials previ-
severe impairments on LTM measures but still retain ously. The hippocampus showed increased activity
simple information, such as the locations of colored during blocks with novel scenes, as compared to blocks
squares, across short delays (e.g., Cave & Squire, with scenes that were highly familiar. Stern et al. also
1992). Given the findings described above, however, examined activation in the hippocampus during perfor-
this is not surprising because simple, familiar, and mance of a target detection task that placed minimal
highly discriminable stimuli could be maintained by demands on STM maintenance. Hippocampal activa-
persistent activity of representations in cortical areas tion did not differentiate between blocks of target
outside of the MTL. The important question, then, is detection with novel stimuli and blocks with highly
whether there are any conditions under which MTL familiar stimuli. This finding suggests that hippocampal
regions do contribute to STM. activation during the STM task was not driven by passive
It is, in fact, quite clear that neocortical areas in the processing of novel stimuli, but rather by the demand
MTL—that is, the perirhinal, entorhinal, and parahip- to actively attend to and actively maintain these stimuli.
pocampal cortex—play a critical role in STM when the In a separate set of experiments, Ranganath &
relevant information is being actively maintained. Early D’Esposito (2001) used event-related fMRI to investi-
evidence for this idea came from studies of neurons in gate the neural correlates of STM for novel faces. In the
“inferior temporal” areas (e.g., Fuster & Jervey, 1982), first experiment, participants performed a delayed-
including perirhinal cortex (E. K. Miller, Erickson, & recognition task, in which a novel sample face was to be
Desimone, 1996; Naya, Yoshida, Takeda, Fujimichi, maintained across a 7-sec delay in anticipation of a
& Miyashita, 2003), that showed persistent, object- recognition probe. Critically, as in the Stern et al., study,
selective activity during short retention delays. Similar the stimuli presented on each trial were novel and not
effects have been reported in entorhinal cortical repeated on subsequent trials. This study found that
neurons (Suzuki, Miller, & Desimone, 1997). Addition- hippocampal activation was increased during the
ally, damage to the perirhinal cortex has been associ- memory delay, consistent with a role in maintaining the
ated with severe impairments in the retention of object face stimulus. Control conditions demonstrated that
information, even across very short delays of a few the hippocampal activation seen during the STM task
seconds (Baxter & Murray, 2001), and humans with was not driven simply by the demand to encode or
extensive perirhinal damage show comparable impair- perform recognition decisions on faces. In the second
ments in STM for complex objects (Buffalo, Reber, & experiment, the authors replicated the findings of hip-
Squire, 1998). These findings make sense, as most of pocampal activation during maintenance of novel faces
the available evidence suggests that perirhinal cortex in a new sample of participants, and extended it by
represents detailed information about the perceptual demonstrating that delay-period activation in the same
and semantic characteristics of objects (Burke et al., region was increased during maintenance of novel
2012; Murray, Bussey, & Saksida, 2007; Ranganath & faces, as compared with familiar faces.
Ritchey, 2012; Tyler et al., 2013). Although less is known The findings from the studies by Stern et al. (2001)
about the parahippocampal cortex, this area shows per- and Ranganath & D’Esposito (2001) prompted several
sistent activity when visual scene information is actively groups to reexamine the question of whether STM
maintained (Schon et al., 2004; Schon et al., 2005; is intact following MTL damage. For example, one
Stern, Sherman, Kirchhoff, & Hasselmo, 2001), and group replicated the fMRI findings of Ranganath and
damage to this area has been associated with spatial D’Esposito (2001) and also demonstrated that amnesic
STM impairments (e.g., Hartley et al., 2007; Pierrot- participants with medial temporal lobe damage due to
Deseilligny, Muri, Rivaud-Pechoux, Gaymard, & Ploner, anoxia or encephalitis were impaired at this task, even
2002; Ploner et al., 2000). though it only required retention across a 7-sec delay
The role of the hippocampus proper in STM pro- (Nichols, Kao, Verfaillie, & Gabrieli, 2006). A similar
cesses is more controversial, but several studies have study conducted by Olson and colleagues also found
532 memory
objects within a complex scene (Hannula et al., 2006). 2013; MacDonald, Lepage, Eden, & Eichenbaum, 2011;
In a follow-up study, these researchers demonstrated Pastalkova, Itskov, Amarasingham, & Buzsaki, 2008).
that the effect was not limited to spatial relations The selective firing response to temporal intervals
by showing that patients were also impaired when could allow learning of items or events that occur at
tested on immediate memory for arbitrary associations specific time points (Hasselmo, 2012). This could
between faces and scenes. Another study investigated mediate the role of hippocampus in memory for tem-
memory for object-location associations (Olson, Page, poral order (Hunsaker & Kesner, 2008). Computational
et al., 2006) by using a paradigm adapted from an modeling shows how these responses could arise from
imaging study (Mitchell et al., 2000), in which subjects persistent spiking activity of neurons in the entorhinal
remembered the locations of objects in a two- cortex (Hasselmo, 2012).
dimensional 3 × 3 grid. Relative to healthy controls,
amnesic patients with MTL damage were significantly
impaired at retaining the object-location associations Cognitive processes that may support
across an 8-sec delay. Similar results were found in other short-term memory
studies of object-location binding (Braun et al., 2011;
Finke et al., 2011) or scene memory (Hartley et al., In the 1950s and 1960s, several researchers developed
2007; King, Burgess, Hartley, Vargha-Khadem, & models proposing that information that is active could
O’Keefe, 2002). Collectively, these studies suggest that be held in a capacity-limited short-term store, and that
short-term retention of relational information may the act of processing this information would result in
take place in the same areas implicated in long-term the development of a memory trace that could be
retention. accessed even after long delays. Interestingly, these
Additional evidence (Axmacher et al., 2009; Schon, models did not necessarily assume that short- and long-
Quiroz, Hasselmo, & Stern, 2009) suggests that MTL term stores were supported by different brain regions.
regions may be involved in STM tasks that use complex For instance, Atkinson & Shiffrin (1971) noted: “One
stimuli and manipulate the number of items to be might consider the short-term store simply as being a
retained (“memory load”). Using intracranial record- temporary activation of some portion of the long-term
ings in humans during a WM task with face stimuli, store” (p. 278).
Axmacher and colleagues (2007) demonstrated that Some of the evidence for dual-store models came
hippocampal and MTL cortical gamma power increased from analyses of serial position effects in verbal learning
with increasing STM load, and they subsequently studies. Specifically, the likelihood of recalling a word
reported modulation of hippocampal gamma power by from a previously studied list is increased for words at
theta phase when multiple items were maintained the beginning (primacy) and end (recency) of the list.
(Axmacher et al., 2010). Several fMRI studies have also Whereas some manipulations disproportionately impact
shown increases in hippocampal activity during memory the primacy effect and recall of middle-list items (e.g.,
delays with increasing memory load (e.g., Axmacher presentation rate) others disproportionately affect the
et al., 2007; Schon et al., 2009; Schon et al., 2013). A recency effect (e.g., lag between end of list and recall
functional connectivity study using a Sternberg task test). Intuitively, it would seem sensible to assume that
with unfamiliar faces reported that the correlation of primacy and middle-item memory is an index of LTM,
activity between the inferior frontal gyrus (IFG) and the whereas recency is additionally influenced by processes
hippocampus increases with increasing STM load that support STM (e.g., phonological rehearsal). In
during a delay period (Rissman, Gazzaley, & D’Esposito, fact, more recent studies have shown that the magni-
2008). Collectively, the results of these studies suggest tude of the primacy effect is actually influenced by
encoding and retention of higher loads across a delay rehearsal (Tan & Ward, 2000), and that robust recency
results in activity within the MTL, including roles for effects can be observed even when phonological
the hippocampus, entorhinal, and perirhinal cortices. rehearsal is not feasible (see Howard & Kahana, 1999,
Although it is clear that sequences of verbalizable for review). Indeed, the factors influencing primacy
items can be maintained independently of the hippo- and recency effects remain controversial. Some suggest
campus (as in the digit span task), it is possible that the that these effects can largely be accounted for by a
hippocampus may nonetheless play a special role in single store (Sederberg, Howard, & Kahana, 2008),
maintenance of temporal sequence information. Recent whereas others suggest that a temporary activation
data show that neurons in the hippocampus may code buffer (akin to the idea proposed by Hebb, 1949) addi-
different temporal intervals within behavioral tasks tionally contributes to recency (Davelaar, Goshen-
(Kraus, Robinson, White, Eichenbaum, & Hasselmo, Gottstein, Ashkenazi, Haarmann, & Usher, 2005).
534 memory
Baxter, M. G., & Murray, E. A. (2001). Opposite relation- Diana, R. A., Yonelinas, A. P., & Ranganath, C. (2007).
ship of hippocampal and rhinal cortex damage to delayed Imaging recollection and familiarity in the medial temporal
nonmatching-to-sample deficits in monkeys. Hippocampus, lobe: A three-component model. Trends Cogn Sci, 11(9),
11(1), 61–71. 379–386.
Bi, G., & Poo, M. (1998). Synaptic modification in cultured Egorov, A. V., Hamam, B. N., Fransén, E., Hasselmo, M. E.,
hippocampal neurons: Dependence on spike timing, syn- & Alonso, A. A. (2002). Graded persistent activity in ento-
aptic strength and postsynaptic cell type. J Neurosci, 18(24), rhinal cortex neurons. Nature, 420(6912), 173–178.
10464–10472. Eichenbaum, H., Yonelinas, A. R., & Ranganath, C. (2007).
Bliss, T. V., & Lomo, T. (1973). Long-lasting potentiation of The medial temporal lobe and recognition memory. Annu
synaptic transmission in the dentate area of the anaesthe- Rev Neurosci, 30, 123–152.
tized rabbit following stimulation of the perforant path. Felleman, D. J., & Van Essen, D. C. (1991). Distributed hier-
J Physiol, 232(2), 331–356. archical processing in the primate cerebral cortex. Cereb
Bragin, A., Jando, G., Nadasdy, Z., Hetke, J., Wise, K., & Cortex, 1(1), 1–47.
Buzsaki, G. (1995). Gamma (40–100 Hz) oscillation in the Finke, C., Ostendorf, F., Braun, M., & Ploner, C. J. (2011).
hippocampus of the behaving rat. J Neurosci, 15, 47–60. Impaired representation of geometric relationships in
Braun, M., Weinrich, C., Finke, C., Ostendorfm F., humans with damage to the hippocampal formation. PLoS
Lehmann, T. N., & Ploner, C. J. (2011). Lesions affecting One, 6(5), e19507.
the right hippocampal formation differentially impair Fransén, E., Alonso, A. A., & Hasselmo, M. E. (2002). Simu-
short-term memory of spatial and nonspatial associations. lations of the role of the muscarinic-activated calcium-
Hippocampus, 21(3), 309–318. sensitive nonspecific cation current INCM in entorhinal
Brown, M. W., & Xiang, J. Z. (1998). Recognition memory: neuronal activity during delayed matching tasks. J Neurosci,
Neuronal substrates of the judgement of prior occurrence. 22(3), 1081–1097.
Prog Neurobiol, 55(2), 149–189. Fransén, E., Tahvildari, B., Egorov, A. V., Hasselmo,
Buffalo, E. A., Reber, P. J., & Squire, L. R. (1998). The M. E., & Alonso, A. A. (2006). Mechanisms of graded per-
human perirhinal cortex and recognition memory. Hippo- sistent cellular activity of entorhinal cortex layer V neurons.
campus, 8(4), 330–339. Neuron, 49(5), 735–746.
Burke, S. N., Maurer, A. P., Hartzell, A. L., Nematollahi, Funahashi, S., Bruce, C. J., & Goldman-Rakic, P. S. (1991).
S., Uprety, A., Wallace, J. L., & Barnes, C. A. (2012). Neuronal activity related to saccadic eye movements in the
Representation of three-dimensional objects by the rat peri- monkey’s dorsolateral prefrontal cortex. J Neurophysiol,
rhinal cortex. Hippocampus, 22(10), 2032–2044. 65(6), 1464–1483.
Cave, C. B., & Squire, L. R. (1992). Intact verbal and non- Fuster, J. M. (1973). Unit activity in prefrontal cortex during
verbal short-term memory following damage to the human delayed-response performance: Neuronal correlates of
hippocampus. Hippocampus, 2(2), 151–163. transient memory. J Neurophysiol, 36, 61–78.
Cohen, M. A., & Grossberg, S. (1983). Absolute stability of Fuster, J. M. (1995). Memory in the cerebral cortex. Cambridge,
global pattern formation and parallel memory storage by MA: MIT Press.
competitive neural networks. IEEE Trans Syst Man Cybern, Fuster, J. M., & Alexander, G. E. (1971). Neuron activity
13, 815–826. related to short-term memory. Science, 173, 652–654.
Cohen, N. J., & Eichenbaum, H. (1993). Memory, amnesia, and Fuster, J. M., & Jervey, J. P. (1982). Neuronal firing in the
the hippocampal system. Cambridge, MA: MIT Press. inferotemporal cortex of the monkey in a visual memory
Colgin, L. L., Denninger, T., Fyhn, M., Hafting, T., task. J Neurosci, 2, 361–375.
Bonnevie, T., Jensen, O., … Moser, E. I. (2009). Frequency Hafting, T., Fyhn, M., Molden, S., Moser, M. B., & Moser,
of gamma oscillations routes flow of information in the E. I. (2005). Microstructure of a spatial map in the entorhi-
hippocampus. Nature, 462, 353–357. nal cortex. Nature, 436(7052), 801–806.
Corkin, S. (1984). Lasting consequences of bilateral medial Haj-Dahmane, S., & Andrade, R. (1998). Ionic mechanism
temporal lobectomy: Clinical course and experimental of the slow afterdepolarization induced by muscarinic
findings in H. M. Semin Neurol, 4, 249–259. receptor activation in rat prefrontal cortex. J Neurophysiol,
Cowan, N. (1997). Oxford psychology series, Vol. 26: Attention and 80(3), 1197–1210.
memory: An integrated framework. New York, NY: Oxford Hannula, D. E., Libby, L. A., Yonelinas, A. P., &
University Press. Ranganath, C. (2013). Medial temporal lobe contribu-
D’Esposito, M., & Postle, B. R. (1999). The dependence of tions to cued retrieval of items and contexts. Neuropsycholo-
span and delayed-response performance on prefrontal gia, 51(12), 2322–2332.
cortex. Neuropsychologia, 37(11), 1303–1315. Hannula, D. E., & Ranganath, C. (2008). Medial temporal
Davelaar, E. J., Goshen-Gottstein, Y., Ashkenazi, A., lobe activity predicts successful relational memory binding.
Haarmann, H. J., & Usher, M. (2005). The demise of J Neurosci, 28(1), 116–124.
short-term memory revisited: Empirical and computational Hannula, D. E., Tranel, D., & Cohen, N. J. (2006). The long
investigations of recency effects. Psychol Rev, 112(1), 3–42. and the short of it: Relational memory impairments
De Almeida, L., Idiart, M., Villavicencio, A., & Lisman, J. in amnesia, even at short lags. J Neurosci, 26(32),
(2012). Alternating predictive and short-term memory 8352–8359.
modes of entorhinal grid cells. Hippocampus, 22(8), Harrison, S. A., & Tong, F. (2009). Decoding reveals the
1647–1651. contents of visual working memory in early visual areas.
Deshmukh, S. S., & Knierim, J. J. (2011). Representation of Nature, 458(7238), 632–635.
non-spatial and spatial information in the lateral entorhinal Hartley, T., Bird, C. M., Chan, D., Cipolotti, L., Husain,
cortex. Front Behav Neurosci, 5, 69. M., Vargha-Khadem, F., & Burgess, N. (2007). The
536 memory
subdivisions of monkey inferotemporal cortex during pair poral interactions with increasing visual working memory
association memory task. Eur J Neurosci, 18(10), 2915–2918. load. Cereb Cortex, 18(7), 1618–1629.
Nichols, E. A., Kao, Y. C., Verfaellie, M., & Gabrieli, J. D. Romo, R., Hernandez, A., Zainos, A., Lemus, L., & Brody,
(2006). Working memory and long-term memory for C. D. (2002). Neuronal correlates of decision-making in
faces: Evidence from fMRI and global amnesia for involve- secondary somatosensory cortex. Nat Neurosci, 5(11),
ment of the medial temporal lobes. Hippocampus, 16(7), 1217–1225.
604–616. Ruchkin, D. S., Grafman, J., Cameron, K., & Berndt, R. S.
O’Keefe, J., & Burgess, N. (2005). Dual phase and rate (2003). Working memory retention systems: A state of acti-
coding in hippocampal place cells: Theoretical significance vated long-term memory. Behav Brain Sci, 26(6), 709–728;
and relationship to entorhinal grid cells. Hippocampus, discussion 728–777.
15(7), 853–866. Schon, K., Atri, A., Hasselmo, M. E., Tricarico, M. D.,
Olson, I. R., Moore, K. S., Stark, M., & Chatterjee, A. LoPresti, M. L., & Stern, C. E. (2005). Scopolamine
(2006). Visual working memory is impaired when the reduces persistent activity related to long-term encoding in
medial temporal lobe is damaged. J Cogn Neurosci, 18(7), the parahippocampal gyrus during delayed matching in
1087–1097. humans. J Neurosci, 25(40), 9112–9123.
Olson, I. R., Page, K., Moore, K. S., Chatterjee, A., & Schon, K., Hasselmo, M. E., LoPresti, M. L., Tricarico,
Verfaellie, M. (2006). Working memory for conjunctions M. D., & Stern, C. E. (2004). Persistence of parahippocam-
relies on the medial temporal lobe. J Neurosci, 26(17), pal representation in the absence of stimulus input
4596–4601. enhances long-term encoding: A functional magnetic reso-
Pastalkova, E., Itskov, V., Amarasingham, A., & Buzsaki, nance imaging study of subsequent memory after a delayed
G. (2008). Internally generated cell assembly sequences in match-to-sample task. J Neurosci, 24(49), 11088–11097.
the rat hippocampus. Science, 321(5894), 1322–1327. Schon, K., Quiroz, Y. T., Hasselmo, M. E., & Stern, C. E.
Pasternak, T., & Greenlee, M. W. (2005). Working memory (2009). Greater working memory load results in greater
in primate sensory systems. Nat Rev Neurosci, 6(2), 97–107. medial temporal activity at retrieval. Cereb Cortex, 19,
Piekema, C., Kessels, R. P., Mars, R. B., Petersson, K. M., & 2561–2571.
Fernandez, G. (2006). The right hippocampus participates Schon, K., Ross, R. S., Hasselmo, M. E., & Stern, C. E.
in short-term memory maintenance of object-location asso- (2013). Complementary roles of medial temporal lobes and
ciations. NeuroImage, 33(1), 374–382. mid-dorsolateral prefrontal cortex for working memory for
Pierrot-Deseilligny, C., Muri, R. M., Rivaud-Pechoux, S., novel and familiar trial-unique visual stimuli. Eur J Neurosci,
Gaymard, B., & Ploner, C. J. (2002). Cortical control of 37(4), 668–678.
spatial memory in humans: The visuooculomotor model. Scoville, W. B., & Milner, B. (1957). Loss of recent memory
Ann Neurol, 52(1), 10–19. after bilateral hippocampal lesions. J Neurol Neurosurg Psych,
Ploner, C. J., Gaymard, B. M., Rivaud-Péchoux, S., 20, 11–21.
Baulac, M., Clémenceau, S., Samson, S., & Pierrot- Sederberg, P. B., Howard, M. W., & Kahana, M. J. (2008).
Deseilligny, C. (2000). Lesions affecting the parahippo- A context-based theory of recency and contiguity in free
campal cortex yield spatial memory deficits in humans. recall. Psychol Rev, 115(4), 893–912.
Cereb Cortex, 10(12), 1211–1216. Shalinsky, M. H., Magistretti, J., Ma, L., & Alonso, A. A.
Postle, B. R. (2006). Working memory as an emergent prop- (2002). Muscarinic activation of a cation current and associ-
erty of the mind and brain. Neuroscience, 139(1), 23–38. ated current noise in entorhinal-cortex layer-II neurons.
Ranganath, C. (2006). Working memory for visual objects: J Neurophysiol, 88(3), 1197–1211.
Complementary roles of inferior temporal, medial tempo- Shrager, Y., Levy, D. A., Hopkins, R. O., & Squire, L. R.
ral, and prefrontal cortex. Neuroscience, 139(1), 277–289. (2008). Working memory and the organization of brain
Ranganath, C., & Blumenfeld, R. S. (2005). Doubts about systems. J Neurosci, 28(18), 4818–4822.
double dissociations between short- and long-term memory. Stern, C. E., Sherman, S. J., Kirchhoff, B. A., & Hasselmo,
Trends Cogn Sci, 9(8), 374–380. M. E. (2001). Medial temporal and prefrontal contributions
Ranganath, C., & D’Esposito, M. (2001). Medial temporal to working memory tasks with novel and familiar stimuli.
lobe activity associated with active maintenance of novel Hippocampus, 11(4), 337–346.
information. Neuron, 31, 865–873. Stokes, M. G., Kusunoki, M., Sigala, N., Nili, H., Gaffan,
Ranganath, C., & D’Esposito, M. (2005). Directing the D., & Duncan, J. (2013). Dynamic coding for cognitive
mind’s eye: Prefrontal, inferior and medial temporal mech- control in prefrontal cortex. Neuron, 78(2), 364–375.
anisms for visual working memory. Curr Opin Neurobiol, Suzuki, W. A., Miller, E. K., & Desimone, R. (1997). Object
15(2), 175–182. and place memory in the macaque entorhinal cortex.
Ranganath, C., & Ritchey, M. (2012). Two cortical systems J Neurophysiol, 78(2), 1062–1081.
for memory-guided behaviour. Nat Rev Neurosci, 13(10), Tan, L., & Ward, G. (2000). A recency-based account of the
713–726. primacy effect in free recall. J Exp Psychol Learn Mem Cogn,
Riches, I. P., Wilson, F. A., & Brown, M. W. (1991). The 26(6), 1589–1625.
effects of visual stimulation and memory on neurons of the Tegner, J., Compte, A., & Wang, X. J. (2002). The dynamical
hippocampal formation and the neighboring parahippo- stability of reverberatory neural circuits. Biol Cybern, 87
campal gyrus and inferior temporal cortex of the primate. (5–6), 471–481.
J Neurosci, 11(6), 1763–1779. Todd, J. J., & Marois, R. (2004). Capacity limit of visual short-
Rissman, J., Gazzaley, A., & D’Esposito, M. (2008). Dynamic term memory in human posterior parietal cortex. Nature,
adjustments in prefrontal, hippocampal, and inferior tem- 428(6984), 751–754.
538 memory
45 The Medial Temporal Lobe
and Memory
LILA DAVACHI AND ALISON PRESTON
ABSTRACT Memory is our access to the past. Recent work in neural underpinnings of episodic memory, focusing on
cognitive neuroscience has provided evidence that our access processes taking place during the experience itself, or
to past events is guided by the reinstatement of the same encoding, and those involved in reactivating or retrieving
neural patterns of activity that were present during original
experiences. In essence, when we remember, the brain returns memories at a later time point. We present a model of
to a prior brain state, allowing us to reexperience our past. In memory formation and retrieval that has received the
this chapter, we highlight existing evidence for the role of the most support to date. This model has motivated most
medial temporal lobe in forming new episodic memories and empirical investigations into episodic memory process-
reinstating those memories during remembering. ing in the brain.
What did she say? Where did you go? What did you eat? Memory as reinstatement: The MAR model
Answering questions like these requires that you access
previous experiences, or episodes, of your life. Episodic The memory as reinstatement (MAR) model (Davachi
memory is memory for past experiences that occurred & Danker, 2013) represents a combination of current
at a particular time and place. By contrast, other forms theory and knowledge regarding how episodic memo-
of memory do not require that you access specifics ries are formed and subsequently accessed. It consists
about a prior encounter, such as knowing what a giraffe of elements drawn from many influential models of
is (semantic memory) or knowing how to ride a bike episodic memory and, thus, should not be considered
(procedural memory). Thus, episodic memory is a a new model, but rather a summary of the common
record of our personal experiences that make up the elements of many existing models (e.g., Alvarez &
narrative of our lives. Squire, 1994; McClelland, McNaughton, & O’Reilly,
The first link between the medial temporal lobe 1995; Moscovitch et al., 2005; Norman & O’Reilly,
(MTL) and episodic memory occurred through the 2003).
study of a patient, Henry Molaison, famously known as During an experience, the MAR model proposes that
H.M. At the age of 27, H.M.’s MTL was surgically the ongoing, dynamic representation of that experi-
removed to alleviate epileptic seizures; that surgery, ence is represented in distributed cortical and subcorti-
while successful in treating his epilepsy, also left him cal patterns of neural activation. These activation
with dense amnesia. He was no longer able to form new patterns are driven by sensory-perceptual (visual, audi-
episodic memories, a condition known as anterograde tory, somatosensory) experience, actions, internal
amnesia. The specificity of the deficit was remarkable. thoughts, and emotions, to name a few. Thus, neural
His intelligence was intact, and he appeared to have activation patterns at any one time point can be thought
normal working memory and procedural memory, but of as representing the current episode and state of the
he was confined to living “in the moment” because the organism. Critically, it is thought that the distributed
present disappeared into the past without a trace. With pattern of cortical and subcortical firing filters into the
H.M., it became clear that the MTL is critical for epi- MTL cortices and converges on the hippocampus,
sodic memory. This major discovery set the stage for where a “microrepresentation” of the current episode
neuroscientists to determine exactly how the MTL con- is created. The connections between hippocampal
tributes to episodic memory formation and retrieval. neurons that make up the memory are thought to
Critically, the MTL is not a single structure but rather undergo subsequent strengthening via the process
is made up of different regions. These include the hip- of long-term potentiation. Specifically, connections
pocampus as well as the entorhinal (ERc), perirhinal between concurrently active hippocampal neurons are
(PRc), and parahippocampal (PHc) cortices. In this more likely to become strengthened compared with
chapter, we review our current understanding of the those that are not (Hebb, 1949). What results from a
davachi and preston: the medial temporal lobe and memory 539
successfully encoded experience is a hippocampal episodic memory formation. These initial findings
neural pattern (HNP) and a corresponding cortical paved the way for future work further refining the use
neural pattern (CNP). Importantly, the HNP is thought of the DM paradigm to ask more specific questions
to contain the critical connections between representa- about how distinct MTL subregions contribute to
tions that allow the CNP to be accessed later and attrib- memory formation. This question is hotly debated
uted to a particular event. both in the animal (Eichenbaum, Sauvage, Fortin,
Importantly, episodic memory retrieval is thought to Komorowski, & Lipton, 2012) and human literatures
involve cue processing that, if successful, will lead to a (Davachi, 2006; Diana, Yonelinas, & Ranganath, 2007;
reinstatement of the HNP. Retrieval cues (e.g., an exter- Eichenbaum, Yonelinas, & Ranganath, 2007; Wixted &
nal stimulus or internal thought) are thought to serve Squire, 2011) and will be discussed in further detail in
as “keys” that unlock the HNP associated with a prior the next section.
experience, a process referred to as hippocampal pattern
completion. Pattern completion refers to the idea that a Hippocampal activity during encoding predicts
complete pattern (a memory) can be reconstructed later associative memory
from only a subset of the elements making up that
pattern. Thus, successful retrieval is thought to involve The hippocampus receives direct input from the MTL
reinstatement of all or some of the HNP established cortical regions (ERc, PRc, and PHc), each of which
during encoding. Finally, reinstatement of the HNP is receives a distinct pattern of inputs from other neocor-
then thought to be instrumental in reinstating the cor- tical and subcortical regions. Most researchers now
responding CNP, resulting in the concurrent reactiva- agree that the function of the underlying MTL cortex
tion of disparate cortical regions that were initially is distinct from the core function of the hippocampus
active during the experience. Importantly, it is thought proper. While the precise nature of this division remains
that this final stage of cortical reinstatement underlies unclear, several studies over the past 10 years have pro-
the subjective experience of recollection and drives vided a broadly consistent picture of the different func-
mnemonic decision making. tions of the hippocampus and MTL cortex in episodic
memory. These studies were motivated by an influential
Functional neuroimaging of episodic encoding computational model of MTL function that posits
that item and associative encoding are supported by
The processes through which individual experiences distinct, yet complementary, learning systems imple-
get transformed into long-lasting memory traces have mented within the hippocampus and PRc (Marr, 1971;
collectively been referred to as encoding mechanisms McClelland et al., 1995; Norman & O’Reilly, 2003;
(Davachi, 2006). The predominant neuroimaging O’Reilly & Rudy, 2000).
approach linking brain activation and episodic encod- These studies have employed the DM paradigm to
ing mechanisms has been the comparison of trial-by- differentiate patterns of brain activation during encod-
trial estimates of blood oxygen level–dependent, or ing that relate to later successful item-recognition
BOLD, activation during experiences that are later memory from those related to later recovery of associ-
remembered relative to activation during events that ated items, context, or source. Many of these studies
are not remembered. This approach has been referred have shown that the magnitude of hippocampal-
to as the difference in memory (DM) or subsequent encoding activation relates to participants’ later memory
memory paradigm (Paller & Wagner, 2002). Using this for the contextual details associated with individual
approach, measures of brain activity during encoding events (Awipi & Davachi, 2008; Davachi, Mitchell, &
can be related to a variety of memory outcomes, mea- Wagner, 2003; Hannula & Ranganath, 2008; Kirwan &
sured by different retrieval tests. For example, one can Stark, 2004; Ranganath et al., 2004; Staresina & Davachi,
determine whether each presented item was or was not 2008, 2009; Staresina, Duncan, & Davachi, 2011;
remembered as well as whether contextual details sur- Uncapher, Otten, & Rugg, 2006; Yu, Johnson, & Rugg,
rounding that were also recovered. 2012). Furthermore, in many such studies, PRc activa-
In two initial groundbreaking studies using the DM tion during encoding related to whether items were
approach, encoding activation in the parahippocampal later recognized, regardless of whether additional con-
gyrus was greater for words (Wagner et al., 1998) and textual details were also available at the time of retrieval
scenes (Brewer, Zhao, Desmond, Glover, & Gabrieli, (Davachi et al., 2003; Haskins, Yonelinas, Quamme, &
1998) that were successfully remembered relative to Ranganath, 2008; Kirwan & Stark, 2004; Ranganath
stimuli that were forgotten. These two studies thus et al., 2004; Staresina & Davachi, 2008, 2009). These
showed that MTL activation correlates with successful highly consistent results across different memory
540 memory
paradigms provide strong evidence for a clear division et al., 2011). Taken together, it is evident that involve-
of labor across MTL regions in their respective contri- ment of MTL cortex in encoding is largely dependent
butions to item and associative memory formation. on the content of the episode and on what aspects of the
Interestingly, these distinctions between encoding episode are attended. By contrast, hippocampal activa-
mechanisms in human PRc and hippocampus corre- tion is selectively related to whether associated details
spond with similar distinctions from single-cell record- are later recovered, irrespective of the content of those
ings in animals (Brown & Aggleton, 2001; Eichenbaum, details. These results also highlight that the role of PRc
Fortin, Sauvage, Robitsek, & Farovik, 2010; Komorowski, and PHc in item versus associative encoding will vary
Manns, & Eichenbaum, 2009; Sauvage, Fortin, Owens, depending on the nature of the stimuli being treated
Yonelinas, & Eichenbaum, 2008). Additionally, there is as the “item” and the “context” (Staresina et al., 2011).
notable evidence from human patient work that damage Thus, taken together, there is strong support for the
to the hippocampus disproportionately impairs recol- idea that hippocampal activation during an event is
lection, compared with item recognition based on correlated with the later recovery of the details associ-
familiarity (Giovanello, Verfaellie, & Keane, 2003; Vann ated with that event. More recent work has turned
et al., 2009; Yonelinas et al., 2002; but see Wixted & toward examining multivoxel activation patterns in
Squire, 2004). In contrast, one recent seminal report hippocampus and cortex to allow for greater precision
showed that selective damage to left PRc resulted in a in measuring the HNP and CNP associated with encod-
higher than average propensity to recollect events with ing to ask whether they are, indeed, related to later
little evidence of familiarity-based memory (Bowles et memory. Whereas standard functional MRI (fMRI)
al., 2007). This finding is critical because it is consistent analyses compare the mean response of a group of
with the growing body of literature linking the PRc with contiguous voxels within a region across experimental
item-encoding mechanisms that allow one to later know conditions, multivoxel pattern analysis uses a computer
that item has previously occurred, even in the absence algorithm, known as a classifier, to determine whether
of remembering the specific episodic context. or not the pattern of activity across a set of voxels differs
In addition to dissociations between item and associa- between two conditions (see Lewis-Peacock & Norman,
tive processes across MTL regions, there is growing chapter 77 in this volume). Using this approach, it has
appreciation that PRc and PHc show preferential been recently shown that, to the extent to which pre-
responses to different kinds of event content and, thus, frontal and temporal lobe cortical activation patterns
may contribute to episodic encoding in a domain-specific during encoding are similar to category level informa-
manner, whereas the hippocampus may be important tion, those items are better remembered on a subse-
in domain-general binding of the various event elements quent memory test (Kuhl, Rissman, & Wagner, 2012).
(Davachi, 2006). A variety of evidence supports this A related experiment further showed that the similar-
framework. First, it has been demonstrated that the ity of an item’s pattern of cortical activation across
human PRc responds more to objects and faces than repeated study trials with the item predicted better
scenes, and that PHc shows the opposite response memory (Xue et al., 2010).
pattern: greater activation to scenes than objects and Much less is known about the relationship between
faces (Litman, Awipi, & Davachi, 2009; Liang, Wagner, MTL-encoding patterns and memory formation.
& Preston, 2013). Second, when study items were scenes However, a recent study showed that cortical similarity
and the associated “context” was devised to be one of (in PRc and PHc) and hippocampal similarity had dis-
six repeating objects, PRc encoding activation now pre- tinct relationships to later memory. Specifically, similar-
dicted later recollection, whereas successful scene ity between the PRc response to a particular stimulus
memory was supported by PHc (Awipi & Davachi, and other stimuli from the same perceptual category
2008). Third, both hippocampal and PRc encoding was related to enhanced memory; the reverse relation-
activation are related to whether object details will be ship was observed in the hippocampus (LaRocque
later recalled, whereas only hippocampal activation et al., 2013). Finally, recent work also suggests that simi-
additionally predicts the recovery of other contextual larity in the representation of events may be diagnostic
details (Staresina & Davachi, 2008). Finally, in a tightly for successful encoding even when the stimuli pre-
controlled study where study items were always words sented are different. Specifically, hippocampal similar-
but participants’ task was to use each cue word to either ity between items in a sequence during encoding is
imagine an object or a scene, it was shown that PRc higher for pairs of items later rated as having occurred
activation predicted later source memory for the object- close together in time (Ezzyat & Davachi, 2014). This
imagery trials, and that PHc activation predicted later finding suggests that similarity in hippocampal response
source memory for the scene-imagery trials (Staresina patterns may also be related to binding items with their
davachi and preston: the medial temporal lobe and memory 541
temporal context (see Manning, Norman, & Kahana, event requires reactivating the specific hippocampal
chapter 47 in this volume). representation associated with that event.
542 memory
memory-related reinstatement. Nonetheless, they found for unique word-scene associations during encoding
a significant correlation between encoding-retrieval were reinstated during retrieval (Staresina, Henson,
pattern similarity in cortex and hippocampal activation Kriegeskorte, & Alink, 2012). Participants were pre-
during retrieval, with hippocampal engagement medi- sented with word cues during retrieval and asked to
ating the link between cortical encoding-retrieval simi- recall the corresponding scene; reinstatement of MTL
larity and memory success. These results are consistent cortical-encoding patterns was observed only when sub-
with the idea that the hippocampus facilitates reinstate- jects successfully recalled the scene, providing the first
ment of cortical memory traces during successful evidence for MTL reinstatement in the human brain.
remembering. Notably, while this study did not measure hippocampal
However, these findings do not speak to a core aspect reinstatement, the degree of MTL cortical reinstate-
of the reinstatement hypothesis—that hippocampal ment was correlated with hippocampal retrieval activa-
and MTL cortical-encoding patterns themselves are tion, consistent with a role for the hippocampus in
reinstated during memory retrieval. Leading memory coordinating CNP reinstatement.
models posit that retrieval should not simply activate
the MTL, but would specifically activate the same MTL Memory reinstatement during new encoding
neurons that were active during the original encoding
experience. A key demonstration of this principle As highlighted in the previous section, memories for
comes from a landmark study using intracranial single- past events are often reinstated during new experi-
cell recordings in epileptic patients (Gelbard-Sagiv, ences. These findings emphasize that memory encod-
Mukamel, Harel, Malach, & Fried, 2008). Hippocampal ing and retrieval are not performed in isolation, but
and MTL cortical neurons were recorded while patients rather are interactive processes. Recent human neuro-
viewed and later recalled a series of short video clips imaging research indicates that new learning influ-
(e.g., a scene from The Simpsons). They found that a ences, and is influenced by, reactivation of existing
subset of MTL neurons showed selective firing during memories. In one such study, participants learned
initial viewing of the video clips (e.g., one hippocampal overlapping (e.g., watch-sink and later watch-pipe) and
neuron fired specifically during The Simpsons clip). Crit- nonoverlapping (e.g., peanut—moose) pairs of pic-
ically, they found that these selective MTL neurons were tures during fMRI scanning (Kuhl, Shah, DuBrow, &
also active during verbal recall of the same videos (i.e., Wagner, 2010). The findings revealed that the degree
the “Simpsons” neuron fired again when the participant to which prior memories (e.g., memory for the watch-
recalled The Simpsons video clip). Consistent with this sink event) were reinstated during encoding of new
electrophysiological work in humans, physiological overlapping experiences (watch-pipe) was associated
studies in rodents have also demonstrated reactivation with greater retention of the originally learned informa-
of hippocampal memory traces during retrieval (for a tion when compared to memory for nonoverlapping
review, see Carr, Jadhav, & Frank, 2011). For instance, information. Moreover, hippocampal engagement during
hippocampal place cells representing a spatial trajec- encoding of the overlapping pairs was related to both
tory through a well-learned environment are replayed cortical memory reinstatement and improved memory
in sequence at remote time points as the animals expe- retention. These findings indicate that memory rein-
rience new environments (Karlsson & Frank, 2009)—a statement during new encoding helps reduce forgetting
phenomenon referred to as hippocampal replay. More- of past events. One possibility is that reduced forgetting
over, recent work has shown that interrupting hippo- was not just the result of strengthening of reactivated
campal replay impairs rodents’ ability to correctly memories, but may also have resulted from a hippocam-
navigate to rewards in a well-learned environment pal-mediated integrative encoding mechanism, whereby
(Jadhav, Kemere, German, & Frank, 2012), providing newly encountered information is integrated with exist-
further evidence for the link between hippocampal ing memories at the time of learning (Shohamy &
replay and memory-guided decision making. Collec- Wagner, 2008; for a review, see Zeithamova, Schlichting,
tively, these findings provide strong evidence that the & Preston, 2012).
reinstatement of hippocampal memory traces plays an According to this mechanism, the fundamental role
important role in guiding behavior and choice. of the hippocampus in memory is not only to form
Until the recent advent of multivariate fMRI analysis relationships among elements within an individual
methods, documenting reactivation of MTL-encoding experience, but also to construct memory representa-
patterns during retrieval in the human brain had proved tions that link memory elements across discrete experi-
challenging. One recent study achieved this goal by ences. This constructive hypothesis about hippocampal
showing that activation patterns in human MTL cortex function dates back to Tolman’s concept of a “cognitive
davachi and preston: the medial temporal lobe and memory 543
map” (Tolman, 1948) and is further exemplified in event encoding (Wimmer & Shohamy, 2012). Thus,
more modern models of hippocampal function (Cohen converging evidence from these studies provides evi-
& Eichenbaum, 1993; Kumaran & McClelland, 2012; dence for a hippocampal-mediated encoding mecha-
O’Keefe & Nadel, 1978). Integrative encoding proposes nism whereby overlapping experiences are integrated
that when a new event shares a common feature, or into a network of related memories as they are learned.
features, with an existing memory trace, the common Together, these findings provide a deeper understand-
features elicit memory reinstatement through hippo- ing of how remembering the past influences how we
campal pattern completion. New experiences would learn about the present.
then not only be encoded in the context of presently
available information in the external world, but would Conclusion
also be bound to reactivated representations of prior
related memories. Furthermore, this mechanism makes In summary, there is strong evidence for hippocampal
the fundamental prediction that by combining informa- and cortical contributions to encoding and retrieval
tion across discrete events, hippocampal memory rep- and growing evidence for the MAR model supporting
resentations would include information that goes the widely held view that memory is supported by the
beyond direct experience. Accordingly, integrative reinstatement of brain patterns that characterize a prior
encoding would not only support strengthening of encoding experience. Furthermore, recent work using
existing memories (e.g., Kuhl et al., 2010), but would multivariate fMRI approaches suggests that reinstate-
also support novel inferences about the relationships ment not only strengthens existing memories but also
between memory elements that were experienced at contributes to establishing links across related experi-
different times and generalization of knowledge to ences. There is still much to learn, however, as under-
entirely new situations, a hallmark of episodic memory. standing what aspects of a prior encounter are reinstated
In recent years, several human neuroimaging studies is still unknown. Furthermore, very little is known about
have set out to test the hypothesis that the hippocampus the temporal relationship between hippocampal and
supports integrative encoding through memory rein- cortical reinstatement.
statement. These studies typically employ similar para-
digms wherein participants learn a set of overlapping
associations (e.g., A goes with B, B goes with C) during REFERENCES
fMRI scanning. The existence of an integrated memory
Alvarez, P., & Squire, L. R. (1994). Memory consolidation
organization is demonstrated by the ability of partici- and the medial temporal lobe: A simple network model.
pants to express knowledge about the inferential rela- Proc Natl Acad Sci USA, 91(15), 7041–7045.
tionships among elements of overlapping pairs that Awipi, T., & Davachi, L. (2008). Content-specific source
were not explicitly studied together (i.e., A is related to encoding in the human medial temporal lobe. J Exp Psychol
C). Using this paradigm, two neuroimaging studies Learn Mem Cogn, 34(4), 769–779.
Bowles, B., Crupi, C., Mirsattari, S. M., Pigott, S. E.,
have shown that hippocampal activation during initial Parrent, A. G., Pruessner, J. C., … Köhler, S. (2007).
encoding of overlapping pairs is uniquely related to Impaired familiarity with preserved recollection after ante-
participants’ ability to make subsequent inference rior temporal-lobe resection that spares the hippocampus.
judgments (Shohamy & Wagner, 2008; Zeithamova & Proc Natl Acad Sci USA, 104(41), 16382–16387.
Preston, 2010). Brewer, J. B., Zhao, Z., Desmond, J. E., Glover, G. H., &
Gabrieli, J. D. (1998). Making memories: Brain activity
A more recent study using MVPA further demon-
that predicts how well visual experience will be remem-
strated that participants reinstate memories for prior bered. Science, 281(5380), 1185–1187.
associations in cortex (e.g., the A stimulus from an AB Brown, M. W., & Aggleton, J. P. (2001). Recognition
pairing) while encoding new related experiences (BC memory: What are the roles of the perirhinal cortex and
pairs; Zeithamova, Dominick, & Preston, 2012). More- hippocampus? Nat Rev Neurosci, 2(1), 51–61.
Cansino, S., Maquet, P., Dolan, R. J., & Rugg, M. D. (2002).
over, the degree to which participants reinstated prior
Brain activity underlying encoding and retrieval of source
events during encoding of new associations was related memory. Cereb Cortex, 12(10), 1048–1056.
to their ability to infer relationships between distinct Carr, M. F., Jadhav, S. P., & Frank, L. M. (2011). Hippocam-
events that shared content, with hippocampal encod- pal replay in the awake state: A potential substrate for
ing activation further tracking the successful use of memory consolidation and retrieval. Nat Neurosci, 14(2),
integrated memories. Other work has shown that 147–153.
Chadwick, M. J., Hassabis, D., & Maguire, E. A. (2011).
memory reinstatement and hippocampal-mediated Decoding overlapping memories in the medial temporal
integration takes place even when participants do not lobes using high-resolution fMRI. Learn Mem, 18(12),
explicitly recall overlapping memories during new 742–746.
544 memory
Chadwick, M. J., Hassabis, D., Weiskopf, N., & Maguire, E. Hannula, D. E., & Ranganath, C. (2008). Medial temporal
A. (2010). Decoding individual episodic memory traces in lobe activity predicts successful relational memory binding.
the human hippocampus. Curr Biol, 20(6), 544–547. J Neurosci, 28(1), 116–124.
Chen, J., Olsen, R. K., Preston, A. R., Glover, G. H., & Haskins, A. L., Yonelinas, A. P., Quamme, J. R., &
Wagner, A. D. (2011). Associative retrieval processes in the Ranganath, C. (2008). Perirhinal cortex supports encod-
human medial temporal lobe: Hippocampal retrieval ing and familiarity-based recognition of novel associations.
success and CA1 mismatch detection. Learn Mem, 18(8), Neuron, 59(4), 554–560.
523–528. Hebb, D. O. (1949). Temperament in chimpanzees; method
Cohen, N. J., & Eichenbaum, H. E. (1993). Memory, amnesia, of analysis. J Comp Physiol Psychol, 42(3), 192–206.
and the hippocampal system. Cambridge, MA: MIT Press. Jadhav, S. P., Kemere, C., German, P. W., & Frank, L. M.
Davachi, L. (2006). Item, context and relational episodic (2012). Awake hippocampal sharp-wave ripples support
encoding in humans. Curr Opin Neurobiol, 16(6), 693–700. spatial memory. Science, 336(6087), 1454–1458.
Davachi, L., & Danker, J. F. (2013). The cognitive neurosci- Karlsson, M. P., & Frank, L. M. (2009). Awake replay of
ence of episodic memory. In K. N. Ochsner & S. M. Kosslyn remote experiences in the hippocampus. Nat Neurosci,
(Eds.), The Oxford handbook of cognitive neuroscience (Vol. I, 12(7), 913–918.
pp. 375–388). New York, NY: Oxford University Pres. Kirwan, C. B., & Stark, C. E. (2004). Medial temporal lobe
Davachi, L., Mitchell, J. P., & Wagner, A. D. (2003). Mul- activation during encoding and retrieval of novel face-name
tiple routes to memory: Distinct medial temporal lobe pro- pairs. Hippocampus, 14(7), 919–930.
cesses build item and source memories. Proc Natl Acad Sci Komorowski, R. W., Manns, J. R., & Eichenbaum, H. (2009).
USA, 100(4), 2157–2162. Robust conjunctive item-place coding by hippocampal
Diana, R. A., Yonelinas, A. P., & Ranganath, C. (2007). neurons parallels learning what happens where. J Neurosci,
Imaging recollection and familiarity in the medial temporal 29(31), 9918–9929.
lobe: A three-component model. Trends Cogn Sci, 11(9), Kriegeskorte, N., Mur, M., & Bandettini, P. (2008).
379–386. Representational similarity analysis: Connecting the
Dobbins, I. G., Rice, H. J., Wagner, A. D., & Schacter, D. branches of systems neuroscience. Front Syst Neurosci, 2(4),
L. (2003). Memory orientation and success: Separable neu- 1–28.
rocognitive components underlying episodic recognition. Kuhl, B. A., Rissman, J., & Wagner, A. D. (2012). Multi-voxel
Neuropsychologia, 41(3), 318–333. patterns of visual category representation during episodic
Eichenbaum, H., Fortin, N., Sauvage, M., Robitsek, R. J., encoding are predictive of subsequent memory. Neuropsy-
& Farovik, A. (2010). An animal model of amnesia that chologia, 50(4), 458–469.
uses Receiver Operating Characteristics (ROC) analysis to Kuhl, B. A., Shah, A. T., DuBrow, S., & Wagner, A. D.
distinguish recollection from familiarity deficits in recogni- (2010). Resistance to forgetting associated with hippocam-
tion memory. Neuropsychologia, 48(8), 2281–2289. pus-mediated reactivation during new learning. Nat Neuro-
Eichenbaum, H., Sauvage, M., Fortin, N., Komorowski, R., sci, 13(4), 501–506.
& Lipton, P. (2012). Towards a functional organization of Kumaran, D., & McClelland, J. L. (2012). Generalization
episodic memory in the medial temporal lobe. Neurosci through the recurrent interaction of episodic memories: A
Biobehav Rev, 36(7), 1597–1608. model of the hippocampal system. Psychol Rev, 119(3),
Eichenbaum, H., Yonelinas, A. P., & Ranganath, C. (2007). 573–616.
The medial temporal lobe and recognition memory. Annu LaRocque, K. F., Smith, M. E., Carr, V. A., Witthoft, N.,
Rev Neurosci, 30, 123–152. Grill-Spector, K., & Wagner, A. D. (2013). Global similar-
Eldridge, L. L., Knowlton, B. J., Furmanski, C. S., ity and pattern separation in the human medial temporal
Bookheimer, S. Y., & Engel, S. A. (2000). Remembering lobe predict subsequent memory. J Neurosci, 33(13),
episodes: A selective role for the hippocampus during 5466–5474.
retrieval. Nat Neurosci, 3, 1149–1152. Liang, J. C., Wagner, A. D., & Preston, A. R. (2013). Content
Ezzyat, Y., & Davachi, L. (2014). Similarity breeds proximity: representation in the human medial temporal lobe. Cereb
Pattern similarity within and across contexts is related to Cortex, 23(1), 80–96.
later mnemonic judgments of temporal proximity. Neuron, Litman, L., Awipi, T., & Davachi, L. (2009). Category-
81(5), 1179–1189. specificity in the human medial temporal lobe cortex. Hip-
Fortin, N. J., Wright, S. P., & Eichenbaum, H. (2004). pocampus, 19(3), 308–319.
Recollection-like memory retrieval in rats is dependent on Marr, D. (1971). Simple memory: A theory for archicortex.
the hippocampus. Nature, 431(7005), 188–191. Philos Trans R Soc Lond B Biol Sci, 262(841), 23–81.
Gelbard-Sagiv, H., Mukamel, R., Harel, M., Malach, R., & McClelland, J. L., McNaughton, B. L., & O’Reilly, R. C.
Fried, I. (2008). Internally generated reactivation of single (1995). Why there are complementary learning systems in
neurons in human hippocampus during free recall. Science, the hippocampus and neocortex: Insights from the suc-
322(5898), 96–101. cesses and failures of connectionist models of learning and
Giovanello, K. S., Schnyer, D. M., & Verfaellie, M. (2004). memory. Psychol Rev, 102, 419–457.
A critical role for the anterior hippocampus in relational Moscovitch, M., Rosenbaum, R. S., Gilboa, A., Addis, D.
memory: Evidence from an fMRI study comparing associa- R., Westmacott, R., Grady, C., & Nadel, L. (2005). Func-
tive and item recognition. Hippocampus, 14(1), 5–8. tional neuroanatomy of remote episodic, semantic and
Giovanello, K. S., Verfaellie, M., & Keane, M. M. (2003). spatial memory: A unified account based on multiple trace
Disproportionate deficit in associative recognition relative theory. J Anat, 207(1), 35–66.
to item recognition in global amnesia. Cogn Affect Behav Norman, K. A., & O’Reilly, R. C. (2003). Modeling hippo-
Neurosci, 3(3), 186–194. campal and neocortical contributions to recognition
davachi and preston: the medial temporal lobe and memory 545
memory: A complementary-learning-systems approach. Impaired recollection but spared familiarity in patients
Psychol Rev, 110(4), 611–646. with extended hippocampal system damage revealed by
O’Keefe, J., & Nadel, L. (1978). The hippocampus as a cognitive 3 convergent methods. Proc Natl Acad Sci USA, 106(13),
map. London: Clarendon. 5442–5447.
O’Reilly, R. C., & Rudy, J. W. (2000). Computational prin- Wagner, A. D., Schacter, D. L., Rotte, M., Koutstaal, W.,
ciples of learning in the neocortex and hippocampus. Hip- Maril, A., Dale, A. M., … Buckner, R. L. (1998). Building
pocampus, 10(4), 389–397. memories: Remembering and forgetting of verbal experi-
Paller, K. A., & Wagner, A. D. (2002). Observing the trans- ences as predicted by brain activity. Science, 281(5380),
formation of experience into memory. Trends Cogn Sci, 6(2), 1188–1191.
93–102. Wheeler, M. E., & Buckner, R. L. (2004). Functional-
Polyn, S. M., Natu, V. S., Cohen, J. D., & Norman, K. A. anatomic correlates of remembering and knowing. Neuro-
(2005). Category-specific cortical activity precedes retrieval Image, 21(4), 1337–1349.
during memory search. Science, 310(5756), 1963–1966. Wheeler, M. E., Petersen, S. E., & Buckner, R. L. (2000).
Ranganath, C., Yonelinas, A. P., Cohen, M. X., Dy, C. J., Memory’s echo: Vivid remembering reactivates sensory-
Tom, S. M., & D’Esposito, M. (2004). Dissociable correlates specific cortex. Proc Natl Acad Sci USA, 97(20),
of recollection and familiarity within the medial temporal 11125–11129.
lobes. Neuropsychologia, 42(1), 2–13. Wimmer, G. E., & Shohamy, D. (2012). Preference by associa-
Ritchey, M., Wing, E. A., Labar, K. S., & Cabeza, R. (2012). tion: How memory mechanisms in the hippocampus bias
Neural similarity between encoding and retrieval is related decisions. Science, 338(6104), 270–273.
to memory via hippocampal interactions. Cereb Cortex, Wixted, J. T., & Squire, L. R. (2004). Recall and recognition
23(12), 2818–2828. are equally impaired in patients with selective hippocampal
Sauvage, M. M., Fortin, N. J., Owens, C. B., Yonelinas, A. damage. Cogn Affect Behav Neurosci, 4(1), 58–66.
P., & Eichenbaum, H. (2008). Recognition memory: Oppo- Wixted, J. T., & Squire, L. R. (2011). The medial temporal
site effects of hippocampal damage on recollection and lobe and the attributes of memory. Trends Cogn Sci, 15(5),
familiarity. Nat Neurosci, 11(1), 16–18. 210–217.
Shohamy, D., & Wagner, A. D. (2008). Integrating memories Xue, G., Dong, Q., Chen, C., Lu, Z., Mumford, J. A., &
in the human brain: Hippocampal-midbrain encoding of Poldrack, R. A. (2010). Greater neural pattern similarity
overlapping events. Neuron, 60(2), 378–389. across repetitions is associated with better memory. Science,
Staresina, B. P., & Davachi, L. (2008). Selective and shared 330(6000), 97–101.
contributions of the hippocampus and perirhinal cortex to Yonelinas, A. P., Kroll, N. E., Quamme, J. R., Lazzara, M.
episodic item and associative encoding. J Cogn Neurosci, M., Sauve, M. J., Widaman, K. F., & Knight, R. T. (2002).
20(8), 1478–1489. Effects of extensive temporal lobe damage or mild hypoxia
Staresina, B. P., & Davachi, L. (2009). Mind the gap: Binding on recollection and familiarity. Nat Neurosci, 5(11),
experiences across space and time in the human hippocam- 1236–1241.
pus. Neuron, 63(2), 267–276. Yu, S. S., Johnson, J. D., & Rugg, M. D. (2012). Hippocampal
Staresina, B. P., Duncan, K. D., & Davachi, L. (2011). Peri- activity during recognition memory co-varies with the accu-
rhinal and parahippocampal cortices differentially contrib- racy and confidence of source memory judgments. Hippo-
ute to later recollection of object- and scene-related event campus, 22(6), 1429–1437.
details. J Neurosci, 31(24), 8739–8747. Zeithamova, D., Dominick, A. L., & Preston, A. R. (2012).
Staresina, B. P., Henson, R. N., Kriegeskorte, N., & Alink, Hippocampal and ventral medial prefrontal activation
A. (2012). Episodic reinstatement in the medial temporal during retrieval-mediated learning supports novel infer-
lobe. J Neurosci, 32(50), 18150–18156. ence. Neuron, 75(1), 168–179.
Tolman, E. C. (1948). Cognitive maps in rats and men. Psychol Zeithamova, D., & Preston, A. R. (2010). Flexible memo-
Rev, 55(4), 189–208. ries: Differential roles for medial temporal lobe and pre-
Uncapher, M. R., Otten, L. J., & Rugg, M. D. (2006). Epi- frontal cortex in cross-episode binding. J Neurosci, 30(44),
sodic encoding is more than the sum of its parts: An fMRI 14676–14684.
investigation of multifeatural contextual encoding. Neuron, Zeithamova, D., Schlichting, M. L., & Preston, A. R.
52(3), 547–556. (2012). The hippocampus and inferential reasoning: Build-
Vann, S. D., Tsivilis, D., Denby, C. E., Quamme, J. R., ing memories to navigate future decisions. Front Hum Neu-
Yonelinas, A. P., Aggleton, J. P., … Mayes, A. R. (2009). rosci, 6, 70.
546 memory
46 The Contribution of Hippocampal
Subregions to Memory Coding
JILL K. LEUTGEB AND STEFAN LEUTGEB
ABSTRACT Episodic memories contain information about what the postrhinal cortex in rodents are corresponding
happened, where, when, and in which sequence. The forma- (figure 46.1A, B). Throughout this chapter we will use
tion of episodic memories requires the medial temporal lobe, the rodent nomenclature, focus on rodent circuits, and
which consists of subregions with unique connection patterns
and specialized cell types. Each subregion can perform spe- mention notable differences to primates.
cialized computations to support memory processing. Particu- The regions of the medial temporal lobe that com-
lar computations may selectively support the acquisition, prise the core circuit for memory are arranged in three
storage, or retrieval phase, or may be flexibly used across the tiers that are reciprocally connected (figure 46.1C).
different phases of memory coding. This chapter describes The first tier integrates information from various corti-
processing streams in the medial temporal lobe, specialized
circuits in each subregion of the hippocampal formation, and
cal association areas and consists of the perirhinal and
how they act in concert to support the various aspects of epi- postrhinal cortex. The second tier consists of the ento-
sodic memory. rhinal cortex, which can be subdivided into lateral and
medial divisions (anterior and posterior divisions in
nonhuman primates). The lateral and medial divisions
Hierarchical processing across subregions are preferentially connected with the perirhinal and
in the medial temporal lobe postrhinal cortex, respectively. The third tier includes
the dentate gyrus, the hippocampal CA regions, and
The medial temporal lobe is essential for memories of the subiculum. This tier is commonly referred to as the
facts and events. Patient H.M. and similar patients with hippocampal formation. In addition to the reciprocal
extensive lesions in the medial temporal lobe have connections between the first and the second and
revealed that the loss of these brain regions results in a between the second and the third tier, subregions
profound impairment in forming new semantic and within a tier have uniquely arranged local circuits and
episodic memories and in remembering events that consist of distinct anatomical and functional cell types
occurred over the previous months and years before the (van Strien, Cappaert, & Witter, 2009). Each subregion
brain damage (Scoville & Milner, 1957; Squire & is thus thought to act as a specialized processing module
Alvarez, 1995). However, the extent of brain damage in that contributes in a unique way to semantic and, in
patients may go beyond the areas that are essential for particular, episodic memories.
memory or only include a subset of the areas that can Episodic memories are composed of three funda-
support memory. The subset of structures in the medial mental elements, namely, “where,” “when,” and “what”
temporal lobe required for memory has thus been (Tulving, 1983). The “where” and “what” elements
defined with better precision in animal models of are initially computed in separate processing streams
amnesia in which the damage can be targeted to selected from sensory cortices to association cortices and are
subregions (Zola-Morgan & Squire, 1993). The core eventually bound together into a unified representa-
circuit for event memories includes, in primates, the tion in the hippocampus (Eichenbaum, Sauvage, Fortin,
perirhinal cortex, the parahippocampal cortex, the Komorowski, & Lipton, 2011). The “when” element
entorhinal cortex, the dentate gyrus, the CA regions of may be directly generated by intrahippocampal compu-
the hippocampus proper (cornu ammonis, or Ammon’s tations. This chapter describes how the separate pro-
horn), and the subiculum. These brain regions are phy- cessing streams converge and how the information is
logenetically conserved divisions of the cortex and can further processed in the hippocampus. These computa-
thus be readily identified across all mammalian lin- tions are known to give rise to place cells in the hip-
eages. A consistent nomenclature is generally used in pocampus of rodents (O’Keefe & Nadel, 1978) and to
different mammalian species, with the notable excep- related cell types in the hippocampus of primates
tion that the parahippocampal cortex in primates and (Ekstrom et al., 2003; Rolls, 1999). Place cells are
Anterior Posterior
Figure 46.1 Anatomy and connectivity of the rodent medial entorhinal cortex (MEC and LEC), perirhinal cortex (PER),
temporal lobe. (A) Schematic of a rat brain, lateral view of and postrhinal cortex (POR). Asterisks in black and white
the left hemisphere. Areas important for memory and spatial indicate corresponding positions across the two images. Scale
navigation are highlighted and labeled. The dashed box indi- bar, 100 μm. (Adapted from Witter et al., 2006, with permis-
cates the position of a horizontal section taken through the sion from Elsevier.) (C) Circuit diagram of brain regions for
hippocampal formation (HF) and adjacent cortical structures memory processing. Different box shades depict three tiers
as shown in (B). (B) Stain of neurons in the HF: dentate gyrus in the cortical hierarchy. (Adapted by permission from Mac-
(DG), CA3, CA2, CA1, and subiculum (S); and in the sur- millan Publishers Ltd: Nature Reviews Neuroscience, from van
rounding input and output structures: medial and lateral Strien et al., 2009.)
principal neurons that reliably fire action potentials in network computations, such as making input patterns
a particular spatial location (figure 46.2A). However, more distinct (i.e., pattern separation) or processing
consistent with the convergence of multiple pathways sequence and temporal information. The outcomes of
in the hippocampus, place cells can also integrate other these computations can then be relayed back to cortical
types of information or even switch between repre- areas to generate enriched representations that include
senting nonspatial and spatial information (Wood, object, spatial, and temporal information as well as
Dudchenko, & Eichenbaum, 1999). For example, mod- information about the sequences of current and past
ulation of firing rates at the preferred firing location of events.
a place cell can code for differences in context or in
internal state (Kennedy & Shapiro, 2009; S. Leutgeb Separate processing streams throughout
et al., 2005b). Importantly, the hippocampal code is cortical modules
distributed and combinatorial, such that some cells may
be more exclusively spatially tuned while others may be The notion that different types of information are ini-
particularly responsive to differences in visual stimuli, tially processed separately is based on anatomical and
odors, elapsed time, or task demands. physiological evidence. In primates, cortical processing
Hippocampal cell populations do not only receive is characterized by parallel pathways (e.g., dorsal and
information to conjointly represent different types of ventral visual stream, and polymodal and unimodal
information, but can also perform a number of unique inputs to parahippocampal and perirhinal cortices;
548 memory
Suzuki & Amaral, 1994; Ungerleider & Mishkin, 1982). A
In rodents, posterior association areas process visuospa-
tial information and project predominantly to postrhi-
nal cortex, whereas anterior association areas process
unimodal sensory information and project predomi-
nantly to perirhinal cortex (van Strien, Cappaert, &
Witter, 2009). The segregated anatomical pathways
result in neural representations for “what” information
primarily in the perirhinal cortex and for “where” infor-
mation primarily in the postrhinal cortex. This segrega-
tion is largely retained in the next processing stage. The
medial entorhinal cortex (MEC) receives most of its
input from the postrhinal cortex. The lateral entorhinal
cortex (LEC) receives most of its input from the peri-
rhinal cortex. Accordingly, MEC or postrhinal lesions
selectively affect navigation and pathfinding, whereas
LEC or perirhinal lesions primarily affect nonspatial B
information processing and, more generally, object-
context associations (Eichenbaum et al., 2011; Suzuki
& Amaral, 2004). The idea for a segregation of function
at the level of the entorhinal cortex is also supported
by the selective presence of specialized cell types in
each subregion, as described in the following sections.
C
Medial Entorhinal Cortex In rodents, MEC can
be distinguished from LEC by differences in cytoarchi-
tecture and in the input and output pathways. In addi-
tion, the principal neurons of these two entorhinal
subregions have distinct firing patterns (Witter & Moser,
2006). The most remarkable cell type in MEC is the grid
cell. Grid cells are principal neurons that fire action
potentials at multiple spatial locations within an envi-
ronment. The spatial receptive fields of these neurons
are arranged in a highly regular triangular grid (figure
46.2B). Grid cells were first described when performing
extracellular recordings from the MEC in awake, behav-
ing rats that explored two-dimensional environments
(Hafting, Fyhn, Molden, Moser, & Moser, 2005). Cor-
responding cells are also found in the primate entorhi-
nal cortex (Killian, Jutras, & Buffalo, 2012). In primates,
the grid pattern emerges while a spatial scene is Figure 46.2 Place cells and grid cells. (A) Spatially tuned
principal neurons are found in all subregions of the rodent
explored by eye movements. During eye movements, hippocampal formation. A gray line indicates the path of the
the firing peaks for where on the screen the eyes fixate animal while exploring a 1 m × 1 m open field. Each action
are arranged in a triangular grid pattern (figure 46.2C). potential of the CA1 neuron is shown as a red dot superim-
Although grid cells are a prominent cell type in the posed on the position of the animal when the spike occurred
most dorsal pole of MEC in rodents and a common cell (left). The firing rate of the same CA1 neuron is represented
by a heat map with high firing rates in warmer colors (right).
type in the posterior entorhinal cortex of primates,
Hippocampal neurons are place cells with a single place field,
there are also numerous other functional cell types in whereas (B) grid cells are found in all layers of medial ento-
entorhinal cortex. Some of these cell types have obvious rhinal cortex (MEC). The multiple firing peaks of grid cells
spatial firing patterns, such as firing along the borders are arranged as equilateral triangles (white dashed lines) that
of a box in rodents or along the border of a screen in form a grid-like representation (right). (C) Grid cells in the
primate MEC during the visual exploration of spatial scenes.
primates (Killian et al., 2012; Solstad, Boccara, Kropff,
(Reprinted by permission from Macmillan Publishers Ltd:
Moser, & Moser, 2008). However, there are also cells Nature, Killian et al., 2012.) (See color plate 38.)
The entorhinal cortex is the gateway to the dentate Processing within dentate gyrus Predictions about a sepa-
gyrus and to the hippocampal CA fields. All major excit- rate role of DG in memory processing are based on
atory inputs to hippocampus originate from neurons elements of its architecture that are unique for the DG
in the entorhinal cortex, and the entorhinal cortex subregion as well as on the sparse firing of dentate
receives back-projections from CA1 and the subiculum. neurons. The firing in DG is sparse in that the mean
Because the entorhino-hippocampal loops are at the firing rates of dentate granule neurons are low and in
core of memory processing and because disorders with that the proportion of active neurons at any given time
memory impairment each have different patterns of is also low (Piatti, Ewell, & Leutgeb, 2013). The sparse
cell loss across entorhinal and hippocampal subregions, activity in DG is the result of a rich inhibitory network.
it can be expected that we will gain a profound under- Such strong inhibition shuts off a large number of
standing of memory by unraveling the information flow neurons and makes the DG a competitive network in
within the circuit. Although questions about memory which only a few neurons fire action potentials (Rolls,
processing will eventually need to be addressed in 2010). Even though dentate activity is sparse, it is
complex memory tasks across different species, substan- coupled to CA3 through the powerful unidirectional
tial progress has been made from examining how spatial mossy fiber projections. Computational models thus
firing patterns emerge and are processed throughout predict that the few active neurons can “detonate” the
the entorhino-hippocampal circuit in rodents. This CA3 network, thereby generating new patterns of
section therefore focuses on our current understanding neural activity and strengthening the connections
of the processing of spatial information within the between activated cells such that new memories are
circuit along with key insights from studies on memory encoded (O’Reilly & McClelland, 1994; Rolls, 2010).
processing. During the storage of new memories, it is not only
important that new activity patterns are imposed onto
Early Processing Stages of the Hippocampal the CA3 network, but also that the new patterns are
Formation distinct from patterns that were stored on previous
occasions. A critical step in the encoding of a new epi-
Connections of the dentate gyrus and the CA3 subregion The sodic memory is therefore the amplification of the dif-
dentate gyrus (DG) is classically considered the first ferences between the representations that are encoded
processing stage of the hippocampal formation. The compared to those that already exist in the network.
inputs of entorhinal cortex layer II neurons to the The powerful effect of the dentate in activating CA3 is
550 memory
therefore paired with a computation that is referred receive is limited by the size of the dendritic tree, a
to as “pattern separation.” Partially overlapping experi- high connection probability between neurons within a
ences are made more distinct by amplifying sensory network can only be achieved if the number of neurons
differences before a representation is encoded (figure in the network is not too high. In the rat, the total
46.3A). In behavioral studies in rodents in which number of CA3 cells is about 300,000 per hemisphere
neurons of the DG were selectively lesioned or manipu- (Rolls & Treves, 1998). In comparisons between pri-
lated, it was found that rats were unable to discriminate mates and rodents, it is interesting to note that the
between adjacent spatial locations while they could increased size of the primate hippocampus is associated
easily discern between spatial locations that were sepa- with a larger increase in the number of dentate and
rated by greater distances (Gilbert, Kesner, & Lee, CA1 neurons rather than of CA3 neurons (West, 1990),
2001). The notion that the DG is necessary for distin- such that the high intrinsic connectivity within the CA3
guishing between similar spatial locations is also sup- subregion can be maintained.
ported by studies of place cells in the DG. Pronounced The high connection probability within the network
changes in dentate firing patterns are observed for of excitatory neurons gives rise to a recurrent or autoas-
small differences in sensory input. Very separate neuro- sociative architecture. Recurrent networks can perform
nal representations can therefore be generated in con- several classes of computations (Amit, 1989). First, they
ditions of high input similarity to the network. Pattern can hold information online when a subgroup of CA3
separation can also be observed in the CA3 network, cells activates other subgroups of CA3 cells. Neuronal
but CA3 requires a higher degree of input difference activity patterns can thus reverberate within the circuit
before it responds in this way. The dentate network can to sustain a short-term memory. Second, the ongoing
thus perform pattern separation in conditions in which reverberation can be used to associate incoming pat-
the neuronal activity in CA3 is not yet distinct (figure terns with patterns that are intrinsically retained. Third,
46.3B; Guzowski, Knierim, & Moser, 2004; Leutgeb, the network can also perform a long-term memory func-
Leutgeb, Moser, & Moser, 2007). tion. Long-term memory relies on first storing a configu-
Pattern separation in the dentate/CA3 region can ration of synaptic strengths during ongoing activity
also be observed with high-resolution functional MRI within the circuit. When the recurrent network is later
(fMRI) when human subjects are presented with images presented with incomplete inputs or with inputs that
of objects that were not seen before (novel), were a somewhat differ from previously learned patterns, the
repetition of a previously viewed object (repeat), or subset of neurons that is first activated can iteratively
were similar but not identical to a previously viewed recover the neuronal activity pattern that fully corre-
object (lure; figure 46.3C). Results revealed that activity sponds to the pattern that was stored at the time of
in the dentate/CA3 region increased during the viewing learning (McNaughton & Morris, 1987). The recovery
of lure images to the same extent as during the viewing of complete neuronal firing patterns from either partial
of novel images (Bakker, Kirwan, Miller, & Stark, 2008). inputs or from partially distinct inputs supports memory
This increase in neural activity was not observed in retrieval and is a neural computation that is referred to
other hippocampal subregions (figure 46.3D). The pro- as “pattern completion” (figure 46.3A). Because pattern
cessing of lures as novel events in the dentate/CA3 completion moves network activity toward a stored firing
region supports the notion that this region is biased pattern, it must also shift neural activity away from other
toward pattern-separation processes during the passive stored patterns. Pattern completion and pattern separa-
encoding of information. Although fMRI imaging in tion thus occur in concert, and the transitions between
humans currently does not have the resolution to sepa- the modes can be sudden (figure 46.3B). However, the
rate the DG from CA3 (Carr, Rissman, & Wagner, 2010), firing patterns of CA3 cells can also respond gradually
the convergent evidence from the data in humans and for sensory configurations that are intermediate to
animals points toward a central role of the competitive learned configurations (J. K. Leutgeb et al., 2005a).
DG network for pattern separation. Importantly, gradual changes in activity patterns not only
code for intermediate sensory inputs, but can also code
Processing within CA3 CA3 receives direct inputs from task parameters that are relevant for memory perfor-
the DG and from the entorhinal cortex and sends mance (Allen, Rawlins, Bannerman, & Csicsvari, 2012).
forward projections to the hippocampal CA1 area and
back-projections to mossy cells in the dentate hilus. In The Output Areas of the Hippocampus
addition, the most exceptional feature of the CA3 sub-
region is the major intrinsic CA3 to CA3 projection. The CA2 subregion Based upon anatomical analysis of
Because the number of synapses that a neuron can hippocampal neurons, several CA subregions have
C D
Figure 46.3 Distinct computations in hippocampal sub- patterns. (C) Examples of sample stimuli (original and lure
fields. (A) Pattern separation mechanisms transform similar versions) viewed by patients during fMRI imaging. (D) Left:
input patterns into more orthogonal output patterns (top). Regions of activity in the medial temporal lobe are overlaid
Pattern completion processes achieve the opposite and in white in the DG/CA3 region (gray surrounded by solid
recover the original input pattern despite subtle changes in white line) and in the CA1 region (black surrounded by
the input pattern (bottom). (Adapted from Yassa & Stark, dashed white line). Right: Mean activity in each region of
2011, with permission from Elsevier.) (B) Output of a neural interest for each trial condition (First presentation, Repeat,
network while gradually changing the input pattern. Different and Lure). In the DG/CA3 region, lures elicit similar levels
computations are observed in each hippocampal subregion. of activity as the first presentation, which indicates pattern
The CA1 network (stippled line) reflects the gradual change separation. In CA1, lures evoke the same level of activity as
in input. In DG (dashed line), subtle input differences are repeated trials, which indicates pattern completion. (Adapted
amplified and become orthogonalized. CA3 (solid black line) from Bakker et al., 2008. Reprinted with permission from
performs both pattern completion and pattern separation AAAS.)
depending on the degree of change in the incoming input
been described (Lorente de No, 1934; Ramon y Cajal, their more proximal dendrites (Chevaleyre & Siegel-
1893). Although frequently left out of standard depic- baum, 2010). The unique convergence of strong excit-
tions of the hippocampal circuit, CA2 is a distinct ana- atory drive from entorhinal cortex to CA2 may result in
tomical region based on protein- and gene-expression distinct contributions to episodic memory. Behavioral
profiles (Lein, Callaway, Albright, & Gage, 2005) and studies in mice, in which the function of the CA2 region
on its anatomical connectivity. The hippocampal CA2 was selectively silenced, have revealed a selective deficit
subregion is unique among the CA subregions in that in storing the “when” component of memories (DeVito
it receives inputs from both layer II and III of entorhi- et al., 2009; Wersinger, Ginns, O’Carroll, Lolait, &
nal cortex. In addition, unlike CA3 and CA1, CA2 Young, 2002). However, additional experiments and
neurons are more strongly excited by entorhinal cortex computational models are needed to describe how such
inputs to their distal dendrites than by CA3 inputs to a contribution to memory could be performed and how
552 memory
temporal information in CA2 may influence the neural Importantly, the neuronal population code in CA1
network mechanisms that represent time in CA1. changes over time in a way such that temporal informa-
tion can be gained without a major loss of spatial and
The CA1 subregion CA1 is essential for routing informa- contextual information (Mankin et al., 2012). The
tion from earlier processing stages in the hippocampal “where,” “what,” and “when” aspects of episodic memo-
loop back out to the neocortex. Recurrent excitatory ries can thus be simultaneously represented in the activ-
connections within the CA1 network are weak such that ity patterns of CA1 neurons. Behavioral studies in which
the processing occurs in a feedforward manner that the CA1 region was selectively lesioned also support a
transforms an input pattern (from CA3, CA2, and layer role for CA1 in binding temporal associations to other
III of entorhinal cortex) into an output pattern (to aspects of memory (Kesner, Hunsaker, & Ziegler, 2010).
subiculum and the deep layers of entorhinal cortex).
The three main excitatory inputs to hippocampal CA1 Subiculum The subiculum is the last processing stage
cells are from CA3 cells, from CA2 cells, and directly in the long loop through hippocampal subregions back
from layer III cells in the entorhinal cortex. Why would to the entorhinal cortex. The subiculum receives inputs
CA1 receive input from entorhinal cortex directly, but from CA1 pyramidal neurons, and similar to CA1
also indirectly through longer loops in which input receives inputs from layer III of LEC and MEC to sepa-
from layer II of entorhinal cortex is routed through the rate regions of the cell layer. The subiculum in turn
DG, CA3, and CA2 to CA1? A major difference in the sends projections to the deep layers of the entorhinal
organization of entorhinal inputs is that dentate, CA3, cortex (Witter, 2006). The contribution of the subicu-
and CA2 cells receive convergent inputs from layer lum to memory processing is unclear, although recent
II of LEC and MEC, while layer III projects to two sepa- human fMRI imaging studies suggest that the subicu-
rate zones within the CA1 area. The direct input from lum and CA1 are selectively activated during periods of
entorhinal cortex layer III to CA1 therefore more memory retrieval rather than periods of memory encod-
closely represents incoming cortical processing streams, ing (Carr et al., 2010). Neural mechanisms in the subic-
while the indirect pathway from CA3 may add informa- ulum that may support memory retrieval are unknown.
tion from long-term or short-term memory. The differ- However, the subiculum differs from CA1 in the degree
ential processing along the indirect pathway may result to which it supports spatial memory. Lesions of the hip-
in some degree of mismatch compared to the direct pocampus result in deficits in spatial memory, while
inputs, which can be detected in CA1 and can be lesions of the subiculum do not lead to similar deficits
flexibly used in updating its representations and in gen- (O’Mara, 2005). Such a dissociation is consistent with
erating arbitrary associations (Lisman & Grace, 2005; a circuit diagram in which all hippocampal information
McNaughton & Morris, 1987; Mizumori, Smith, & flows through CA1, but in which CA1 can support
Puryear, 2007). circuit function by either directly projecting to entorhi-
The different input streams can also be used to inter- nal cortex or by routing information through the
leave incoming firing patterns and to generate the subiculum.
sequential activation of CA1 cells at time scales over
which ongoing neuronal activity can be associated Integrated function across medial temporal
within the hippocampal circuit (Eichenbaum, 2013; lobe subregions
O’Keefe & Recce, 1993). However, there is also a fun-
damentally different mechanism for temporal coding, By combining the unique computations of neural net-
which does not depend on continuously sustained hip- works within each subregion, the medial temporal lobe
pocampal firing. It has been proposed that a stable generates neuronal processing streams that enable the
neural code can be compared to a neural code that, acquisition and initial retention of fact and event mem-
when reinstated, has changed over time (Howard & ories. The position of a subregion within the circuit, the
Kahana, 2002). Although this mechanism could be unique intrinsic neuronal architecture of a subregion,
used over any time period, the neuronal activity pat- and the specialized neuronal firing patterns within each
terns in CA1 show such drift over periods of hours to subregion determine which specialized computations
weeks. Importantly, neuronal activity in CA3 remains are performed. A particular computation may selec-
stable over the corresponding time intervals. The tively support the acquisition, storage, or retrieval of
retention of a stable reference representation in CA3 memories, or a computation may be flexible such that it
enables a comparison in which the similarity between can support memory processing across different phases.
the stable and the time-varying representation can serve For example, memory acquisition is thought to acti-
as an indication of how long ago an event occurred. vate input stages from entorhinal cortex to DG and
554 memory
Howard, M. W., & Kahana, M. J. (2002). A distributed Ramon y Cajal, S. (1893). Estructura del asta de Ammon
representation of temporal context. J Math Psychol, 46, y fascia dentata. Ann Soc Esp Hist Nat, 22.
269–299. Rolls, E. T. (1999). Spatial view cells and the representation
Kennedy, P. J., & Shapiro, M. L. (2009). Motivational states of place in the primate hippocampus. Hippocampus, 9,
activate distinct hippocampal representations to guide 467–480.
goal-directed behaviors. Proc Natl Acad Sci USA, 106, Rolls, E. T. (2010). A computational theory of episodic
10805–10810. memory formation in the hippocampus. Behav Brain Res,
Kesner, R. P., Hunsaker, M. R., & Ziegler, W. (2010). The 215, 180–196.
role of the dorsal CA1 and ventral CA1 in memory for the Rolls, E. T., & Treves, A. (1998). Neural networks and brain
temporal order of a sequence of odors. Neurobiol Learn Mem, function. New York, NY: Oxford University Press.
93, 111–116. Scoville, W. B., & Milner, B. (1957). Loss of recent memory
Killian, N. J., Jutras, M. J., & Buffalo, E. A. (2012). A map after bilateral hippocampal lesions. J Neurol Neurosurg Psy-
of visual space in the primate entorhinal cortex. Nature, 491, chiatry, 20, 11–21.
761–764. Solstad, T., Boccara, C. N., Kropff, E., Moser, M. B., &
Koenig, J., Linder, A. N., Leutgeb, J. K., & Leutgeb, S. Moser, E. I. (2008). Representation of geometric borders
(2011). The spatial periodicity of grid cells is not sustained in the entorhinal cortex. Science, 322, 1865–1868.
during reduced theta oscillations. Science, 332, 592–595. Squire, L. R., & Alvarez, P. (1995). Retrograde amnesia and
Lein, E. S., Callaway, E. M., Albright, T. D., & Gage, F. H. memory consolidation: A neurobiological perspective. Curr
(2005). Redefining the boundaries of the hippocampal Opin Neurobiol, 5, 169–177.
CA2 subfield in the mouse using gene expression and Suzuki, W. A., & Amaral, D. G. (1994). Perirhinal and para-
3-dimensional reconstruction. J Comp Neurol, 485, 1–10. hippocampal cortices of the macaque monkey: Cortical
Leutgeb, J. K., Leutgeb, S., Moser, M. B., & Moser, E. I. afferents. J Comp Neurol, 350, 497–533.
(2007). Pattern separation in the dentate gyrus and CA3 of Suzuki, W. A., & Amaral, D. G. (2004). Functional neuro-
the hippocampus. Science, 315, 961–966. anatomy of the medial temporal lobe memory system.
Leutgeb, J. K., Leutgeb, S., Treves, A., Meyer, R., Barnes, Cortex, 40, 220–222.
C. A., McNaughton, B. L., … Moser, E. I. (2005a). Pro- Tsao, A., Moser, M. B., & Moser, E. I. (2013). Traces of
gressive transformation of hippocampal neuronal represen- experience in the lateral entorhinal cortex. Curr Biol, 23,
tations in “morphed” environments. Neuron, 48, 345–358. 399–405.
Leutgeb, S., Leutgeb, J. K., Barnes, C. A., Moser, E. I., Tulving, E. (1983). Elements of episodic memory. Oxford, UK:
McNaughton, B. L., & Moser, M. B. (2005b). Indepen- Clarendon Press.
dent codes for spatial and episodic memory in hippocam- Ungerleider, L. G., & Mishkin, M. (1982). Two cortical
pal neuronal ensembles. Science, 309, 619–623. visual systems. In D. J. Ingle, M. A. Goodale, & R. J. W.
Lisman, J. E., & Grace, A. A. (2005). The hippocampal-VTA Mansfield (Eds.), Analysis of visual behavior (pp. 549–586).
loop: Controlling the entry of information into long-term Cambridge, MA: MIT Press.
memory. Neuron, 46, 703–713. van Strien, N. M., Cappaert, N. L., & Witter, M. P. (2009).
Lorente de No, R. (1934). Studies on the structure of the The anatomy of memory: An interactive overview of the
cerebral cortex. II. Continuation of the study of the parahippocampal-hippocampal network. Nat Rev Neurosci,
ammonic system. J Psychol Neurol, 46, 113–117. 10, 272–282.
Mankin, E. A., Sparks, F. T., Slayyeh, B., Sutherland, R. J., Wersinger, S. R., Ginns, E. I., O’Carroll, A. M., Lolait,
Leutgeb, S., & Leutgeb, J. K. (2012). Neuronal code for S. J., & Young, W. S., 3rd (2002). Vasopressin V1b receptor
extended time in the hippocampus. Proc Natl Acad Sci USA, knockout reduces aggressive behavior in male mice. Mol
109, 19462–19467. Psychiatry, 7, 975–984.
McNaughton, B. L., & Morris, R. G. M. (1987). Hippocam- West, M. J. (1990). Stereological studies of the hippocampus:
pal synaptic enhancement and information-storage within A comparison of the hippocampal subdivisions of diverse
a distributed memory system. Trends Neurosci, 10, 408–415. species including hedgehogs, laboratory rodents, wild mice
Mizumori, S. J., Smith, D. M., & Puryear, C. B. (2007). Hip- and men. Prog Brain Res, 83, 13–36.
pocampal and neocortical interactions during context dis- Witter, M. P. (2006). Connections of the subiculum of the
crimination: Electrophysiological evidence from the rat. rat: Topography in relation to columnar and laminar orga-
Hippocampus, 17, 851–862. nization. Behav Brain Res, 174, 251–264.
O’Keefe, J., & Nadel, L. (1978). The hippocampus as a cognitive Witter, M. P. (2007). The perforant path: Projections from
map. Oxford, UK: Oxford University Press. the entorhinal cortex to the dentate gyrus. Prog Brain Res,
O’Keefe, J., & Recce, M. L. (1993). Phase relationship 163, 43–61.
between hippocampal place units and the EEG theta Witter, M. P., & Moser, E. I. (2006). Spatial representation
rhythm. Hippocampus, 3, 317–330. and the architecture of the entorhinal cortex. Trends Neuro-
O’Mara, S. (2005). The subiculum: What it does, what it sci, 29, 671–678.
might do, and what neuroanatomy has yet to tell us. J Anat, Wood, E. R., Dudchenko, P. A., & Eichenbaum, H. (1999).
207, 271–282. The global record of memory in hippocampal neuronal
O’Reilly, R. C., & McClelland, J. L. (1994). Hippocampal activity. Nature, 397, 613–616.
conjunctive encoding, storage, and recall: Avoiding a trade- Yassa, M. A., & Stark, C. E. (2011). Pattern separation in the
off. Hippocampus, 4, 661–682. hippocampus. Trends Neurosci, 34, 515–525.
Piatti, V. C., Ewell, L. A., & Leutgeb, J. K. (2013). Neuro- Zola-Morgan, S., & Squire, L. R. (1993). Neuroanatomy of
genesis in the dentate gyrus: Carrying the message or dictat- memory. Annu Rev Neurosci, 16, 547–563.
ing the tone. Front Neurosci, 7, 50.
ABSTRACT In this chapter we discuss the role of context in are sensory representations of words that persist only
organizing episodic memories. We define context as slowly while the word is being presented; above, there are
drifting information (i.e., information that persists over a rela-
representations of thoughts or mental states that persist
tively long time scale in a person’s brain; for example, a
representation of a person’s location). First, we use the tem- over longer time scales.
poral context model of memory search (Howard & Kahana, The central idea of this chapter is that slowly drifting
2002) to illustrate how binding slowly drifting contextual information (i.e., information that persists over rela-
information to more transient representations (e.g., of studied tively long time scales) can be used to time-stamp and
words) serves to organize our memories of the more transient organize more quickly drifting information. This time-
information. We next present electrophysiological studies
examining the role of slowly drifting representations in orga- stamping is accomplished by means of the hippocam-
nizing episodic memories, and we provide an overview of the pus binding co-active representations together (O’Reilly
brain systems involved in representing slowly drifting informa- & Rudy, 2001), including representations that are drift-
tion. Finally, we discuss sources of variability in the rate of ing at different rates. In the scenario shown in figure
contextual drift. 47.1, these bindings allow words to cue retrieval of co-
active contextual threads and vice versa. For example, if
Context is simultaneously one of the most fundamental the participant recalls the word shark, he or she might
and elusive concepts in memory research. Memory also recall being hungry when that word was presented,
researchers often define context by exclusion: in a which in turn might trigger retrieval of the fact that leaf
memory experiment, there is a set of items that the was also presented while the person was hungry. When
participant is being asked to memorize (e.g., a list of leaf is retrieved, this might trigger retrieval of having
words), and then there is context, which reflects every- been worried about exams, and then the participant
thing else that is represented in the person’s brain might recall also hearing the word skull when worrying
during the experiment. Context might include, for about exams, and so on. The set of contextual threads
example, information about the external environment, that were active during a previously experienced event
mood, thoughts about recently encountered items, (e.g., studying the word skull) constitutes a unique
plans concerning the future, and incidental features of time-stamp for that event, and we will refer to the
the stimuli such as the color and the spatial location of process of reactivating these contextual threads as con-
a word on the screen (for a review, see Smith & Vela, textual reinstatement. Effectively, this contextual reinstate-
2001). This list makes it clear that defining context in ment process allows the participant to mentally “jump
terms of the type of information being represented is back in time” to when the contextual threads were ini-
futile. Under this definition, anything can be context— tially active. If the participant succeeds in reinstating a
for example, if you test memory for which colors were large number of contextual threads that were linked to
seen, rather than word identity, then the background an item at study, they will all convergently cue the asso-
colors become the items and word identities become ciated event, thereby boosting the probability of retriev-
part of the context. ing the event.
Instead of defining context in terms of specific types To understand the role of context in episodic memory,
of information (and the roles they play in particular we need to understand the factors that give rise to slow
experiments), we focus here on the time scale of infor- drift in the brain, and the role that slow drift plays in
mation representation. Figure 47.1 illustrates, in time- time-stamping memories. Speaking generally, there are
line form, what might be going through a person’s head two different (and non-mutually exclusive) ways to get
as he or she learns a list of words in a typical memory slow drift. The first possibility is that slow drift can arise
experiment. The figure shows how information is rep- from the brain representing slowly drifting features of
resented at different time scales. At the bottom, there the world. For example, because we cannot teleport
manning, kahana, and norman: the role of context in episodic memory 557
Testing room
Itchy
Exams
Hungry
Weather
Figure 47.1 Illustration of contextual drift. A hypothetical external stimuli (e.g., the testing room) as well as internal
participant studies the seven words in the bottom row over states (feeling itchy, worrying about upcoming exams, etc.).
the course of five minutes (times are indicated by the digital These thoughts persist for different amounts of time and can
clock faces). In addition to thoughts related to the studied overlap.
words, the participant experiences thoughts related to other
Explaining behavioral memory data using the to recall the items in any order. Free recall is a useful
temporal context model test bed for models of context and memory because it
provides data regarding both accuracy (i.e., which
The worth of a psychological theory may be measured memories were recalled) and the order in which items
by how well it explains detailed patterns of behavioral were recalled.
and neural data. Toward this end, researchers have built To provide an intuition for how slow drift and con-
computational models that instantiate the principles textual reinstatement can account for behavioral
outlined above (slow drift and contextual reinstate- memory data, we will focus our discussion on one spe-
ment) and fit the models to detailed patterns of memory cific model, the temporal context model (TCM; Howard
data. Here we focus on modeling data from the free-recall & Kahana, 2002), schematized in figure 47.2. TCM is a
paradigm, followed by a brief survey of other relevant member of a large family of models that incorporate
paradigms. In the free-recall paradigm, participants slow drift and contextual reinstatement (e.g., Davelaar,
study lists of items (typically words) and then attempt Goshen-Gottstein, Ashkenazi, Haarmann, & Usher,
558 memory
2005; Dennis & Humphreys, 2001; Polyn, Norman, & recency effect, which refers to participants’ ability to more
Kahana, 2009; Sederberg, Howard, & Kahana, 2008; easily retrieve information pertaining to recent experi-
Shankar & Howard, 2012). The purpose of this section ences (e.g., items from the end of a just-studied list)
is not to differentiate between models within this than information pertaining to long-ago experiences
family, but rather to illustrate these models’ shared (Murdock, 1962). The contiguity effect is another funda-
predictions. mental regularity in free recall, and refers to partici-
In TCM, there are two interconnected layers of nodes pants’ tendency to successively recall items that occupied
(or computing elements). These two layers represent nearby positions in the study lists (Kahana, 1996). For
the same information but at different time scales. The example, if a list contained the sub-sequence apple cat
item layer represents the item that is currently being boat and the participant recalled the word cat, it is far
studied on the list, and the context layer represents a more likely that the next response would be either boat
running average of recently studied items. As the par- or apple than some other list item.
ticipant studies each item on a list (e.g., the words Both the recency effect and the contiguity effect
apple, cat, boat, dog), the corresponding node in the exhibit time-scale invariance, meaning that these same
item layer is activated. In the figure, the participant is patterns are observed at short and long time scales. For
currently viewing the word dog, so the dog node is example, recency effects are observed in immediate
active (denoted by bright shading) and the apple, cat, free-recall experiments (where participants are given a
and boat nodes are inactive (denoted by darker list of words and are tested immediately afterwards) and
shading). As each node of the item layer is activated, also in situations where the items on the list are spaced
the item becomes associated with the current state of out across multiple days (Glenberg, Bradley, Kraus, &
the context layer. The context layer is continually Renzaglia, 1983) or even longer (Moreton & Ward,
updated by averaging together the current item repre- 2010). Likewise, contiguity effects are observed in
sentation with the previous state of context. Because the immediate free recall and also in continual distractor free
context layer computes a running average, the state of recall, where participants perform a distracting task after
context evolves gradually as the participant studies studying each item (Bjork & Whitten, 1974; Howard &
the list. Kahana, 1999). Contiguity effects have even been
Recall is simulated in TCM by using the current state observed across word lists (i.e., given that a participant
of context as a retrieval cue. Each item’s node in the has just recalled an item from the fourth list, he or she
item layer is activated according to how similar that is more likely to recall the next item from the third or
item’s associated state of context is to the current state fifth list than from the second or sixth lists; Howard,
of context. Items stochastically compete to be recalled, Youker, & Venkatadass, 2008).
with more active items serving as stronger competitors. The recency and contiguity effects (along with some
When an item wins the recall competition, two things degree of time-scale invariance) emerge naturally out
happen. First, the recalled item is incorporated into the of retrieved-context models of episodic memory like
current state of context (e.g., if you recall dog, that is TCM. According to these models, recency effects arise
blended into the context vector). Second, the current because the context present at test is relatively more
state of context is also updated with retrieved mental similar to the context associated with the end-of-list
context (i.e., the state of mental context that was linked items than the context associated with earlier items.
to the just-retrieved item at study). The updated state This fact is true regardless of whether the items were
of context is then used to probe memory for other presented within a few seconds of each other or days
items. As discussed earlier in this chapter, reinstating apart, which explains why the model predicts recency
an item’s study-phase context can be construed as effects at both short and long time scales.
mentally jumping back to the moment when the just- Retrieved-context models like TCM explain the con-
recalled item was studied. Cuing with reinstated contex- tiguity effect in terms of slow drift at study and contex-
tual information increases the chances that the next tual reinstatement at test. When participants retrieve an
item recalled will be one associated with a similar item, they retrieve contextual features associated with
context. that item, and then they use the retrieved contextual
features to probe their memories for more items. Due
Modeling Key Regularities in Free-Recall Behav- to slow drift at study, items studied nearer in time will
ior TCM and similar models have enjoyed extensive have been associated with (relatively) more similar
success at explaining many stereotyped behaviors states of context than far-apart items. Just as with the
observed during free recall and other episodic memory- recency effect, this relative match property holds
related tasks. One fundamental regularity is termed the regardless of the time delay between successive item
manning, kahana, and norman: the role of context in episodic memory 559
presentations, explaining why retrieved-context models directed-forgetting paradigm suggests that when partici-
predict contiguity effects at a variety of time scales. pants are asked to forget list A items prior to studying
The fact that context models can explain recency and list B, they respond to this forget cue instruction by delib-
contiguity does not mean that contextual drift is always erately shifting their state of context, thereby making
responsible for these effects. For example, it may be the list A items less accessible (for a review of the
possible to explain recency effects in immediate free relevant evidence see Sahakyan, Delaney, Foster, &
recall in terms of participants actively maintaining end- Abushanab, 2013).
of-list items in working memory, and then directly Contextual matching may also be used to explain
recalling these items from working memory (e.g., Atkin- how people judge the order in which a series of events
son & Shiffrin, 1968). Likewise, it is possible to explain occurred. Specifically, if our current context is more
contiguity effects in immediate free recall in terms of similar to the context associated with event y than with
participants directly forming links between adjacently event x, we may judge event y to have occurred more
studied items. However, these alternative accounts recently than event x (see discussion of Manns, Howard,
cannot explain why recency and contiguity effects & Eichenbaum, 2007, below, for neural data relating
persist across long time scales. recency judgments to contextual drift).
Importantly, many retrieved-context models (includ-
ing TCM) are not fully time-scale invariant. For example, Electrophysiological Evidence for Slow Drift
TCM has a characteristic contextual drift rate which and Contextual Reinstatement Recent advances
determines how rapidly the state of context evolves to in neural recording and analysis methods have allowed
incorporate new information. Events that occur at time researchers to test whether neural patterns during epi-
scales slower than this drift rate will not show substantial sodic memory experiments are consistent with retrieved-
recency or contiguity effects. A recently developed context models. As described above, retrieved-context
theory from Shankar and Howard (2012) extends TCM models predict that recalling a studied item should lead
to account for precise timing and full time-scale invari- to the reinstatement of a gradually evolving contextual
ance by positing that drift occurs across a spectrum of representation. To test this prediction, Manning, Polyn,
time scales (also see Howard & Eichenbaum, 2013). Baltuch, Litt, and Kahana (2011) recorded electrical
signals from electrodes implanted throughout the
Other Findings That Can Be Addressed Using brains of human neurosurgical patients as they partici-
This Framework While modelers interested in how pated in a delayed free-recall experiment (figure 47.3A,
context shapes episodic memory have focused primarily B). The researchers first sought to isolate slowly drifting
on free recall, it is important to note that contextual neural patterns that might be involved in representing
tagging and contextual reinstatement contribute to context. Next, the researchers examined whether those
some degree to virtually every episodic memory para- (putative) context representations were reinstated as
digm. For example, in tests of recognition memory, the participants recalled the studied words.
presenting information about a past event (e.g., display- The researchers isolated candidate context represen-
ing a previously studied word) can reinstate the context tations by identifying temporally autocorrelated neural pat-
in which the word was studied, such that words studied terns (i.e., patterns that were more similar during the
in similar contexts are subsequently remembered more study of nearby words than temporally distant words) as
easily (Schwartz, Howard, Jing, & Kahana, 2005). Con- the participants studied the words. After the patients
textual reinstatement can also lead to memory errors. had studied and recalled words from many lists, the
For example, suppose that a participant studies two lists researchers computed the similarity between the neural
of items, A and B. If the participant is reminded of patterns recorded just prior to recalling a word and the
studying list A prior to studying list B, the contextual neural patterns recorded at study.
threads associated with list A may be reinstated and If the neural patterns recorded as a participant recalls
bound to the list B items. This binding can lead partici- an item reflect only item information, then they should
pants to misattribute memories of list B items to list A match the neural patterns recorded when the item was
(Gershman, Schapiro, Hupbach, & Norman, 2013; studied and also possibly afterwards (to the extent that
Hupbach, Gomez, Hardt, & Nadel, 2007; Sederberg, the item representations persisted in the participant’s
Gershman, Polyn, & Norman, 2011). Just as contextual brain). In contrast, if the neural pattern at retrieval
reinstatement can facilitate access to memories linked reflects reinstated, slowly drifting context from the
to those contextual threads, contextual shifts can study phase, then it should match the patterns recorded
inhibit access to memories associated with out-of-date both before and after that item was studied, with similar-
contextual threads. Evidence from the list-method ity falling off gradually in both directions. Figure 47.3C
560 memory
A Study Recall
1 2 15 16 17 18 19 20 1 2 3 4
MUD SKI ANT TAPE NOSE VASE SHEEP PHONE “Sheep” “Nose” “Sword” “Snail”
Normalized PCA
56
activity
ECoG
scores
–17
B C D
0.06
of context reinstatement
r = 0.42
4
p = 0.007
Neural similarity
Neural signature
0.04
2
0.02
0
0.00
–0.02 –2
–4 –2 0 2 4 0.4 0.5 0.6 0.7 0.8
Lag Contiguity effect
Figure 47.3 Neural evidence for contextual reinstatement electrode implanted in the brain of a neurosurgical patient.
in humans. (A) After studying a list of 15 or 20 words and (C) Similarity between the activity recorded during recall of
performing a brief distraction task, participants recalled as a word from serial position i and study of a word from serial
many words as they could remember, in any order. ECoG position i + lag (the black dot denotes the study and recall of
activity was recorded during each study and recall event. The the same word; i.e., lag = 0). (D) Participants who exhibited
similarity between the recorded patterns was computed as a stronger neural signatures of context reinstatement also
function of lag. (B) Each dot marks the location of a single exhibited more pronounced contiguity effects.
shows that the data matched this latter pattern, thereby cause neural patterns in PFC to change more slowly
supporting retrieved-context models. Furthermore, than they would otherwise (see Ranganath, Hasselmo,
the degree to which individual patients exhibited this & Stern, chapter 44 in this volume). In keeping with
neural signature of contextual reinstatement was cor- this idea, a number of neural-recording studies have
related with the behavioral contiguity effect (i.e., par- found direct evidence that neural patterns in PFC drift
ticipants’ tendency to successively recall neighboring slowly. For example, Hyman, Ma, Balaguer-Ballester,
list items; figure 47.3D). In a related study, Howard, Durstewitz, and Seamans (2012) recorded from dozens
Viskontas, Shankar, and Fried (2012) collected extracel- of neurons in the rodent medial prefrontal cortex
lular recordings from various medial temporal lobe (mPFC) as rats navigated in two environments. They
regions in humans during a recognition memory test, found that the firing rates of mPFC neurons were tem-
and found a similar neural signature of contextual porally autocorrelated. Furthermore, Jenkins and Ran-
reinstatement. ganath (2010) found that patterns of functional MRI
(fMRI) activity in the right lateral PFC drifted slowly
while participants studied lists of pictures; the rate of
Brain systems involved in representing context neural drift predicted how accurately participants could
remember when particular pictures were presented
Prefrontal Cortex As reviewed by Polyn and over the time course of the experiment. Converging
Kahana (2008), prefrontal cortex (PFC) has several evidence for the PFC’s role in representing contextual
properties that make it an especially good candidate for information comes from studies of patients with frontal
representing contextual information. In particular, PFC lobe damage. These studies have found that damage to
can actively maintain patterns of neural firing in the frontal regions impairs memory performance on
face of distraction (e.g., Miller, Erickson, & Desimone, strongly contextually mediated tasks like free recall,
1996). This capacity for active maintenance should whereas performance is relatively spared on tasks where
manning, kahana, and norman: the role of context in episodic memory 561
context plays a lesser role (e.g., Shimamura, 1994; (e.g., sensory representations of words on a study list)
Wheeler, Stuss, & Tulving, 1995). to more slowly drifting information, thereby making it
possible for studied items to cue reinstatement of slowly
MTL Structures Studies have found evidence for drifting contextual information, and vice versa (Cohen
slowly drifting patterns of neural activity in several & Eichenbaum, 1993; Diana, Yonelinas, & Ranganath,
medial temporal lobe structures, including parahippo- 2007; O’Reilly & Rudy, 2001). To play this binding role,
campal cortex (PHC) and hippocampus (e.g., Howard the hippocampus needs to receive inputs from areas
et al., 2012). We elaborate on the roles of each of these representing slowly drifting information. This is accom-
structures below (for additional discussion of MTL plished via connections from areas like PFC and PHC
contributions to episodic memory, see chapter 45, this that go through the entorhinal cortex (which itself
volume). exhibits gradually evolving neural patterns; Egorov,
Hamam, Fransen, Hasselmo, & Alonso, 2002) into the
Parahippocampal cortex and the posterior medial system hippocampus.
Several recent papers have argued that PHC represents A recent study by Manns et al. (2007) provides clear
a person’s inference about the situation they are cur- evidence that patterns of hippocampal firing evolve
rently in (e.g., reading a book, listening to music, gradually over time, and that these gradually evolving
cooking dinner, etc.). For example, Bar and Aminoff patterns are behaviorally relevant. In their study, Manns
(2003) found that the PHC shows greater activation in et al. (2007) simultaneously recorded activity from mul-
response to objects that are strongly diagnostic of situ- tiple neurons in the CA1 hippocampal subregion as rats
ational context (e.g., a roulette wheel or a beach chair) sequentially sampled a “list” of odors. After sampling
than to objects without a strong associated context the sequence of odors, the rat had to choose which of
(e.g., a cherry or a fly), although see Epstein and Ward two odors in the sequence had been presented more
(2010) for an alternative interpretation. Insofar as a recently. The authors found that patterns of neural
person’s representation of the situation they are in firing in CA1 changed gradually as animals sampled the
changes gradually over time (except at event boundar- odors, and that the degree of neural drift over the
ies; see “What Drives Contextual Drift?” below), brain course of the list predicted behavioral accuracy on a
regions that represent situational information should recency-discrimination test. The authors interpret this
exhibit gradually changing neural patterns. In keeping finding in terms of the idea that greater neural drift
with this view, the slow drift property predicted by indicates greater contextual separation, which (in turn)
context models has been demonstrated in PHC using makes it easier to temporally discriminate between
both electrophysiology (Howard et al., 2012) and fMRI items on the recency test. Notably, in addition to
(Turk-Browne, Simon, & Sederberg, 2012). showing a within-list neural drift effect, Manns et al.
Although PHC plays an important role in represent- (2007) also observed slow drift across lists of odors (for
ing situational context, it is not the only such region. As discussion, see Howard & Eichenbaum, 2013); this fits
reviewed by Ranganath and Ritchey (2012), PHC is part with the idea (mentioned above in the section “Explain-
of a densely interconnected network of regions called ing Behavioral Memory Data Using the Temporal
the posterior medial system that includes retrosplenial Context Model”) that context drifts at multiple time
cortex, the mamillary bodies, anterior thalamic nuclei, scales (Shankar & Howard, 2012). For a related finding
presubiculum, parasubiculum, posterior cingulate, pre- showing slow drift in CA1, see Mankin et al. (2012).
cuneus, angular gyrus, and ventromedial PFC. Ranga-
nath and Ritchey (2012) propose that structures in the Temporal Receptive Windows In addition to the
posterior medial system work together to match incom- aforementioned regions, how can we discover other
ing cues about the current context to internal situation areas involved in representing context? Naïvely, one
models that specify the spatial, temporal, and causal rela- could just look for regions exhibiting gradually evolving
tionships that define specific situations. For example, neural patterns using fMRI. The problem with this
the situation model for going to a movie might describe approach is that the fMRI signal is constrained to drift
the properties of movie theaters, the typical time slowly due to the sluggishness of the blood flow response
sequence of events during a movie-going outing, movie that it measures, regardless of the drift rates of the
theater etiquette, and so on (Zacks, Speer, Swallow, underlying neurocognitive processes. Therefore the
Braver, & Reynolds, 2007; Zwaan & Radvansky, 1998). mere presence of slow drift in the fMRI signal is not
diagnostic of slow drift in the person’s thoughts.
The hippocampus The hippocampus is the key structure To address this problem, Hasson, Yang, Vallines,
responsible for binding quickly drifting information Heeger, and Rubin (2008) devised a new technique for
562 memory
measuring a brain region’s sensitivity to information at found that (controlling for elapsed time) participants
different time scales. Instead of directly measuring the are impaired at recalling details from the previous event
drift rate of neural patterns, they measured the history- compared to the current event (e.g., Radvansky & Cope-
dependence of neural activity in a region. Specifically, land, 2006; Swallow, Zacks, & Abrams, 2009). Using a
they manipulated what came before a particular stimu- long-term memory paradigm, Ezzyat and Davachi
lus (by rearranging scenes in a movie), and asked (2011) found that participants had difficulty recalling
whether the response of a region to a particular scene associations between adjacent sentences that spanned
was altered by changing the scenes that came before it. an event boundary, compared to sentences that were
For example, if a region’s response to a scene is altered part of the same event. These behavioral findings are
by changing what happened 5 minutes previously (but consistent with the idea that event boundaries induce a
not 10 minutes ago), this indicates that the region sharp discontinuity in context, resulting in decreased
retains information from 5 minutes ago (but not 10 accessibility of details from the previous event and also
minutes ago). Hasson et al. (2008) define the temporal decreased contiguity effects in long-term memory tests
receptive window (or TRW) for a particular region as the (for relevant neural evidence, see Swallow et al., 2011).
length of time, prior to the stimulus presentation, Explaining these event-boundary effects poses a
during which the presentation of other information major challenge for computational models of contex-
may affect the neural response to the stimulus. tual drift. Computational models that update context
Using a variant of this approach, Lerner, Honey, via a simple integration process (i.e., by computing a
Silbert, and Hasson (2011) found that the temporal running average of recently encountered stimuli) posit
parietal junction (including the angular gyrus and that, when the situation shifts, information about previ-
supramarginal gyrus) and precuneus were sensitive to ously encountered stimuli will gradually fade out of
the previous sentence in an auditory story, and medial context rather than exhibiting a rapid shift. As dis-
PFC showed an even longer TRW (extending to the cussed by Polyn et al. (2009), this gradual fade is not
previous paragraph and possibly further). Hasson et al. enough to explain the sharp drop in recall observed at
(2008) identified a similar set of “long TRW” regions event boundaries. Polyn et al. (2009) created situational
using a movie stimulus, including precuneus and the shifts at study by having participants switch (multiple
temporal parietal junction. Importantly, there is strong times) between encoding tasks as they studied lists of
overlap between the set of long TRW regions and the words, and then had participants freely recall the
posterior medial network regions identified by Ranga- studied items. To model the effects of these task switches
nath and Ritchey (2012). We should emphasize that on free recall, Polyn et al. (2009) had to incorporate an
regions can show long TRWs for a variety of reasons— extra context-disruption mechanism that was triggered
for example, a region might have a long TRW because whenever participants switched between encoding
it has intrinsic integrator properties (e.g., Arnsten, tasks.
Wang, & Paspalas, 2012), or because it can actively While this context-disruption mechanism helps to fit
maintain specific patterns of activity (as in PFC), or the data, it does not provide a clear mechanistic account
because it is receiving information from other regions of why context is disrupted at event boundaries (it just
involved in memory storage (e.g., the hippocampus). posits that it happens). Modeling work by Shankar, Jaga-
disan, and Howard (2009) may provide some insight
What drives contextual drift? into this issue. Their model (which they call pTCM, for
predictive TCM) modifies TCM such that, instead of
One of the main goals of theories of context and updating context with item information, context is
memory is understanding variability in contextual drift updated with a prediction of which items will be pre-
by explaining the circumstances that result in mental sented next. Insofar as event boundaries are marked by
context changing more or less quickly. A key implica- sharp changes in predictions (i.e., what you predict at
tion of the situation model view described above in the end of one event is very different from what you
the section “Brain Systems Involved in Representing predict at the beginning of another), this model may
Context” is that mental context will change sharply be able to simulate the findings described above (also
when a person’s (inferred) situation changes. see Reynolds, Zacks, & Braver, 2007).
This view is supported by data showing that event
boundaries (moments when participants infer a change Conclusion
in their situation; e.g., shifting from eating dinner to
washing dishes) can cause forgetting. In the event- Context has long been the “dark matter” of memory
processing literature, several behavioral studies have theories. Researchers have found it necessary to posit a
manning, kahana, and norman: the role of context in episodic memory 563
gradually evolving context representation in order to Epstein, R., & Ward, E. (2010). How reliable are visual
explain patterns of memory data from free recall and context effects in the parahippocampal place area? Cereb
Cortex, 20(2), 294.
other tasks. This gradually evolving representation (and
Ezzyat, Y., & Davachi, L. (2011). What constitutes an episode
the idea that it can be reinstated during retrieval) is the in episodic memory? Psychol Sci, 22(2), 243–252.
glue that holds together most modern theories of Gershman, S. J., Schapiro, A. C., Hupbach, A., &
memory retrieval. However, until recently, no one had Norman, K. A. (2013). Neural context reinstatement
been able to observe contextual drift or reinstatement predicts memory misattribution. J Neurosci, 33(20),
directly. Instead, the role of context during memory 8590–8595.
Glenberg, A. M., Bradley, M. M., Kraus, T. A., &
encoding and retrieval had been something indirectly Renzaglia, G. J. (1983). Studies of the long-term recency
inferred through its effects on behavioral memory effect: Support for a contextually guided retrieval theory. J
performance. Exp Psychol Learn Mem Cogn, 12, 413–418.
In this chapter, we have reviewed recent progress in Hasson, U., Yang, E., Vallines, I., Heeger, D. J., & Rubin,
the cognitive neuroscience of memory that has allowed N. (2008). A hierarchy of temporal receptive windows in
human cortex. J Neurosci, 28(10), 2539–2550.
us to start bringing the dark matter of context into the Howard, M. W., & Eichenbaum, H. (2013). The hippocam-
light. Neurophysiological and fMRI studies have given pus, time, and memory across scales. J Exp Psychol Gen,
us a much better idea of which regions are most strongly 142(4), 1211–1230.
involved in representing contextual information and, Howard, M. W., & Kahana, M. J. (1999). Contextual vari-
more importantly, they have given us the ability to track ability and serial position effects in free recall. J Exp Psychol
Learn Mem Cogn, 25, 923–941.
how neural activity drifts within those regions. In the
Howard, M. W., & Kahana, M. J. (2002). A distributed
coming years, the ability to track this drift and relate it representation of temporal context. J Math Psychol, 46,
to memory behavior will allow us to develop even more 269–299.
powerful models of how our brains time-stamp our Howard, M. W., Viskontas, I. V., Shankar, K. H., & Fried,
memories and how these time-stamps allow us to retrieve I. (2012). Ensembles of human MTL neurons “jump back
in time” in response to a repeated stimulus. Hippocampus,
information concerning the past.
22, 1833–1847.
Howard, M. W., Youker, T. E., & Venkatadass, V. (2008).
The persistence of memory: Contiguity effects across
hundreds of seconds. Psychon Bull Rev, 15, 58–63.
REFERENCES
Hupbach, A., Gomez, R., Hardt, O., & Nadel, L. (2007).
Arnsten, A. F. T., Wang, M. J., & Paspalas, C. D. (2012). Reconsolidation of episodic memories: A subtle reminder
Neuromodulation of thought: Flexibilities and vulnerabili- triggers integration of new information. Learn Mem, 14,
ties in prefrontal cortical network synapses. Neuron, 76, 47–53.
223–239. Hyman, J., Ma, L., Balaguer-Ballester, E., Durstewitz, D.,
Atkinson, R. C., & Shiffrin, R. M. (1968). Human memory: & Seamans, J. (2012). Contextual encoding by ensembles
A proposed system and its control processes. In K. W. of medial prefrontal cortex neurons. Proc Natl Acad Sci USA,
Spence & J. T. Spence (Eds.), The psychology of learning and 109(13), 5086–5091.
motivation (Vol. 2, pp. 89–105). New York, NY: Academic Jenkins, L. J., & Ranganath, C. (2010). Prefrontal and
Press. medial temporal lobe activity at encoding predicts temporal
Bar, M., & Aminoff, E. (2003). Cortical analysis of visual context memory. J Neurosci, 30(46), 15558–15565.
context. Neuron, 38, 347–358. Kahana, M. J. (1996). Associative retrieval processes in free
Bjork, R. A., & Whitten, W. B. (1974). Recency-sensitive recall. Mem Cognit, 24, 103–109.
retrieval processes in long-term free recall. Cognit Psychol, 6, Lerner, Y., Honey, C. J., Silbert, L. J., & Hasson, U. (2011).
173–189. Topographic mapping of a hierarchy of temporal recep-
Cohen, N. J., & Eichenbaum, H. (1993). Memory, amnesia, and tive windows using a narrated story. J Neurosci, 31(8),
the hippocampal system. Cambridge, MA: MIT Press. 2906–2915.
Davelaar, E. J., Goshen-Gottstein, Y., Ashkenazi, A., Mankin, E. A., Sparks, F. T., Slayyeh, B., Sutherland, R. J.,
Haarmann, H. J., & Usher, M. (2005). The demise of Leutgeb, S., & Leutgeb, J. K. (2012). Neuronal code for
short-term memory revisited: Empirical and computational extended time in the hippocampus. Proc Natl Acad Sci USA,
investigations of recency effects. Psychol Rev, 112, 3–42. 109(47), 19462–19467.
Dennis, S., & Humphreys, M. S. (2001). A context noise Manning, J. R., Polyn, S. M., Baltuch, G., Litt, B., &
model of episodic word recognition. Psychol Rev, 108, Kahana, M. J. (2011). Oscillatory patterns in temporal lobe
452–478. reveal context reinstatement during memory search. Proc
Diana, R. A., Yonelinas, A. P., & Ranganath, C. (2007). Natl Acad Sci USA, 108(31), 12893–12897.
Imaging recollection and familiarity in the medial temporal Manns, J. R., Howard, M. W., & Eichenbaum, H. (2007).
lobe: A three-component model. Trends Cogn Sci, 11(9), Gradual changes in hippocampal activity support remem-
379–386. bering the order of events. Neuron, 56(3), 530–540.
Egorov, A., Hamam, B., Fransen, E., Hasselmo, M., & Miller, E. K., Erickson, C. A., & Desimone, R. (1996).
Alonso, A. (2002). Graded persistent activity in entorhinal Neural mechanisms of visual working memory in prefrontal
cortex neurons. Nature, 420(6912), 173–178. cortex of the macaque. J Neurosci, 16, 5154.
564 memory
Moreton, B. J., & Ward, G. (2010). Time scale similarity and Sederberg, P. B., Howard, M. W., & Kahana, M. J. (2008).
long-term memory for autobiographical events. Psychon Bull A context-based theory of recency and contiguity in free
Rev, 17(4), 510–515. recall. Psychol Rev, 115(4), 893–912.
Murdock, B. B. (1962). The serial position effect of free Shankar, K. H., & Howard, M. W. (2012). A scale-invariant
recall. J Exp Psychol, 64, 482–488. internal representation of time. Neural Comp, 24, 134–193.
O’Reilly, R. C., & Rudy, J. W. (2001). Conjunctive represen- Shankar, K. H., Jagadisan, U. K. K., & Howard, M. W.
tations in learning and memory: Principles of cortical and (2009). Sequential learning using temporal context. J Math
hippocampal function. Psychol Rev, 108(2), 311–345. Psychol, 53, 474–485.
Polyn, S. M., & Kahana, M. J. (2008). Memory search and Shimamura, A. P. (1994). Memory and frontal lobe function.
the neural representation of context. Trends Cogn Sci, 12, In M. S. Gazzaniga (Ed.), The cognitive neurosciences (pp.
24–30. 803–815). Cambridge, MA: MIT Press.
Polyn, S. M., Norman, K. A., & Kahana, M. J. (2009). A Smith, S. M., & Vela, E. (2001). Environmental context-
context maintenance and retrieval model of organizational dependent memory: A review and meta-analysis. Psychon
processes in free recall. Psychol Rev, 116(1), 129–156. Bull Rev, 8(2), 203–220.
Radvansky, G. A., & Copeland, D. E. (2006). Walking Swallow, K. M., Barch, D. M., Head, D., Maley, C. J.,
through doorways causes forgetting: Situation models and Holder, D., & Zacks, J. M. (2011). Changes in events alter
experienced space. Mem Cognit, 34(5), 1150–1156. how people remember recent information. J Cogn Neurosci,
Ranganath, C., & Ritchey, M. (2012). Two cortical systems 23(5), 1052–1064.
for memory-guided behavior. Nat Rev Neurosci, 13, Swallow, K. M., Zacks, J. M., & Abrams, R. A. (2009). Event
713–726. boundaries in perception affect memory encoding and
Reynolds, J. R., Zacks, J. M., & Braver, T. S. (2007). A com- updating. J Exp Psychol Gen, 138(2), 236–257.
putational model of event segmentation from perceptual Turk-Browne, N. B., Simon, M. G., & Sederberg, P. B.
prediction. Cognitive Sci, 31, 613–643. (2012). Scene representations in parahippocampal
Sahakyan, L., Delaney, P. F., Foster, N. L., & Abushanab, cortex depend on temporal context. J Neurosci, 32(21),
B. (2013). List-method directed forgetting in cognitive and 7202–7207.
clinical research: A theoretical and methodological review. Wheeler, M. A., Stuss, D. T., & Tulving, E. (1995). Frontal
Psychol Learn Motiv, 59, 131–189. lobe damage produces episodic memory impairment. J Int
Schwartz, G., Howard, M. W., Jing, B., & Kahana, M. J. Neuropsych Soc, 1, 525–536.
(2005). Shadows of the past: Temporal retrieval effects in Zacks, J. M., Speer, N. K., Swallow, K. M., Braver, T. S., &
recognition memory. Psychol Sci, 16, 898–904. Reynolds, J. R. (2007). Event perception: A mind-brain
Sederberg, P. B., Gershman, S. J., Polyn, S. M., & Norman, perspective. Psychol Bull, 133, 273–293.
K. A. (2011). Human memory reconsolidation can be Zwaan, R. A., & Radvansky, G. A. (1998). Situation models
explained using the temporal context model. Psychon Bull in language comprehension and memory. Psychol Bull,
Rev, 18(3), 455–468. 123(2), 162–185.
manning, kahana, and norman: the role of context in episodic memory 565
48 Parietal Lobe Mechanisms Subserving
Episodic Memory Retrieval
MELINA R. UNCAPHER, ALAN M. GORDON, AND ANTHONY D. WAGNER
ABSTRACT Episodic memory enables conscious remembrance item-memory strength or familiarity—that is, the sense
of events past and the recognition of previously encountered of prior encounter that is unaccompanied by remem-
stimuli. Functional neuroimaging investigations of episodic brance of contextual detail. By contrast, ventral PPC
retrieval indicate that retrieval-related activity extends well
beyond the medial temporal lobe and prefrontal cortex, and (vPPC) is engaged during recollection—that is, the
consistently includes multiple, functionally distinct retrieval retrieval of contextual details of a prior event. While
effects in lateral posterior parietal cortex (PPC). Here we this coarse dorsal/ventral distinction in PPC old/new
review what is known about PPC activity at retrieval and how effects has prompted multiple hypotheses and moti-
this activity relates to the broad range of cognitive operations vated novel experimentation, recent meta-analyses and
subserved by PPC (e.g., attention, multifeatural binding, deci-
sion making, and action intention). We conclude by introduc-
empirical studies indicate a more fine-grained parcella-
ing a working model of PPC contributions to episodic tion of PPC retrieval-related effects (figure 48.2; e.g.,
remembering. Hutchinson, Uncapher, & Wagner, 2009; Hutchinson et
al., 2014; Sestieri, Shulman, & Corbetta, 2010), calling
Episodic memory enables humans to discriminate into question whether two-process (dorsal/ventral)
novel from previously encountered stimuli and to models are sufficient to account for PPC computations
retrieve details of past events. Given its importance for during retrieval.
everyday functioning and how it is disrupted in disease, This chapter summarizes fMRI research on PPC func-
extensive neuroimaging research has sought to delin- tion during episodic retrieval, and considers how mul-
eate the neural mechanisms that subserve episodic tiple PPC mechanisms may influence memory-guided
remembering in healthy humans. Extant functional action. We first introduce prominent hypotheses regard-
MRI (fMRI) data implicate several large-scale brain ing PPC contributions to retrieval and briefly review the
regions and their network interactions in episodic current state of the field. Critically, we argue that any
retrieval, including the medial temporal lobe (MTL), model of PPC contributions to retrieval must take into
prefrontal cortex (PFC), and posterior parietal cortex account the fine-grained functional heterogeneity in
(PPC; figure 48.1). While rich neuropsychological lit- PPC. We then offer a working model of PPC function
eratures demonstrate that MTL and PFC damage can that incorporates seemingly conflicting hypotheses.
lead to noticeable memory impairments (Shimamura, Broadly, we suggest that multiple PPC functions—
1995; Squire, 1992), lateral PPC damage is associated including mechanisms of attention, multifeatural
with subtle memory changes (Berryhill, 2012). This binding, decision making, and action intention—may
puzzling result has sparked debate about the function each explain a particular subset of PPC retrieval effects
of lateral PPC (hereafter PPC) during retrieval. and interact to support retrieval-guided behavior.
Interest in PPC contributions to episodic retrieval
emerged from event-related fMRI observations of “old/ Attention
new” or “retrieval success” effects, wherein PPC activity
(predominantly left-lateralized) is greater during recog- Attention to Memory From one perspective, PPC
nition of previously encountered stimuli as “old” (hits) old/new effects reflect the differential engagement of
relative to classification of novel stimuli as “new” (correct attention depending on retrieval outcomes. A particu-
rejections). Initial studies suggested a dorsal/ventral larly influential attention-based hypothesis—termed
axis of PPC functional organization (Cabeza, Ciara- attention to memory (AtoM; Cabeza et al., 2008; Ciara-
melli, Olson, & Moscovitch, 2008; Vilberg & Rugg, melli, Grady, & Moscovitch, 2008)—builds on a model
2008; Wagner, Shannon, Kahn, & Buckner, 2005). In of perceptual attention (Corbetta, Patel, & Shulman,
general, fMRI blood oxygen level–dependent (BOLD) 2008) in which dPPC supports goal-directed (“top-
activity in dorsal PPC (dPPC) tracks differences in down”) orienting of attention, while vPPC supports
Figure 48.2 PPC functional dissociations. (A) Hutchinson fMRI data. Line graphs depict schematics of representative
et al. (2014) revealed a quadruple dissociation of activity pat- activity patterns. Note: AnG activity tracks recollection (left
terns during retrieval: lat-IPS (red) activity showed a mono- graph) and sometimes also shows a novelty effect (right
tonic relationship with item-memory strength; AnG (blue) graph). “Remember,” Remember responses; “hi confid,” high
activity tracked recollection of event details; mIPS/SPL confidence. (B) Meta-analyses of top-down and bottom-up
(yellow) demonstrated a decision uncertainty effect, with TPJ attention effects, and recollection and familiarity-based
(green) qualitatively showing the reverse. Black outlines on retrieval effects reveal largely nonoverlapping parietal regions
the brain indicate parcellation identified by Nelson et al. supporting attention and memory (adapted from Hutchinson
(2010) using graph-theoretic analyses of resting-state and task et al., 2014). (See color plate 39.)
568 memory
stimulus-driven (“bottom-up”) reorienting of attention. expectation violations elicit activity that overlaps old/
Extending this attention model from perception to new effects in vPPC (O’Connor, Han, & Dobbins, 2010).
memory, AtoM posits that dPPC mechanisms are Similarly, old/new effects overlap with effects posited to
recruited to allocate attention to the goal of memory be a proxy for bottom-up attention (Ciaramelli et al.,
retrieval (e.g., when effortful pre- and/or post-retrieval 2010): greater vPPC activity was observed when partici-
processing is required to make a memory decision), pants were presented with recombined word pairs that
and that relevant memory cues or recollected memories may elicit a memory-based expectation violation rela-
result in vPPC bottom-up attentional capture. tive to intact word pairs (which presumably confirm
The dorsal component of AtoM garners support from memory-based expectations). Third, in a study that
studies in which demands on top-down attention at investigated (bottom-up) target detection in a memory
retrieval are thought to differ across conditions. First, task and a perceptual task, activity during mnemonic
relative to high-confidence decisions, low-confidence and perceptual target detection overlapped in vPPC
decisions, which should require greater top-down atten- (Cabeza et al., 2011). While none of these studies
tion, elicit greater dPPC activity (Cabeza et al., 2008; directly relate PPC activity during recollection or high-
Vilberg & Rugg, 2008). Second, recognition decisions confidence memory decisions to that during conditions
accompanied by familiarity may be associated with known to demand bottom-up attention, they can be
greater dPPC activity relative to recollective decisions interpreted as suggesting that vPPC-mediated atten-
largely because, in the framework of many studies, tional processes are engaged under various retrieval
familiarity-based decisions are more effortful and thus conditions.
more demanding of top-down attention (Cabeza et al.,
2008). When a recollective task was designed to be Beyond Dual-Attention Accounts While the pre-
more effortful than an item-memory task, greater dPPC ceding findings (and others) lend support to AtoM, a
activity was observed in the recollection condition growing literature demonstrates that the PPC regions
(Ciaramelli, Grady, Levine, Ween, & Moscovitch, 2010). engaged during attention tasks are not the same as
Third, attentional orienting during a visual search task those typically showing old/new effects. First, Sestieri
was shown to engage dPPC regions that are similar to and colleagues (2010) demonstrated that top-down
those engaged during a memory search task (Cabeza perceptual and mnemonic search tasks elicit activity in
et al., 2011). adjacent but nonoverlapping regions of dPPC, with the
The ventral component of AtoM also garners support perceptual task recruiting medial intraparietal sulcus
from multiple lines of evidence. First, the detection of (IPS) and the memory task recruiting lateral IPS.
recollected or high-strength mnemonic information is Second, Hutchinson and colleagues (2014) reported
proposed to be analogous to the detection of target complementary findings (figure 48.2A): while BOLD
information in perception (Ciaramelli et al., 2008); activity in angular gyrus (AnG) tracked recollection,
such retrieval events are thought to be salient and thus and activity in lateral IPS tracked item-memory
should elicit bottom-up attentional capture (putatively strength, medial IPS/superior parietal lobule (SPL)
associated with vPPC activity). Consistent with this activity tracked (1) top-down visuospatial attention
account, greater vPPC activity is observed (1) when (as evidenced by overlap with “attendotopic maps”—
recognition is accompanied by the subjective report of that is, topographic maps indicating where in visual
“remembering” the past event—putatively indicating space top-down attention is allocated) and (2) retrieval
retrieval of some detail(s) of the original study episode— decision uncertainty (i.e., low-confidence > high-
relative to when recognition is accompanied by the sub- confidence recognition decisions). Moreover, activity
jective sense of familiarity (“know” and high-confidence in a fourth region—temporoparietal junction (TPJ)—
“familiar”/“old” responses), and (2) when participants demonstrated a pattern that qualitatively resembled
objectively recollect details associated with a test probe’s the inverse of that in medial IPS/SPL. In a third line
past encounter, including retrieval of a specific contex- of evidence, Uncapher and colleagues (Uncapher,
tual (source) detail, retrieval of an associate of a cue, DuBrow, Hutchinson, & Wagner, 2011) manipulated
and retrieval of more rather than fewer event details when attention and memory operations were likely to
(Cabeza et al., 2008; Rugg, Johnson, & Uncapher, in occur during a retrieval task, and demonstrated paral-
press; Vilberg & Rugg, 2008; Wagner et al., 2005). lel dissociations as reported in Hutchinson et al.
Second, consistent with bottom-up attention being (2014): AnG activity tracked recollection success, SPL
engaged when expectations are violated (e.g., oddball activity tracked top-down attentional orienting, and
effects and invalid vs. valid trials during attention TPJ activity exhibited bottom-up attentional reorient-
cueing; Corbetta, Patel, & Shulman, 2008), memory-based ing effects.
570 memory
Figure 48.3 Working model of PPC operations during episodic retrieval (see text for details).
the same holds for the distinct functional profiles in TPJ functionally couples with these MTL structures at rest
and AnG. We next discuss other hypotheses advanced and during tasks.
to understand retrieval effects in AnG and lateral IPS. The episodic buffer hypothesis proposes that AnG
acts as an interface between episodic memory opera-
Buffer/binding accounts of angular tions and executive processes engaged in service of
gyrus function memory-guided decisions. By this account, AnG mecha-
nisms serve to maintain recollected memory content, so
As reviewed above, extensive evidence indicates that that the retrieved mnemonic information can be inter-
AnG activity tracks recollection at retrieval. Two closely rogated in service of retrieval goals (Vilberg & Rugg,
related hypotheses—the episodic buffer (Vilberg & Rugg, 2008). Consistent with this account, AnG demonstrates
2008) and cortical binding of relational activity (CoBRA; a “recollection load effect” (Hutchinson et al., 2014;
Shimamura, 2011) accounts—posit that AnG operates Rugg et al., in press; Vilberg & Rugg, 2008)—that is,
on the products of hippocampally mediated pattern activity is greater when more versus less information (or
completion (see chapters 45 and 46 in this volume). more specific vs. more general information) is recol-
AnG is thought to be well positioned to represent and lected, and thus presumably when more information
operate over the multifeatural information reinstated must be maintained. Moreover, Vilberg and Rugg
by the MTL, given that AnG (1) receives strong disyn- (2012) demonstrated that, whereas hippocampus is
aptic input from the hippocampus by way of parahip- only transiently engaged during recollection, AnG activ-
pocampal gyrus and retrosplenial cortex, and (2) ity persists for variable durations depending on how
572 memory
is accumulated (rather than correct rejections being action to be performed (e.g., manual vs. oculomotor
based on the absence of strong evidence that the probe responses). To date, no study has directly tested either
is old), then a U-shaped activity function might be pre- prediction. Here, we review evidence from the action-
dicted (rather than a monotonic item-memory strength intention literature that suggests a pattern of effector-
function). Such a pattern is sometimes observed in specificity compatible with this hypothesis.
AnG, rather than lateral IPS, during recognition deci- In nonhuman primates, effector-specificity is observed
sions (figure 48.2B). in PPC during preparation of eye, arm, and hand move-
Others have posited that AnG implements an accu- ments (Andersen & Cui, 2009). Neurons in the “pari-
mulative integration-to-bound mechanism during etal reach region” of PPC exhibit preferential activity
retrieval (Criss, Wheeler, & McClelland, 2013). Criss during preparation of arm movements, anterior intra-
and colleagues observed that AnG BOLD activity peaked parietal cortical neurons show preference for grasping
more quickly when participants rejected foils in a block movements, and a “parietal eye field” in PPC shows
that contained strongly encoded old items relative to a strong preference for eye movements. Although mon-
block that contained weakly encoded old items, which key-human homologies are underspecified, it is thought
offers tentative support for the view that the level of the human homolog of the parietal eye field is medial
evidence (as indexed by perceived novelty) influences IPS, anterior intraparietal area is thought to be homolo-
the rise rate of AnG activity (N.B.: this result is also gous to anterior IPS, and the parietal reach region may
consistent with binding/buffer accounts). Other data dissociate into two regions in humans, medial IPS and
potentially challenge an accumulator account of AnG superior parietal-occipital sulcus (Vesia & Crawford,
function. Guerin and Miller (2011) had participants 2012).
encode faces that occurred with low (once or twice) or There is growing fMRI evidence for effector-specific-
high frequency (five or six times), and then make a ity in human PPC (Vesia & Crawford, 2012), with a
frequency discrimination between pairs of faces (“which saccade bias in more posterior/medial areas and a
face appeared more often?”). Test pairs consisted of hand/arm bias in more anterior/lateral areas. For
easy versus difficult discriminations (low-high pairs vs. instance, Beurze, de Lange, Toni, and Medendorp
low-low or high-high pairs). Importantly, AnG activity (2009) demonstrated that anterior IPS exhibits hand-
was greater when frequency discriminations involved specific activity, the superior parietal-occipital sulcus
high-frequency faces (i.e., high-high > low-high > low- exhibits eye-specific activity, and medial IPS is sensitive
low), rather than when discriminations were easy versus to both effectors. Interestingly, recent data suggest that
difficult, suggesting that AnG activity may scale with the this posterior-anterior gradient may not reflect effector-
amount of retrieved information independent of deci- specificity per se, but instead may relate to the different
sion certainty. This pattern appears to favor binding/ functional goals that different effectors enact (i.e.,
buffer accounts of AnG function during retrieval. saccade goals operate on eye-centered reference frames,
whereas limb goals interact with both eye- and body-
Action intention reference frames; Vesia & Crawford, 2012).
Given the central role of dPPC regions in action
PPC computations support the planning of movements, intention, a largely unexplored hypothesis is that PPC
as evidenced by extensive data on the role of PPC in activity during retrieval—particularly in IPS—reflects
eye, arm, and hand movements (Andersen & Cui, the transformation of mnemonic evidence to action
2009). The vast majority of retrieval studies required intention. As noted, during manually signaled memory
participants to indicate their memory decisions by decisions, medial IPS/SPL demonstrates a decision
manual button press, raising the possibility that aspects uncertainty effect, whereas lateral IPS (often spanning
of PPC activity during retrieval reflect action-intention mid- to anterior IPS) monotonically tracks perceived
processes (which may interact with processes of atten- item-memory strength, raising an intriguing possibility
tion, binding, and evidence accumulation). From this that retrieval-related dissociations may partially reflect
perspective, some PPC old/new effects may reflect the role of action computations during recognition
the preparation of memory-guided actions, with the decisions. In the only study (to our knowledge) to inves-
strength of memory evidence serving to modulate the tigate action intention during retrieval (Shannon &
strength or duration of action-intention processes. This Buckner, 2004), participants made button presses either
hypothesis predicts that (1) existing PPC old/new to old items or to new items, withholding responses to
effects will localize to regions that exhibit preference the other class of items (manipulated between experi-
for hand movements, and (2) the localization of old/ ments). While PPC old/new effects did not vary accord-
new effects will shift depending on the memory-guided ing to response contingency, which seems to argue
574 memory
methods with greater temporal resolution. For example, Heekeren, H. R., Marrett, S., Bandettini, P. A., &
a better understanding of PPC mnemonic accumulative Ungerleider, L. G. (2004). A general mechanism for
perceptual decision-making in the human brain. Nature,
processes could be obtained with measurements of the
431(7010), 859–862.
slope of neural responses during retrieval. We are opti- Ho, T. C., Brown, S., & Serences, J. T. (2009). Domain
mistic that such investigations will continue to reveal general mechanisms of perceptual decision making in
the unexpected, multifaceted nature of PPC contribu- human cortex. J Neurosci, 29(27), 8675–8687.
tions to memory. Hutchinson, J. B., Uncapher, M. R., & Wagner, A. D.
(2009). Posterior parietal cortex and episodic retrieval:
Convergent and divergent effects of attention and memory.
REFERENCES
Learn Memory, 16(6), 343–356.
Andersen, R. A., & Cui, H. (2009). Intention, action plan- Hutchinson, J. B., Uncapher, M. R., Weiner, K. S., Bressler,
ning, and decision making in parietal-frontal circuits. D. W., Silver, M. A., Preston, A. R., & Wagner, A. D.
Neuron, 63(5), 568–583. (2014). Functional heterogeneity in posterior parietal
Berryhill, M. E. (2012). Insights from neuropsychology: Pin- cortex across attention and episodic memory retrieval. Cereb
pointing the role of the posterior parietal cortex in episodic Cortex, 24(1), 49–66.
and working memory. Front Integr Neurosci, 6, 31. Kayser, A. S., Buchsbaum, B. R., Erickson, D. T., &
Beurze, S. M., de Lange, F. P., Toni, I., & Medendorp, W. P. D’Esposito, M. (2010). The functional anatomy of a per-
(2009). Spatial and effector processing in the human pari- ceptual decision in the human brain. J Neurophysiol, 103(3),
etofrontal network for reaches and saccades. J Neurophysiol, 1179–1194.
101(6), 3053–3062. Nelson, S. M., Cohen, A. L., Power, J. D., Wig, G. S.,
Cabeza, R., Ciaramelli, E., & Moscovitch, M. (2012). Miezin, F. M., Wheeler, M. E., … Petersen, S. E. (2010).
Cognitive contributions of the ventral parietal cortex: An A parcellation scheme for human left lateral parietal cortex.
integrative theoretical account. Trends Cogn Sci, 16(6), Neuron, 67(1), 156–170.
338–352. Nelson, S. M., McDermott, K. B., & Petersen, S. E. (2012).
Cabeza, R., Ciaramelli, E., Olson, I. R., & Moscovitch, M. In favor of a fractionation view of ventral parietal cortex:
(2008). The parietal cortex and episodic memory: An atten- Comment on Cabeza et al. Trends Cogn Sci, 16(8),
tional account. Nat Rev Neurosci, 9(8), 613–625. 399–400.
Cabeza, R., Mazuz, Y. S., Stokes, J., Kragel, J. E., Woldorff, Nelson, S. M., McDermott, K. B., Wig, G. S., Schlaggar,
M. G., Ciaramelli, E., … Moscovitch, M. (2011). Over- B. L., & Petersen, S. E. (2013). The critical roles of localiza-
lapping parietal activity in memory and perception: Evi- tion and physiology for understanding parietal contribu-
dence for the attention to memory model. J Cogn Neurosci, tions to memory retrieval. Neuroscientist, 19(6), 578–591
23(11), 3209–3217. O’Connor, A. R., Han, S., & Dobbins, I. G. (2010). The
Caspers, S., Eickhoff, S. B., Rick, T., von Kapri, A., inferior parietal lobule and recognition memory: Expec-
Kuhlen, T., Huang, R., … Zilles, K. (2011). Probabilistic tancy violation or successful retrieval? J Neurosci, 30(8),
fibre tract analysis of cytoarchitectonically defined human 2924–2934.
inferior parietal lobule areas reveals similarities to macaques. Ploran, E. J., Nelson, S. M., Velanova, K., Donaldson,
NeuroImage, 58(2), 362–380. D. I., Petersen, S. E., & Wheeler, M. E. (2007). Evidence
Caspers, S., Geyer, S., Schleicher, A., Mohlberg, H., accumulation and the moment of recognition: Dissociating
Amunts, K., & Zilles, K. (2006). The human inferior pari- perceptual recognition processes using fMRI. J Neurosci,
etal cortex: Cytoarchitectonic parcellation and interindi- 27(44), 11912–11924.
vidual variability. NeuroImage, 33(2), 430–448. Ratcliff, R. (1988). Continuous versus discrete information
Ciaramelli, E., Grady, C., Levine, B., Ween, J., & processing modeling accumulation of partial information.
Moscovitch, M. (2010). Top-down and bottom-up atten- Psychol Rev, 95(2), 238–255.
tion to memory are dissociated in posterior parietal cortex: Rugg, M. D., Johnson, J. D., & Uncapher, M. R. (in press).
Neuroimaging and neuropsychological evidence. J Neurosci, Encoding and retrieval in episodic memory: Insights
30(14), 4943–4956. from fMRI. In A. Duarte, M. Berense, & D. R. Addis
Ciaramelli, E., Grady, C. L., & Moscovitch, M. (2008). (Eds.), Cognitive neuroscience of memory. Hoboken, NJ: Wiley-
Top-down and bottom-up attention to memory: A hypoth- Blackwell.
esis (AtoM) on the role of the posterior parietal cortex in Scheperjans, F., Eickhoff, S. B., Hömke, L., Mohlberg, H.,
memory retrieval. Neuropsychologia, 46(7), 1828–1851. Hermann, K., Amunts, K., & Zilles, K. (2008). Probabilis-
Corbetta, M., Patel, G., & Shulman, G. L. (2008). The tic maps, morphometry, and variability of cytoarchitectonic
reorienting system of the human brain: From environment areas in the human superior parietal cortex. Cereb Cortex,
to theory of mind. Neuron, 58(3), 306–324. 18(9), 2141–2157.
Criss, A. H., Wheeler, M. E., & McClelland, J. L. (2013). Sestieri, C., Shulman, G. L., & Corbetta, M. (2010). Atten-
A differentiation account of recognition memory: Evidence tion to memory and the environment: Functional special-
from fMRI. J Cogn Neurosci, 25(3), 421–435. ization and dynamic competition in human posterior
Felleman, D. J., & Van Essen, D. C. (1991). Distributed hier- parietal cortex. J Neurosci, 30(25), 8445–8456.
archical processing in the primate cerebral cortex. Cereb Shadlen, M. N., & Newsome, W. T. (2001). Neural basis of a
Cortex, 1(1), 1–47. perceptual decision in the parietal cortex (area LIP) of the
Guerin, S. A., & Miller, M. B. (2011). Parietal cortex tracks rhesus monkey. J Neurophysiol, 86(4), 1916–1936.
the amount of information retrieved even when it is not the Shannon, B. J., & Buckner, R. L. (2004). Functional-
basis of a memory decision. NeuroImage, 55(2), 801–807. anatomic correlates of memory retrieval that suggest
576 memory
49 Memory as Decision Making
MICHAEL B. MILLER AND IAN G. DOBBINS
ABSTRACT Memory researchers investigating the brain regions depend on episodic memory evidence; for example, did
involved in successfully retrieving a past event have revealed I park the car in this lot? have I read that book before?
a robust and ubiquitous pattern of activity across lateral did my wife tell me to pick up the kids? Strategic deci-
regions of the left prefrontal and parietal cortex. While some
of these regions may play a direct causal role in episodic sion processes are involved in each of these mnemoni-
memory retrieval, such as the accumulation of mnemonic cally guided decisions, although sometimes in subtle
evidence or the buffering of retrieved representations, we ways. For example, judging whether one read a particu-
present evidence that much of the activity can be accounted lar book (a recognition task) may be influenced by
for by decision and control processes and not memory factors other than the memory “signal” itself, such as an
retrieval per se. Two general lines of research demonstrate
that brain activity greater for hits than correct rejections can
assessment of one’s general familiarity with the author,
reflect decision biases and/or reorientation. We present a knowledge of the release date of the book, or other
biasing and orienting model of parietal contribution that con- factors, which may bias one toward a positive or nega-
verges with studies of patients with parietal lobe damage and tive final conclusion (Johnson, Hashtroudi, & Lindsay,
syntheses with research on visual attention and valuation. 1993; Mandler, 1980; Schacter, 1996; Tulving, 1983).
Recognition tests have been used to probe the contents
Episodic or event memory pervades our everyday lives, of memory as early as 1913, when Hollingworth stated
but its study is challenging because inferences about in his report in The American Journal of Psychology that
memory functioning are fairly indirect. These infer- “the value of a single presentation is greater in recogni-
ences rely on establishing the links between the original tion than in recall, and the difference between the
events that yield the potential for subsequent memory values of repetitions becomes still greater the more
expression and a retrieval demand that may take place meaning the material possesses” (Hollingworth, 1913,
minutes, days, or years following the events. The p. 543). As George Mandler put it, to “recognize is the
researcher cannot directly manipulate raw memory act of perceiving something as previously known … the
signals in the same manner he or she might manipulate recognition of the prior occurrence of an event” (1980,
frequency in, say, an audition experiment. Instead, the p. 252). He postulated that recognition consists of two
properties of memory representations must be inferred separate and additive processes: (1) the recognition of
through far more indirect methods. Hence, the advent familiarity which is a continuous value retrieved quickly
of functional brain imaging using positron emission and automatically based largely on the perceptual char-
tomography (PET) and then functional magnetic reso- acteristics of the previous exposure, and (2) the identi-
nance imaging (fMRI) provided the opportunity to fication as a result of a retrieval process that is thought to
more directly observe the process of episodic memory be initiated once the familiarity judgment fails to
retrieval (a.k.a. ecphory) that guides our behavior in a provide an unequivocal decision. These two processes
wide variety of contexts. The overarching message of are now commonly known as familiarity and recollec-
this chapter is that while these techniques have greatly tion (Jacoby, 1991; Yonelinas, 1997), although current
informed our understanding of episodic recognition models generally assume that recollection is sought
memory, in doing so, they suggest that much of the during every recognition trial in standard recognition
observed brain activation reflects various decision and tests, and not merely those trials that yield ambiguous
control processes and not memory-retrieval processes sensations of familiarity, and it is thought that familiar-
per se. Thus the “simple” act of recognition appears to ity is often based on conceptual characteristics of the
tap a host of complex control processes geared toward previous exposure.
tuning and biasing memory judgments. Mandler’s well-known analogy of these two separable
In everyday life, episodic memories are retrieved in forms of memory content invites you to imagine walking
various ways, including spontaneous encounters with onto a bus and seeing a man whose unexpected famil-
cues that trigger the retrieval of a memory, as well as iarity convinces you that you have seen him before, but
through purposeful searches of past events. We also you cannot immediately recall where or when. Follow-
make decisions throughout the course of the day that ing a deliberate search of memory where various
578 memory
Figure 49.1 Activation likelihood map of “retrieval success” effect during recognition. (Taken from Spaniol et al. 2009.)
important for source-memory attribution and other judgments is then compared to activity associated with
complex recall tasks, patients with fairly extensive PFC item memory judgments alone. Even when stimulus
damage were thought to be quite normal on simple materials are completely matched across source and
verbal recognition tasks (Janowsky, Shimamura, & item memory conditions, source judgments are associ-
Squire, 1989; Milner, Corsi, & Leonard, 1991; Shimam- ated with greater activity in left ventrolateral PFC
ura, Janowsky, & Squire, 1990; though see Wheeler, (Dobbins, Foley, Schacter, & Wagner, 2002). However,
Stuss, & Tulving, 1997). These neuropsychological the level of activation is quite similar regardless of
studies suggested that memory tasks that made demands whether or not the source attribution is correct
on executive functioning or inhibitory control, such as (Dobbins, Rice, Wagner, & Schacter, 2003). In addition,
free recall, source memory, and temporal ordering, the magnitude of activity in this region during the
were impaired by PFC damage because these secondary source task appears to be linked to the degree that
processes were damaged; however, basic recognition the probe’s semantic as opposed to perceptual charac-
ability remained intact (Shimamura, 1995). The pre- teristics are potentially relevant for the source judg-
frontal cortex has reciprocal connections throughout ment (Dobbins & Wagner, 2005). Consistent with this
the cortex, and it appears to filter and control much of finding, damage to this region (Thompson-Schill et al.,
the flow of information between sensory inputs and 1998) and disruption via transcranial magnetic stimula-
motor outputs (Miller & Cohen, 2001). However, these tion (Gough, Nobre, & Devlin, 2005) have impaired
connections vary greatly across PFC regions, indicating semantic processing. Collectively, these findings suggest
considerable functional heterogeneity in this large that the left ventrolateral PFC region supports the
region. controlled or strategic semantic processing of probes
during source-memory attempts and presumably during
Functional specialization in prefrontal cortex memory attributions in general. This process, known as
during retrieval attempts semantic elaboration, can improve retrieval outcomes
to the extent that the semantic features evident during
Although considerable uncertainty remains, a better the initial encounter match the features attended to at
understanding of the functional contribution of some test (Roediger & Geraci, 1990; Tulving & Thomson,
of the PFC regions to memory is beginning to develop. 1973).
Here we briefly consider two regions and outline
putative functions supported by functional imaging Left Dorsolateral Prefrontal Cortex (∼BA 6/8)
findings. This region is often activated in close conjunction
with dorsolateral parietal cortex, with which it is
Left Ventrolateral Prefrontal Cortex (∼BA 47) directly anatomically connected. In controlled judg-
This region, located along the inferior frontal gyrus, is ment domains outside of memory, these two regions are
not only revealed by the retrieval success contrast, but often described as members of a frontoparietal control
it is also implicated in studies examining source memory network (Dosenbach et al., 2007). The left DLPFC also
and semantic memory. In source-memory studies, sub- demonstrates greater activation during source- versus
jects encode information that originated from a par- item-memory decisions for matched verbal probes;
ticular context within the study session. Then, at test, however, unlike the left ventrolateral region, it appears
on some trials subjects are asked to determine the par- to be engaged even prior to the arrival of the probes
ticular context through which the item was encoded, and thus demonstrates a greater activation when the
whereas on other trials, they are asked to make an item retrieval question dictates a source requirement instead
memory judgment and are not required to recall the of an item requirement (Dobbins & Han, 2006). The
context. Brain activity associated with source-memory region then increases activation during both types of
580 memory
well (e.g., Van Zandt & Maldonado-Molina, 2004). A biasing and orienting model of
Support for the accumulation hypothesis of parietal parietal contributions
activation has been gained by demonstrating that activa-
tion tracks the number of original contextual details We propose a model that arises out of two general lines
that are recovered by the participant (Vilberg & Rugg, of research: one in which observers are required to shift
2009a). However, a recent fMRI study on frequency memory decision biases adaptively during blocks of
judgments found that parietal activity tracked with the trials, and one in which they are required to incorpo-
absolute amount of information even when that infor- rate biases into their recognition judgments on a trial-
mation was not the basis of the decision (Guerin & by-trial basis. Both lines of research suggest that parietal
Miller, 2011). Finally, a related hypothesis suggests that regions are important for the implementation and
parietal responses reflect the operation of an episodic- adjustment of decision biases during recognition.
memory output buffer, in which the recovered contents Before briefly describing these paradigms and their
of episodic memory are temporarily stored in a buffer findings, it is important to note why bias during deci-
that makes them rapidly accessible to decision making. sion making is critical, particularly in the case of memory
Such a buffer had been proposed by Baddeley (2000) decisions.
as the missing component to his working memory The phrase “successful retrieval effect” implies that
model, and it would work similarly to visual and verbal the effect directly supports retrieval processes; however,
working memory buffers. a more appropriate phrase might be the “recognition
judgment effect,” which does not presuppose a direct
link between the activations and the availability of
Criteria for fashioning a parietal lobe memory content. Critically, optimal decision making in
functional model recognition (and all discrimination tasks) depends not
only upon the current stimulus evidence, but also
Despite a host of candidates, there remains little con- upon the context in which that evidence is encoun-
sensus on the functional roles of the parietal lobe tered. This is formalized under Bayesian reasoning
during recognition memory. Building on the discussion through the incorporation of prior probabilities with
of Wagner, Shannon, Kahn, and Buckner (2005), we current observed evidence in order to arrive at a poste-
propose a series of criteria that a successful model must rior probability. Equivalently, under the signal-detec-
achieve. tion framework, the observer uses a decision bias that
1. The characterization must be compatible with can be adjusted in order to maximize outcomes. Thus
extant neuropsychological findings and be able to the use of informative priors in Bayesian reasoning
account for the historical absence of any link between and flexible biases (a.k.a. criteria) in signal-detection
parietal damage and recognition-memory impairment. theory serve an identical function, namely, to modu-
2. The characterization must clearly distinguish late decisions using information other than that which
between causal and noncausal functional models with is directly perceived or remembered. For example,
respect to memory-retrieval ability. A causal interpreta- even if recognition evidence were quite strong, one
tion reflects a characterization that would be essential would nonetheless want to be highly cautious in
for successful retrieval behavior to occur. For example, judging an individual as recognized in a context in
the episodic buffer account is causal because damage which there was little prior probability that familiar
or removal of the buffer would yield behavioral amnesia, individuals would be encountered (e.g., a foreign
since the contents of episodic memory would be unavail- airport terminal). In such situations, under the signal-
able. In contrast, a model that assumed the response detection framework, ideal observers would use what is
reflected the implementation of a decision bias would termed a “strict” or “conservative” decision bias, requir-
be noncausal, in that damage would yield an inability ing extremely high levels of evidence to judge an indi-
to flexibly bias memory decisions but would not prevent vidual as recognized. In contrast, in more familiar
basic memory functioning. environments they would use a more “lax” or “liberal”
3. The characterization should incorporate both con- bias. The adopted bias can also be informed by the
dition-level blood oxygen level–dependent, or BOLD, relative benefits and costs of correct and incorrect
effects (e.g., hits greater than correct rejection) and it judgments as well as the prior probabilities (Gold &
should anticipate individual differences. For example, Shadlen, 2001; Green & Swets, 1966; Macmillan &
the episodic buffer model would predict that individual Creelman, 2005). Thus, unlike the pejorative lay
differences in accuracy should track with increases in use of the term “decision bias,” ideal responding
blood oxygen level–dependent activity. actually requires biasing one’s judgments; the same
recognition evidence should not lead to the same with manipulations of sensitivity. A group-level analysis
response on all occasions. revealed that more activation occurred in dorsolateral
Recent behavioral work from our labs demonstrates prefrontal and dorsolateral parietal cortex during
that there are considerable individual differences in blocks in which the decision criterion shifted on a trial-
observers’ ability to appropriately bias their recognition by-trial basis compared to blocks in which the decision
judgments using blockwise manipulations of target criterion remained stable. However, a similar analysis
probabilities, blockwise manipulations of payoffs, or with blocks of high d ′ (sensitivity) compared to blocks
trialwise manipulations of target probability (Aminoff of low d ′ revealed only medial activations, with no acti-
et al., 2012; Selmeczy & Dobbins, 2012). An example vations in parietal cortex whatsoever. A more recent
of this variability and its consequences is shown in event-related fMRI study of 95 subjects found that
table 49.1. similar regions were activated for trials in which the
criterion shifted compared to trials in which the crite-
The Link Between Parietal Cortex and the Biasing rion remained the same (Aminoff et al., submitted; see
of Decisions In the visual attention literature, the figure 49.2). Further, Aminoff and colleagues found
link between biasing judgments and parietal cortex is that the regions sensitive to criterion shifting signifi-
well established. When observers use predictive cues cantly overlapped with regions that exhibited greater
that reliably anticipate the spatial location of subse- activity for hits compared to correct rejections. Even the
quent perceptual probes, activation is increased in the regions of successful retrieval activity that did not
PPC, specifically in the intraparietal sulcus (IPS), com- overlap with regions of criterion-shift activity using the
pared to situations in which environmental transients strict thresholds were significantly correlated with indi-
occur at the same location as an upcoming target (e.g., vidual differences in the conservativeness of the crite-
Kincade, Abrams, Astafiev, Shulman, & Corbetta, 2005). rion (see below).
The former is typically referred to as an endogenous In a recent fMRI study, Vilberg and Rugg (2009b)
shift of spatial attention; however, it is equally appropri- manipulated the test-wide prevalence of targets (25%
ate to refer to it as a spatial judgment bias. In contrast, or 75%) during a combined source- and item-recogni-
when probes are encountered at unexpected locations, tion judgment task. At test, subjects indicated that they
compared to validly endogenously cued locations, there recognized the items and remembered one of two
is increased activation in ventral parietal regions sur- source contexts, merely recognized the items but
rounding the temporoparietal junction. As noted by couldn’t recollect the source context, or believed the
Hutchinson, Uncapher, and Wagner (2009) and Nelson, items to be new. Critically, they found that regions in
McDermott, and Petersen (2012), these regions do not the lateral post-central gyrus and dorsal superior pari-
coincide with those implicated during recognition etal lobule that demonstrated “retrieval success” effects
memory. However, as suggested by Cabeza et al. (2012), were modulated by the target base rates. However, they
they may serve similar roles at the algorithmic level. also found that more lateral and inferior parietal
An early fMRI study suggested a role for recognition regions, along with the middle portion of the IPS, dem-
decision bias in the parietal cortex (Miller, Handy, onstrated a “retrieval success” effect that was insensitive
Cutler, Inati, & Wolford, 2001). Using a block design, to the listwide target probabilities, with the authors sug-
Miller and colleagues crossed manipulations of criteria gesting this pattern was directly reflective of retrieval.
582 memory
Figure 49.2 A comparison of activation maps (n = 95 sub- conservative and liberal conditions (see Aminoff et al.,
jects; p > .05, false-discovery rate corrected) for the shifting submitted).
criterion contrast and the “retrieval success” contrast in the
But this interpretation was hampered by the fact that in both the conservative and liberal conditions was sig-
the subjects were explicitly instructed to ignore any nificantly related to individual differences in criterion
changes in the base rate of the targets, and the changes but not to individual differences in accuracy. Overall,
in the target probabilities were not cued in any way. these data suggest that the parietal retrieval response
Several behavioral studies have demonstrated that cri- must take into account the adopted decision biases.
terion shifts only occur if the subject is aware of the In order to examine the interaction between expecta-
change in the conditions (Estes & Maddox, 1995; tions and memory evidence, O’Connor, Han, and
Rhodes & Jacoby, 2007; Wixted & Stretch, 2000). Dobbins (2010) used a trial-wise cuing paradigm in
Indeed, the listwide density manipulation in the Vilberg which each recognition memory probe was preceded
and Rugg (2009b) study failed to induce any behavioral by a verbal cue (Likely Old or Likely New) that indi-
decision biases. As we discuss below, in paradigms yield- cated its probable memory status. During critical runs,
ing clear behavioral evidence of biases, whether due to these cues were valid on 80% of trials. Consistent with
the shifting or the conservativeness of the bias, much Aminoff et al. (submitted), these cues yielded promi-
of the lateral parietal response is affected. nent decision biases. With respect to brain activity, the
For example, Aminoff et al. (submitted) used a critical comparison of invalidly versus validly cued trials
blockwise biasing paradigm in which target probability (figure 49.3) resulted in prominent prefrontal and pari-
(70% or 30%) was indicated by a color cue for small etal activations. The results also demonstrated that
blocks of six to nine trials for both face and word invalid cuing activations occurred in supramarginal and
stimuli that were tested in separate runs. Participants angular gyrus regions for both old and new materials
were informed of this contingency and displayed prom- that were correctly identified. This invalid cued response
inent behavioral biases induced by these cues. In terms partially overlaps with the response demonstrated when
of the behavioral results, Aminoff and colleagues observers must shift the criterion applied to memory
(2012) recently reported clear individual differences in judgments (figure 49.2); however, it appears to extend
the degree that biases were induced across participants. more laterally. Because correctly identified new materi-
These were stable domain-general differences that als are unlikely to have any episodic content, these
were observed across materials. As for the brain activa- researchers concluded that the functional role of these
tion, there were three critical effects that emerged regions cannot reflect episodic retrieval or the accumu-
from the study (Aminoff et al., submitted). First, as lation or buffering of episodic content. Furthermore,
noted above, when observers were required to shift the in an independent fMRI data set of basic uncued rec-
bias, there was a transient response at the beginning of ognition, activation in these parietal regions was shown
the block in the SPL that extended along the IPS, to correlate with individual differences in adopted deci-
regardless of the direction of shift or the accuracy of sion bias, and not retrieval accuracy. Specifically, this
the response. Second, although an apparent “retrieval analysis demonstrated that for individuals who were
success” effect was observed in dorsolateral parietal increasingly conservative, there was an increase in the
cortex during blocks inducing a conservative bias (i.e., hits>correct rejections signal difference. This corre-
strong expectation of new materials), the effect was sponds to an increase during identification of the
virtually eliminated under blocks that induced a liberal memory class least expected by the participant, and the
bias (i.e., strong expectation of old materials; see figure lateral parietal region was held to signal disconfirma-
49.2). Finally, activity associated with “retrieval success” tions of decision biases that were either cue-induced or
general characteristics of the participants (see also biases. From this overall perspective, the saliency of
Herron & Rugg, 2003). memory signals is governed by the bias induced by the
A final recent fMRI study looking at trialwise decision cues. This notion is further supported by the results of
biases afforded sufficient power to separately examine an individual differences analysis of the anterior angular
the two types of violations possible, namely, unexpected gyrus response. Within this region, the unexpected
familiarity following the Likely New cue and unex- familiarity response (hits vs. correct rejections following
pected novelty following the Likely Old cue (Jaeger, the Likely New cue) was strongly associated with indi-
Konkel, & Dobbins, 2013). Figure 49.4 shows three vidual differences in accuracy. This might have been
regions demonstrating three different patterns, includ- mistakenly construed as evidence that the region sup-
ing sensitivity to unexpected familiarity, sensitivity to ports retrieval in a causal manner. However, the activa-
unexpected novelty, and sensitivity to both unexpected tions under the Likely Old and uncued conditions bore
familiarity and unexpected novelty. Postcentral gyrus no relationship to individual differences in accuracy
and medial SPL areas demonstrated greater activation within those conditions. This is precisely the marker of
for new than old materials following the Likely Old cue, saliency anticipated under an orienting model, in that
consistent with orienting toward unexpected novelty. those individuals who discriminate well showed the
This interpretation is further supported by the failure most marked differential response to unexpectedly
to detect any discernible signal difference in these familiar stimuli that conflicted with the Likely New bias
regions when observers instead expected new materials versus expected novel stimuli that confirmed the Likely
following the Likely New cue. The opposite pattern of New bias.
results occurred in anterior angular gyrus. Here differ- Critically, the findings in Jaeger et al. (2013) point to
ential activation occurred following the Likely New considerable functional heterogeneity in parietal cortex
cues, in which there was a stronger response for old during recognition judgment. Neither the unexpected
versus new materials, which is consistent with orienting familiarity nor unexpected novelty responses have been
toward unexpected familiarity. There was no signal dif- previously isolated, but their close proximity to the
ference between the materials when the observers more general mid-IPS response may mean that these
instead expected familiarity following the Likely Old various responses may have been collapsed in prior
cues. Finally, the intraparietal sulcus demonstrated designs and discussions. Furthermore, the three pat-
increased activation for whichever class of memoranda terns of response in figure 49.4 corresponded well with
was unexpected. This pattern of activation led to the recent parcellations of parietal cortex suggested by the
conclusion that this region supported the orienting of analysis of resting connectivity data via graph theory
attention toward unexpected memory content, and (Nelson et al., 2010). Moving forward, then, it may be
hence may be critical for overriding adopted decision important to distinguish between general control
584 memory
Figure 49.4 Selective responses to particular types of viola- in blue demonstrated an unexpected familiarity response,
tions of memory expectations. Targeted contrasts and masking with bar plot (B), illustrating the pattern of response across
procedures isolated three different patterns of response in cue conditions and item types. The mid-IPS region in red
left lateral parietal cortex. The left anterior IPS/PoCG region demonstrated a general unexpected memory effect, with bar
in green illustrated an unexpected novelty response pattern, plot (C) illustrating the pattern of response across cue condi-
with bar plot (A) illustrating the pattern across cue conditions tions and item types. (Taken from Jaeger et al., 2013.) (See
and item types. The posterior anterior angular gyrus region color plate 40.)
mechanisms that underlie criterion shifts and violations In summary, recent fMRI findings demonstrate that
of adopted criteria (that presumably engender shifts) adopting a liberal decision bias eliminates previously
from material specific effects, such as unexpected observed successful retrieval activation differences in
novelty, and unexpected familiarity responses that may intraparietal sulcus and supramarginal gyrus regions.
be linked to mobilization of different investigatory Furthermore, the requirement to shift a recognition
responses depending on the nature of the violation. For decision bias yields prominent activations in these
example, figure 49.2 illustrates that the most prominent regions as well. There also appears to be functional
response to facing a requirement to shift the recogni- specificity in the parietal cortex when adopted biases
tion criterion occurs along the IPS (figure 49.2, left are violated versus confirmed by memory materials,
panel), regardless of the direction of shift. Somewhat both in the case of novelty (in regions not typically
analogously, the mid-IPS response in figure 49.4 associated with “retrieval success” effects) and familiar-
occurred whenever a memorandum violated the expec- ity (in regions typically associated with “retrieval success”
tation held under the current bias/criterion. Addition- effects). Finally, individual difference analyses using
ally, both contrasts also implicated similar lateral PFC ROIs in or near the anterior angular gyrus do not
regions, and so both may capture a process or processes support a direct role in retrieval. Instead they suggest
that are important when an observer must update or that activation for old materials is governed by the
reevaluate the adequacy of the current decision crite- salience of familiarity or perhaps recollection signals,
rion, regardless of whether it is liberal or conservative. which is a function of both the cue condition and the
586 memory
of episodic content, and (4) the motivational signifi- and constructed experience tasks. Neuropsychologia, 48,
cance of particular features, which is known to drive 1385–1393.
Buckner, R. L., Koutstaal, W., Schacter, D. L., Dale,
parietal activation during saccade decisions in nonhu-
A. M., Rotte, M. R., & Rosen, B. R. (1998). Functional-
man primates (Leathers & Olson, 2012; e.g., cells anatomic study of episodic retrieval. II. Selective averaging
driven by the motivational salience of the cue but not of event-related fMRI trials to test the retrieval success
the action value). Thinking about parietal activation at hypothesis. NeuroImage, 7, 163–175.
retrieval as a function of the relative salience of differ- Buckner, R. L., Koutstaal, W., Schacter, D. L., Wagner,
ent types of memory information is a fairly unexplored A. D., & Rosen, B. R. (1998). Functional-anatomic study of
episodic retrieval using fMRI. I. Retrieval effort versus
framework. However, it is clear that externally cued retrieval success. NeuroImage, 7, 151–162.
decision biases, which govern expectations about the Cabeza, R., Ciaramelli, E., & Moscovitch, M. (2012). Cog-
types of memory signals that should be encountered nitive contributions of the ventral parietal cortex: An
and also heavily influence behavioral response pat- integrative theoretical account. Trends Cogn Sci, 16(6),
terns, concomitantly have very large effects on the dis- 338–352.
Cabeza, R., Ciaramelli, E., Olson, I. R., & Moscovitch, M.
tribution of and nature of parietal responses during (2008). The parietal cortex and episodic memory: An atten-
recognition. Additionally, the salience of expectations tional account. Nat Rev Neurosci, 9, 613–625.
will be modulated by the discrimination ability of the Chafee, M. V., & Goldman-Rakic, P. S. (2000). Inactivation
subjects. In other words, those who discriminate well of parietal and prefrontal cortex reveals interdependence
should also show greater response to violations because of neural activity during memory-guided saccades. J Neuro-
physiol, 83(3), 1550–1566.
the conflicting signals will be strong (Jaeger, Konkel, &
Ciaramelli, E., Grady, C., Levine, B., Ween, J., &
Dobbins, 2013). Nonetheless, a biasing and orienting Moscovitch, M. (2010). Top-down and bottom-up atten-
framework assumes the functional significance of these tion to memory are dissociated in posterior parietal cortex:
activations lies not in directly supporting retrieval Neuroimaging and neuropsychological evidence. J Neurosci,
success, but instead in biasing judgments for more effi- 30(14), 4943–4956.
Corbetta, M., & Shulman, G. L. (2002). Control of goal-
cient processing when those biases are confirmed and,
directed and stimulus-driven attention in the brain. Nat Rev
alternatively, orienting and exploring memory contents Neurosci, 3, 201–215.
more thoroughly when those biases are disconfirmed. Dobbins, I. G., Foley, H., Schacter, D. L., & Wagner, A. D.
(2002). Executive control during episodic retrieval: Multi-
ACKNOWLEDGMENTS We wish to thank Danielle King for helpful ple prefrontal processes subserve source memory. Neuron,
suggestions. The research cited in this chapter was supported 35(5), 989–996.
by the Institute for Collaborative Biotechnologies through Dobbins, I. G., & Han, S. (2006). Cue- versus probe-
contract no. W911NF-09-D-0001 from the U.S. Army Research dependent prefrontal cortex activity during contextual
Office to MBM and by the National Institute of Mental Health remembering. J Cogn Neurosci, 18(9), 1439–1452.
R01-MH073982 to IGD. Dobbins, I. G., Jaeger, A., Studer, B., & Simons, J. S. (2012).
Use of explicit memory cues following parietal lobe lesions.
Neuropsychologia, 50, 2992–3003.
Dobbins, I. G., Rice, H. J., Wagner, A. D., & Schacter,
D. L. (2003). Memory orientation and success: Separable
REFERENCES
neurocognitive components underlying episodic recogni-
Ally, B. A., Simons, J. S., McKeever, J. D., Peers, P. V., & tion. Neuropsychologia, 41, 318–333.
Budson, A. E. (2008). Parietal contributions to recollec- Dobbins, I. G., & Wagner, A. D. (2005). Domain-general and
tion: Electrophysiological evidence from aging and patients domain-sensitive prefrontal mechanisms for recollecting
with parietal lesions. Neuropsychologia, 46(7), 1800–1812. events and detecting novelty. Cereb Cortex, 15(11), 1768–1778.
Aminoff, E. M., Clewett, D., Freeman, S., Frithsen, A., Dosenbach, N. U. F., Fair, D. A., Miezin, F. M., Cohen,
Tipper, C., Johnson, A., … Miller, M. B. (2012). Indi- A. L., Wenger, K. K., Dosenbach, R. A. … Petersen, S. E.
vidual differences in shifting decision criterion: A recogni- (2007). Distinct brain networks for adaptive and stable
tion memory study. Mem Cognit, 40, 1016–1030. task control in humans. Proc Natl Acad Sci USA, 104(26),
Aminoff, E. M., Freeman, S., Clewett, D., Tipper, C., 11073–11078.
Frithsen, A., Johnson, A., … Miller, M. B. (submitted). Estes, W. K., & Maddox, W. T. (1995). Interactions of stimu-
Adapting to a cautious state of mind during a recognition lus attributes, base-rate and feedback in recognition. J Exp
test: A large-scale fMRI study. Psychol Learn Mem Cogn, 21, 1075–1095.
Baddeley, A. (2000). The episodic buffer: A new component Feredoes, E., Heinen, K., Weiskopf, N., Ruff, C., &
of working memory? Trends Cogn Sci, 4(11), 417–423. Driver, J. (2011). Causal evidence for frontal involvement
Berryhill, M. E. (2012). Insights from neuropsychology: Pin- in memory target maintenance by posterior brain areas
pointing the role of the posterior parietal cortex in episodic during distracter interference of visual working memory.
and working memory. Front Integr Neurosci, 6, 31. Proc Natl Acad Sci USA, 108(42), 17510–17515.
Berryhill, M. E., Picasso, L., Arnolds, R. A., Drowos, Gold, J. I., & Shadlen, M. N. (2001). Neural computations
D. B., & Olson, I. R. (2010). Similarities and differences that underlie decisions about sensory stimuli. Trends Cogn
between parietal and frontal patients in autobiographical Sci, 5(1), 10–16.
588 memory
Shadlen, M. N., & Newsome, W. T. (2001). Neural basis of a of retrieval in episodic memory: Auditory sentence recogni-
perceptual decision in the parietal cortex (area LIP) of the tion. Proc Natl Acad Sci USA, 91, 2012–2015.
rhesus monkey. J Neurophysiol, 86, 1916–1936. Tulving, E., & Thomson, D. M. (1973). Encoding specificity
Shimamura, A. P. (1995). Memory and frontal lobe func- and retrieval processes in episodic memory. Psychol Rev,
tion. In M. S. Gazzaniga (Ed.), The cognitive neurosciences 80(5), 352–373.
(pp. 803–813). Cambridge, MA: MIT Press. Van Zandt, T., & Maldonado-Molina, M. M. (2004).
Shimamura, A. P., Janowsky, J. S., & Squire, L. R. (1990). Response reversals in recognition memory. Cognition, 30(6),
Memory for the temporal order of events in patients with 1147–1166.
frontal lobe lesions and amnesic patients. Neuropsychologia, Vilberg, K. L., & Rugg, M. D. (2009a). Functional signifi-
28, 803–813. cance of retrieval-related activity in lateral parietal cortex:
Simons, J. S., Peers, P. V., Mazuz, Y. S., Berryhill, M. E., & Evidence from fMRI and ERPs. Hum Brain Mapp, 30(5),
Olson, I. R. (2010). Dissociation between memory accu- 1490–1501.
racy and memory confidence following bilateral parietal Vilberg, K. L., & Rugg, M. D. (2009b). An investigation of
lesions. Cereb Cortex, 20, 479–485. the effects of relative probability of old and new test items
Spaniol, J., Davidson, P. S. R., Kim, A. S. N., Han, H., on the neural correlates of successful and unsuccessful
Moscovitch, M., & Grady, C. L. (2009). Event-related source memory. NeuroImage, 45, 562–571.
fMRI studies of episodic encoding and retrieval: Meta- Wagner, A. D., Shannon, B. J., Kahn, I., & Buckner, R. L.
analyses using activation likelihood estimation. Neuropsycho- (2005). Parietal lobe contributions to episodic memory
logia, 47, 1765–1779. retrieval. Trends Cogn Sci, 9, 445–453.
Thompson-Schill, S. L., Swick, D., Farah, M. J., D’Esposito, Wheeler, M. E., Stuss, D. T., & Tulving, E. (1997). Toward
M., Kan, I. P., & Knight, R. T. (1998). Verb generation in a theory of episodic memory: The frontal lobes and autono-
patients with focal frontal lesions: A neuropsychological test etic consciousness. Psychol Bull, 121(3), 331–354.
of neuroimaging findings. Proc Natl Acad Sci USA, 95, Wixted, J. T., & Stretch, V. (2000). The case against a cri-
15855–15860. terion-shift account of false memory. Psychol Rev, 107(2),
Tulving, E. (1983). Elements of episodic memory. Oxford, UK: 368–376.
Clarendon. Yonelinas, A. P. (1997). Recognition memory ROCs for item
Tulving, E., Kapur, S., Markowitsch, H. J., Craik, F. I. M., and associative information: The contribution of recollec-
Habib, R., & Houle, S. (1994). Neuroanatomical correlates tion and familiarity. Mem Cognit, 25(6), 747–763.
ABSTRACT A central challenge in memory research has been that these habit-like action mechanisms are insufficient
to characterize the cognitive and neural mechanisms by which to explain many sorts of reward-directed decision
habits are formed. Here we review recent advances in under- behavior, leading to a recent focus on dissociating them
standing how habits are learned and how they interact with
other forms of learning to guide decisions and actions. We from other, more deliberative or explicit influences on
address these questions by reviewing research on multiple decisions. Though not yet fully understood, these sorts
memory systems alongside a complementary body of work of behaviors—known as goal-directed or model-based
on decision making and learning from reward, highlighting actions—may serve as a bridge from declarative memory
converging evidence from animal physiology, computational to behavior.
models, human functional MRI, and patient research.
Together, this work has demonstrated that reward prediction
The convergence between memory and decision
models of midbrain dopamine neurons successfully account making thus serves to fill in gaps in either field. Here
for a wide—but also limited—range of motivated behaviors; we survey these developments and discuss how under-
specifically, those that underlie habitual, stimulus-response standing the convergence between these areas provides
learning. We further review emerging research that has begun a framework for understanding the mechanisms by
to answer new questions, such as how habit learning interacts
which habits are learned, how different forms of
with goal-directed behavior, how other forms of memory con-
tribute to decision making, and how behavior transitions from memory guide decisions, and the neural mechanisms
goal-directed to habitual control. supporting these behaviors. Viewing these systems
together, as jointly supporting decision behavior, has
How does memory support behavior? Research on the begun to lead to an enriched view of both, and particu-
neurobiology of learning and memory has led to the larly of their interactions.
proposal that different forms of memory are supported
by different brain structures. A focus of research within Multiple memory systems supporting learning
this framework is to determine which structures con- and memory
tribute to memory for facts and events, termed declara-
tive memory, and which to memory for habits or skills, A prominent idea in cognitive neuroscience is that
often termed procedural memory. there are different kinds of memory, each supported by
Procedural memory, by its very definition, is more different neural systems (Cohen & Eichenbaum, 1993;
closely linked to action than declarative memory. Yet in Gabrieli, 1998; Squire & Zola, 1996). Decades of
studies of memory, declarative memory has traditionally research into declarative memory originated from studies
been more fully investigated, whereas many questions with patients with memory deficits, such as patient H.M.
remain about the neural, psychological, and comp- As the case of H.M. demonstrated, damage to the
utational mechanisms supporting procedural memo- medial temporal lobe (MTL), including the hippocam-
ries. At the same time, declarative memories clearly pus and surrounding cortical regions, leads to severe,
inform decisions and other day-to-day behaviors, but and selective, declarative memory impairments—an inabil-
the links to behavior from the more abstract cognitive ity to learn new facts and to form new memories for
capacity as studied in the laboratory remain largely events (Scoville & Milner, 1957). This initial discovery
underexplored. set the stage for decades of detailed investigation into
In this chapter, we address these questions by review- the role of the hippocampus and surrounding MTL
ing research on multiple memory systems alongside a cortices in memory. Although the precise role is still
complementary body of work on decision making and debated, substantial progress has been made both in
learning from reward. Much research in the latter area anatomical precision and in functional characterization
has detailed a neural mechanism for learning about (as described in chapters 45 and 46, this volume).
actions, which appears to correspond, at least in part, The selectivity of memory impairments in patient
to procedural learning as envisioned by memory H.M. and others highlighted another important fact
researchers. Importantly, however, there has been about memory organization: damage to the MTL did
increasing consciousness among decision researchers not impair other forms of learning: the ability to learn
Figure 50.1 Structure and task events from a probabilistic habit-learning mechanisms. Patients with disrupted striatal
classification task, often referred to as the “weather-predic- function due to Parkinson’s disease are slow to learn the
tion” task. (A) Each of four visual cues—cards with shapes—is probabilistic associations—and healthy participants show
independently and probabilistically associated with either increased activation in the striatum—when learning is driven
“rain” or “sun.” (B) On each trial, a combination of one to by immediate, response-contingent feedback. Performing this
three cards is shown. Subjects respond based on their predic- task when there is a distracting secondary task at the same
tion of the weather for that trial, and receive response-contin- time—a behavioral hallmark of habits—is also related to
gent feedback. Learning this task is thought to involve implicit increased activation in the striatum.
592 memory
(measured as increased accuracy in performance) Dopamine and habits
despite having no explicit memory for the testing
episode (revealed by multiple-choice questions about A different line of research suggests a mechanism that
details of the testing events). By contrast, Parkinson’s might underlie a dopaminergic role in habit formation.
disease patients, especially those with severe symptoms, The dopaminergic input to striatum (and indeed to
were impaired at incremental learning of the task, but other parts of the forebrain, including hippocampus)
had intact explicit memory for the task events (Knowl- arises from neurons whose cell bodies are located in two
ton et al., 1996; Knowlton, Squire, & Gluck, 1994). small midbrain nuclei, the substantia nigra pars com-
This double dissociation was consistent with other pacta and the ventral tegmental area. These are the
similar double dissociations obtained in animal research neurons whose degeneration is primarily implicated in
between different forms of learning, such as place Parkinson’s disease.
versus response learning (e.g., Packard, 1999). Together, Recordings from these dopaminergic neurons in
such findings were central in advancing the notion that behaving monkeys suggest that they carry a very sugges-
the basal ganglia and MTL support two dissociable tive signal related to reward. These neurons tend to
memory systems. In particular, the findings provided behave similarly to one another; they have a low back-
evidence that the basal ganglia are necessary for non- ground firing rate punctuated by brief, phasic excita-
motor, incremental learning of stimulus-response asso- tions and inhibitions. Following a series of reports
ciations. These findings in humans converge with a describing various circumstances under which these
large body of animal lesion work also demonstrating a phasic firing modulations occur (see Schultz, 1992, for
key role for the basal ganglia in incremental, stimulus- an early review), seminal computational modeling work
response learning (e.g., Foerde & Shohamy, 2011; pointed out that many of these responses could collec-
Mishkin, Malamut, & Bachevalier, 1984; Packard, 1999; tively be understood as a signal known as a reward predic-
Yin & Knowlton, 2006). tion error (Houk, Adams, & Barto, 1995; Montague,
These findings highlighted the basal ganglia as a Dayan, & Sejnowski, 1996; Schultz, Dayan, & Montague,
potential substrate of habit learning, raising many 1997). A reward prediction error is the difference
questions about the mechanisms by which such learn- between the reward you expected and the reward you
ing takes place. First, while Parkinson’s disease patients got; in computer science and engineering, such signals
are impaired on some types of implicit learning, they are commonly used for learning to make such predic-
are not impaired at others, indicating the need for a tions, since they carry information about how to adjust
careful characterization of what exactly a habit is, and errant predictions in light of experience (Sutton &
how habit learning differs from other sorts of nonde- Barto, 1998).
clarative memories. Second, questions were raised Figure 50.2A shows a typical response in a dopamine
about whether participants completing the “weather- neuron (Fiorillo, Tobler, & Schultz, 2003). In this case,
prediction” task were necessarily learning the contin- a monkey was trained that five different pictures would
gencies implicitly; subsequent studies demonstrated be followed by a drop of juice, or not, with five different
that healthy subjects could use simple, explicit strate- probabilities ranging in steps from 0 to 100 percent.
gies to support learning. Furthermore, studies with The figure shows the neural response when a drop of
functional imaging in humans found that the hippo- juice is received following each of these pictures. If
campus is activated during learning of the “weather- the monkey has learned the contingencies (which his
prediction task,” in addition to the basal ganglia behavior demonstrates he has), then he should expect
(Poldrack et al., 2001; Shohamy, Myers, Onlaor, & reward to different degrees, ranging from not at all to
Gluck, 2004). Indeed, as subsequent studies demon- certainty. When a reward is actually received, then, the
strated, both healthy people and Parkinson’s disease prediction error should range from completely unpre-
patients might in fact rely on the hippocampus when dicted to no error at all, with various degrees of surprise
learning in this task (Foerde, Knowlton, & Poldrack, in between. Accordingly, across the conditions the
2006; Moody, Bookheimer, Vanek, & Knowlton, 2004). neuron shows a phasic excitation to reward propor-
Thus, even in what is considered to be a prototypical tional to this error. The figure also shows that when a
implicit learning task, multiple memory systems may reward is expected but omitted, the neuron shows an
work together to contribute to learning. inhibitory response consistent with the (negative) pre-
As outlined below, many similar questions—and diction error. However, due to the low background
some answers—have come up in a different line of firing rate of the neurons, it has been harder to detect
research demonstrating a role for midbrain dopamine a quantitative relationship between the size of the
neurons in learning. negative error and the magnitude of the inhibition
(Bayer & Glimcher, 2005; Bayer, Lau, & Glimcher, 2007; or chess: they help chain together behaviors by provid-
Fiorillo et al., 2003). ing feedback for actions that set the stage for further
Figure 50.2A also shows that in addition to the successful ones.
response to received or omitted primary rewards, the These basic findings have now been replicated and
neuron also responds to cues (here, the pictures) that extended in many studies in monkeys and rodents.
are predictive of reward, and more so for those most Human functional imaging in similar tasks also tells
predictive of reward. This can also be understood as a a similar story: the blood oxygen level–dependent
reward prediction error, in a particular class of models (BOLD) signal at dopaminergic targets (principally
called temporal-difference learning models (Sutton, ventral striatum; see figure 50.2B) shows positive and
1988). Here the goal is to predict not just immediate negative excursions related to positive and negative
rewards but cumulative future rewards, and reward pre- reward prediction errors (McClure, Berns, & Mon-
diction errors occur whenever there is an unexpected tague, 2003; O’Doherty, Dayan, Friston, Critchley, &
change in your future prospects: here, when an unex- Dolan, 2003), and indeed this activity looks very much
pected cue comes on signaling the availability of future like the dopaminergic responses of figure 50.2A, but
reward. In computer science, these sorts of predictive smeared out by a slow hemodynamic response. Of
responses are particularly useful for learning tasks course, the metabolic activity detected by functional
requiring multiple behaviors in sequence, such as mazes MRI (fMRI) is not specifically diagnostic of a particular
594 memory
underlying neural cause such as dopamine. However, dopamine activity might, by strengthening or weaken-
there are a number of indications, for example, ing synapses onto these neurons, therefore strengthen
from pharmacological and Parkinson’s disease studies or weaken the chance of repeating those movements in
(Pessiglione, Seymour, Flandin, Dolan, & Frith, 2006; the future. Such a rule—strengthening actions that lead
Schonberg et al., 2010) that the prediction-error related to positive reward prediction error and weakening
portion of the striatal BOLD signal is affected by those that lead to negative prediction error—is used in
dopamine. computational models of action learning, known as the
actor-critic. In psychology, learning rules of this sort
Dopamine and Movement How does all this relate to have a long history going back to Thorndike’s “law of
behavior, and more specifically to habit learning? A effect” (Thorndike, 1911).
basic abstract concept is that reward prediction errors
are useful for learning to predict rewards, and learning Dopamine, Learning, and the Basal Ganglia A
to predict rewards is useful for learning how to choose dopaminergic role in learning from reward prediction
the most rewarding actions. But how does this actually errors, then, is a candidate mechanism for habit learn-
work? ing and some of the cognitive symptoms in Parkin-
One clue is that the medium spiny neurons of stria- son’s disease. But so far, we have presented mostly
tum (MSNs; the projection neurons of the structure, correlational (unit recording) evidence supporting
which are targeted by dopaminergic inputs) are strongly this idea. Do manipulations of dopamine actually
implicated in movement, for instance, due to the prom- affect learning in a manner predicted by these theo-
inent motor symptoms of Parkinson’s disease. A classic ries? A very suggestive first clue is that dopamine in
understanding of their functional anatomy (Alexander striatum appears to be a common link across drugs of
& Crutcher, 1990; DeLong, 1990) holds that they receive abuse: many, like cocaine and amphetamines, mimic
inputs from frontal cortex (notably, motor-related or release dopamine directly, whereas many others
areas), and ultimately reciprocate them indirectly, via a (such as nicotine and alcohol) appear to do so indi-
series of connections through additional basal ganglia rectly. These effects have been modeled in terms of
and thalamic structures. Two such pathways from stria- the reward prediction error learning theories described
tum back to cortex have been classically described above (Redish, 2004); the idea is simply that drugs
(although there are likely more), called the direct and amplify or mimic positive reward prediction errors
indirect pathways. Due to the number of inhibitory and promote further drug-seeking.
synapses along the route, it is believed that the direct In humans, research in Parkinson’s patients, building
pathway has a net excitatory and the indirect a net on the early work with the weather-prediction task men-
inhibitory effect on cortex. The concept is that these tioned above, has sought cognitive deficits in reward
two pathways act a bit like a gas pedal and brakes for learning more directly related to the emerging picture
behavior, exciting or inhibiting cortical neurons that of dopaminergic function. For instance, Shohamy,
control movement. Myers, Onlaor et al. (2004), inspired by the temporal-
Thus, dopamine is well positioned to affect move- difference model of predictive dopaminergic responses
ment behavior via impinging upon striatal MSNs in the such as the cue-related responses in figure 50.2, tested
direct and indirect pathways. Indeed, one effect of Parkinson’s patients on a multistep chaining task in
dopamine is thought to be directly exciting MSNs pro- which they had to learn a sequence of behaviors by trial
jecting into the direct pathway and inhibiting those and error. Patients were specifically impaired at such
projecting into the indirect pathway, so as to have an learning over multiple steps relative to a simple single-
overall invigorating effect on movement. The removal step association. Second, based on the idea that phasic
of this input (and the consequent imbalance favoring dopamine promotes repeating successful actions (and
the indirect pathway) is a classic explanation for many the absence of dopamine favors avoiding unsuccessful
of the movement difficulties in Parkinson’s disease ones), Frank, Seeberger, and O’Reilly (2004) combined
(DeLong, 1990). a probabilistic learning task with a transfer phase that
However, dopamine does more than directly excite tested to what extent the resulting behavior was sup-
or inhibit MSNs; it also modulates the plasticity of their ported by learning to repeat successful versus learning
inputs from cortex (Reynolds & Wickens, 2002). This is to avoid unsuccessful actions. Parkinson’s patients
a candidate site for learning driven by the dopaminer- tested off their dopamine-replacement medication
gic prediction error described in the previous section. tended to learn via avoidance, while medication pro-
In short, MSNs appear to be involved in producing or duced more learning from success. Similar asymmetries
withholding movements, and plasticity driven by phasic between learning from positive and negative feedback
596 memory
with the sort of habits predicted by the procedural There has been much additional work using the
learning mechanism discussed here—whereas more weather-prediction task, testing and confirming many
moderately trained behaviors are devaluation sensitive, predictions that derive from our understanding of how
that is, goal-directed in a manner inconsistent with such dopamine, prediction errors, and the striatum support
theories. These two modes appear to be learned in learning, as well as testing more extensively the habitual
parallel, but to trade off in controlling behavior, as nature of the representations that are learned.
though in competition. For instance, lesion studies in First, the reinforcement-driven nature of the learning
animals and functional imaging in humans suggest that appears to be an essential element for determining
goal-directed and habitual behaviors arise from disso- whether or not the striatum is involved. When healthy
ciable neural pathways, involving distinct loops con- participants learn the same task by observing the correct
necting regions of frontal cortex to associated parts of outcome, instead of making a response and receiving
the dorsal striatum. Animals with lesions to areas feedback, learning in healthy people depends more on
involved with goal-directed action, such as the dorsome- the hippocampus and less on the striatum. Removing
dial striatum, can learn to lever-press for food, but even the feedback-based component of learning in this task
early in training (when goal-directed action would nor- also ameliorates the learning deficits in patients with
mally dominate), devaluation probes demonstrate this Parkinson’s disease (Poldrack et al., 2001; Shohamy,
behavior is habitual. Conversely, lesions to habit areas, Myers, Grossman et al., 2004). In fact, the same is true
such as the dorsolateral striatum, render even over- even when the feedback on each trial is simply delayed
trained lever-pressing perpetually goal-directed: unlike by several seconds, as shown in figure 50.3 (Foerde,
in intact animals, it does not transition to devaluation- Race, Verfaellie, & Shohamy, 2013; Foerde & Shohamy,
insensitive (see Yin & Knowlton, 2006, for review). 2011). In addition, and as predicted by the reward pre-
Results like these suggest that procedural learning does diction errors recorded in dopamine neurons, the
not strictly depend on extensive training—rather, it is learning impairments in Parkinson’s disease differ
learned even early in training but normally dominates based on whether learning is driven by rewards or by
only later. punishments, and each is differentially impacted by
All this brings us back to dopamine. The temporal- dopaminergic medication. Notably, under most condi-
difference learning hypothesis predicts that dopamine tions people are poor at explicitly expressing what they
should affect behavior by producing habits. This is the have learned. Thus, the striatum appears to be special-
sort of learning for which the signals recorded from ized for learning from immediate, response-contingent
dopamine neurons (described above) appear to be feedback, regardless of the implicit versus explicit
appropriate. The experiments described in this section nature of the learning.
verify that habits of this sort do indeed exist in humans Second, manipulating the circumstances and the
and animals, although they appear to be accompanied neural systems driving probabilistic learning can change
by an additional, goal-directed decision mechanism. the representation of what was learned from flexible to
But can they be tied to dopamine? For the most part, inflexible in systematic ways. One hallmark feature of
the experiments that tie dopamine to decisions in a habits is that they are more likely to guide behavior
causal sense, from the weather-prediction task to recent when people are distracted. Indeed, learning in the
optogenetic studies, demonstrate that dopamine is weather-prediction task engages the striatum more,
involved in guiding choices without using a manipula- and results in more habitual (less flexible) knowledge,
tion such as reward devaluation to examine whether when learned under dual-task conditions compared
these choices are specifically of the habitual sort. There with learning under single-task conditions, which elicits
are as yet only a couple of studies suggesting that inter- more hippocampal activation and more flexibility
fering with dopamine specifically blocks the formation (Foerde et al., 2006). Learning under dual-task condi-
of habits, while leaving goal-directed behavior intact tions has also been shown to shift learning to be less
(Faure, Haberland, Conde, & El Massioui, 2005; Wang goal-directed and more habitual in other learning
et al., 2011). tasks that bear a closer logical resemblance to the
classic devaluation work in rodents (Otto, Gershman,
Markman, & Daw, 2013).
Relating memory to action Together, these findings suggest that which systems
support learning depends on the circumstances under
How does all this relate, then, to the declarative versus which learning takes place, rather than the structure
procedural distinction—and the classic work with the of the knowledge to be learned, and that which system
weather-prediction task, with which we began? is engaged during learning will have important
598 memory
and both are closely tied to hippocampus in rodents Interactions Finally, viewing the declarative versus
and humans (Dusek & Eichenbaum, 1997; Shohamy & procedural distinction in the broader context of how
Wagner, 2008). these systems might ultimately give rise to action also
We earlier mentioned that the traditional view of helps to showcase different possibilities for how they
procedural memory holds that it differs from declara- must interact in the service of behavior. From this
tive memory in being inflexible, implicit, and gradual. view, the simple picture of two independent systems
The experiments and theories discussed above formal- underlying memory for different sorts of information
ize the sense in which habitual behaviors are more seems unlikely to explain the rich interactions
inflexible than goal-directed ones. What about the between these mechanisms, both psychologically and
other characteristics? neurally. Understanding these interactions is a major
In memory research, traditionally a great deal of topic of current research in both the memory and
weight has been put on consciousness as a defining decision areas.
feature of different memory systems. Although the First, the circuitry associated with these two systems
essential role of consciousness in memory is now ques- is deeply interconnected. Hippocampus provides a
tioned (e.g., Henke, 2010), a common assumption major input to ventral striatum. Dopamine, famous for
going back to the multiple memory systems theory is its role in driving learning in the striatum, also inner-
that declarative memories are consciously aware, while vates the hippocampus (Gasbarri, Packard, Campana,
procedural learning is not. In contrast, this feature has & Pacitti, 1994; Groenewegen, Vermeulen-Van der Zee,
not, historically, been considered a core characteristic te Kortschot, & Witter, 1987; Samson, Wu, Friedman, &
of goal-directed versus habitual behaviors, which were Davis, 1990). It has been suggested that this innervation
instead defined more operationally in terms of the sorts is so important that dopamine is necessary for long-
of associations or representations that drive the actions term plastic changes in hippocampal neurons (e.g.,
(Dickinson, 1985). Interestingly, recent work in memory Huang & Kandel, 1995; for review, see Bethus, Tse, &
has raised questions about whether consciousness aware- Morris, 2010; Lisman, Grace, & Duzel, 2011). Under-
ness is actually a fundamental and necessary feature of standing the role of dopamine in hippocampal learn-
the distinction between memory systems. Recent reports ing—and how this role interacts with and complements
demonstrate that flexible, hippocampally guided behav- its role in driving learning in the basal ganglia—is still
ior often occurs in the absence of awareness (Henke, an active area of investigation, and many open ques-
2010; Schapiro, Kustner, & Turk-Browne, 2012; Wimmer tions remain. Nonetheless, there is evidence to suggest
& Shohamy, 2012). This suggests that a more computa- that learning in the hippocampus, like in the basal
tional or process-based perspective on the nature of the ganglia, is sensitive to motivation and to outcomes,
representations may be more useful, and leaves open effects that are likely mediated by dopaminergic inputs
the question of how, when, and if consciousness plays to both of these systems (for review, see Shohamy &
into learning and memory. Adcock, 2010).
Similarly, the speed of learning (one shot vs. incre- Similarly, research on the brain’s action systems has
mental over many experiences) is another feature that now clearly rejected the traditional view of a subcorti-
has been traditionally associated with multiple memory cal (basal ganglia) habit system alongside a cortical
systems, but again does not play a defining role in the (prefrontal or hippocampal) goal-directed action
decision work. This, too, most likely does not align in system. Such a view was always suspicious given the
so simple a way with different neural memory systems. “loop” structure by which frontal cortex and striatum
For instance, the aforementioned acquired equivalence are tightly interconnected (Alexander & Crutcher,
and transitive inference effects are both characteristic 1990). Accordingly, the functional anatomy from lesion
examples of flexible, MTL-mediated learning, but at and imaging studies now makes clear that both sorts of
least some forms of them are not one-shot; instead, the action each rely on different parts of both basal ganglia
relationships that support flexible transfer on these and cortex, and in particular that there are parts of
tasks must be built up slowly over multiple episodes. striatum associated with goal-directed action and parts
Meanwhile, the lesion studies discussed above suggest of frontal cortex involved in habits (Yin & Knowlton,
that habits do not arise only after extensive, incremen- 2006).
tal overtraining, but can support the initial acquisition Anatomical features like these challenge the tradi-
of lever-pressing as well, albeit perhaps not one-shot tional view of the hippocampus and the basal ganglia
learning. Instead here too a critical factor is likely the as independent and distinct. Consistent with this, there
representation—what is learned—rather than the speed is much evidence to suggest that the hippocampus and
with which it is learned. the basal ganglia interact in complex ways to learn and
600 memory
the multiple phases of reward-seeking behavior. J Neurosci, Dickinson, A. (1985). Actions and habits: The development
31(30), 10829–10835. of behavioural autonomy. Philos Trans R Soc Lond B Biol Sci,
Adcock, R. A., Thangavel, A., Whitfield-Gabrieli, S., 308(1135), 67–78.
Knutson, B., & Gabrieli, J. D. (2006). Reward-motivated Dickinson, A., & Balleine, B. (2002). The role of learning
learning: Mesolimbic activation precedes memory forma- in the operation of motivational systems. In C. R. Gallistel
tion. Neuron, 50(3), 507–517. (Ed.), Stevens’ handbook of experimental psychology, Vol. 3:
Alexander, G. E., & Crutcher, M. D. (1990). Functional Learning, motivation and emotion (3rd ed., pp. 497–534). New
architecture of basal ganglia circuits: Neural substrates of York, NY: Wiley.
parallel processing. Trends Neurosci, 13(7), 266–271. Dusek, J. A., & Eichenbaum, H. (1997). The hippocampus
Balleine, B. W., & O’Doherty, J. P. (2010). Human and and memory for orderly stimulus relations. Proc Natl Acad
rodent homologies in action control: Corticostriatal deter- Sci USA, 94(13), 7109–7114.
minants of goal-directed and habitual action. Neuropsycho- Duzel, E., Bunzeck, N., Guitart-Masip, M., Wittmann, B.,
pharmacology, 35(1), 48–69. Schott, B. H., & Tobler, P. N. (2009). Functional imaging
Bayer, H. M., & Glimcher, P. W. (2005). Midbrain dopamine of the human dopaminergic midbrain. Trends Neurosci,
neurons encode a quantitative reward prediction error 32(6), 321–328.
signal. Neuron, 47(1), 129–141. Eichenbaum, H., Fagan, A., Mathews, P., & Cohen, N. J.
Bayer, H. M., Lau, B., & Glimcher, P. W. (2007). Statistics (1988). Hippocampal system dysfunction and odor discrim-
of midbrain dopamine neuron spike trains in the awake ination learning in rats: Impairment or facilitation depend-
primate. J Neurophysiol, 98(3), 1428–1439. ing on representational demands. Behav Neurosci, 102(3),
Bethus, I., Tse, D., & Morris, R. G. (2010). Dopamine and 331–339.
memory: Modulation of the persistence of memory for Faure, A., Haberland, U., Conde, F., & El Massioui, N.
novel hippocampal NMDA receptor-dependent paired (2005). Lesion to the nigrostriatal dopamine system dis-
associates. J Neurosci, 30(5), 1610–1618. rupts stimulus-response habit formation. J Neurosci, 25(11),
Bodi, N., Keri, S., Nagy, H., Moustafa, A., Myers, C. E., 2771–2780.
Daw, N., … Gluck, M. A. (2009). Reward-learning and the Fiorillo, C. D., Tobler, P. N., & Schultz, W. (2003). Dis-
novelty-seeking personality: A between- and within-subjects crete coding of reward probability and uncertainty by dopa-
study of the effects of dopamine agonists on young Parkin- mine neurons. Science, 299(5614), 1898–1902.
son’s patients. Brain, 132(Pt. 9), 2385–2395. Foerde, K., Knowlton, B. J., & Poldrack, R. A. (2006).
Cavaco, S., Anderson, S. W., Allen, J. S., Castro-Caldas, Modulation of competing memory systems by distraction.
A., & Damasio, H. (2004). The scope of preserved proce- Proc Natl Acad Sci USA, 103(31), 11778–11783.
dural memory in amnesia. Brain, 127(Pt. 8), 1853–1867. Foerde, K., Race, E., Verfaellie, M., & Shohamy, D. (2013).
Chang, Q., & Gold, P. E. (2003a). Intra-hippocampal lido- A role for the medial temporal lobe in feedback-driven
caine injections impair acquisition of a place task and facili- learning: Evidence from amnesia. J Neurosci, 33(13),
tate acquisition of a response task in rats. Behav Brain Res, 5698–5704.
144(1–2), 19–24. Foerde, K., & Shohamy, D. (2011). Feedback timing modu-
Chang, Q., & Gold, P. E. (2003b). Switching memory systems lates brain systems for learning in humans. J Neurosci,
during learning: Changes in patterns of brain acetylcholine 31(37), 13157–13167.
release in the hippocampus and striatum in rats. J Neurosci, Frank, M. J., Seeberger, L. C., & O’Reilly, R. C. (2004). By
23(7), 3001–3005. carrot or by stick: Cognitive reinforcement learning in par-
Cohen, N. J., & Eichenbaum, H. (1993). Memory, amnesia, and kinsonism. Science, 306(5703), 1940–1943.
the hippocampal system. Cambridge, MA: MIT Press. Gabrieli, J. D. (1998). Cognitive neuroscience of human
Cools, R., Altamirano, L., & D’Esposito, M. (2006). memory. Annu Rev Psychol, 49, 87–115.
Reversal learning in Parkinson’s disease depends on Gasbarri, A., Packard, M. G., Campana, E., & Pacitti, C.
medication status and outcome valence. Neuropsychologia, (1994). Anterograde and retrograde tracing of projections
44(10), 1663–1673. from the ventral tegmental area to the hippocampal forma-
Corbit, L. H., & Balleine, B. W. (2000). The role of the tion in the rat. Brain Res Bull, 33(4), 445–452.
hippocampus in instrumental conditioning. J Neurosci, Gluck, M. A., Shohamy, D., & Myers, C. (2002). How do
20(11), 4233–4239. people solve the “weather prediction” task? Individual vari-
Corbit, L. H., Ostlund, S. B., & Balleine, B. W. (2002). ability in strategies for probabilistic category learning. Learn
Sensitivity to instrumental contingency degradation is Mem, 9(6), 408–418.
mediated by the entorhinal cortex and its efferents via the Groenewegen, H. J., Vermeulen-Van der Zee, E., te
dorsal hippocampus. J Neurosci, 22(24), 10976–10984. Kortschot, A., & Witter, M. P. (1987). Organization of
Daw, N. D., Gershman, S. J., Seymour, B., Dayan, P., & the projections from the subiculum to the ventral striatum
Dolan, R. J. (2011). Model-based influences on humans’ in the rat. A study using anterograde transport of Phaseolus
choices and striatal prediction errors. Neuron, 69(6), vulgaris leucoagglutinin. Neuroscience, 23(1), 103–120.
1204–1215. Henke, K. (2010). A model for memory systems based on
Daw, N. D., Niv, Y., & Dayan, P. (2005). Uncertainty-based processing modes rather than consciousness. Nat Rev Neu-
competition between prefrontal and dorsolateral striatal rosci, 11(7), 523–532.
systems for behavioral control. Nat Neurosci, 8(12), Houk, J. C., Adams, J. L., & Barto, A. G. (1995). A model of
1704–1711. how the basal ganglia generate and use neural signals that
DeLong, M. R. (1990). Primate models of movement predict reinforcement. In J. C. Houk, J. L. Davis, & D. G.
disorders of basal ganglia origin. Trends Neurosci, 13(7), Beiser (Eds.), Models of information processing in the basal
281–285. ganglia (pp. 249–270). Cambridge, MA: MIT Press.
602 memory
Shohamy, D., & Adcock, R. A. (2010). Dopamine and adap- Thorndike, E. L. (1911). Animal intelligence: Experimental
tive memory. Trends Cogn Sci, 14(10), 464–472. studies. New York, NY: Macmillan.
Shohamy, D., Myers, C. E., Grossman, S., Sage, J., Gluck, M. Tolman, E. C. (1948). Cognitive maps in rats and men. Psychol
A., & Poldrack, R. A. (2004). Cortico-striatal contributions Rev, 55(4), 189–208.
to feedback-based learning: Converging data from neuro- Tsai, H. C., Zhang, F., Adamantidis, A., Stuber, G. D.,
imaging and neuropsychology. Brain, 127(Pt. 4), 851–859. Bonci, A., de Lecea, L., & Deisseroth, K. (2009). Phasic
Shohamy, D., Myers, C. E., Onlaor, S., & Gluck, M. A. firing in dopaminergic neurons is sufficient for behavioral
(2004). Role of the basal ganglia in category learning: How conditioning. Science, 324(5930), 1080–1084.
do patients with Parkinson’s disease learn? Behav Neurosci, Wang, L. P., Li, F., Wang, D., Xie, K., Wang, D., Shen, X., &
118(4), 676–686. Tsien, J. Z. (2011). NMDA receptors in dopaminergic
Shohamy, D., & Wagner, A. D. (2008). Integrating memories neurons are crucial for habit learning. Neuron, 72(6),
in the human brain: Hippocampal-midbrain encoding of 1055–1066.
overlapping events. Neuron, 60(2), 378–389. Wimmer, G. E., & Shohamy, D. (2012). Preference by associa-
Smith, K. S., Virkud, A., Deisseroth, K., & Graybiel, A. M. tion: How memory mechanisms in the hippocampus bias
(2012). Reversible online control of habitual behavior by decisions. Science, 338(6104), 270–273.
optogenetic perturbation of medial prefrontal cortex. Proc Wise, R. A. (2004). Dopamine, learning and motivation. Nat
Natl Acad Sci USA, 109(46), 18932–18937. Rev Neurosci, 5(6), 483–494.
Squire, L. R., & Zola, S. M. (1996). Structure and function Wittmann, B. C., Schott, B. H., Guderian, S., Frey, J. U.,
of declarative and nondeclarative memory systems. Proc Natl Heinze, H. J., & Duzel, E. (2005). Reward-related FMRI
Acad Sci USA, 93(24), 13515–13522. activation of dopaminergic midbrain is associated with
Sutton, R. S. (1988). Learning to predict by the methods of enhanced hippocampus-dependent long-term memory
temporal differences. Mach Learn, 3(1), 9–44. formation. Neuron, 45(3), 459–467.
Sutton, R. S., & Barto, A. G. (1998). Reinforcement learning: Yin, H. H., & Knowlton, B. J. (2006). The role of the basal
An introduction. Cambridge, MA: MIT Press. ganglia in habit formation. Nat Rev Neurosci, 7(6),
Tan, K. R., Yvon, C., Turiault, M., Mirzabekov, J. J., 464–476.
Doehner, J., Labouebe, G., … Luscher, C. (2012). GABA
neurons of the VTA drive conditioned place aversion.
Neuron, 73(6), 1173–1183.
ABSTRACT On the face of it, prediction, imagination, and temporally distant equivalents. Similarly, D’Argembeau
memory seem to be distinct cognitive functions. However, and van der Linden (2006) showed that individual dif-
metacognitive, cognitive, neuropsychological, and neuroim- ferences, such as capacity for visual imagery, affected
aging evidence is emerging that they are not, suggesting inti-
mate links in their underlying processes. Here we explore the phenomenological experience of episodic memory
these empirical findings and the evolving theoretical frame- and episodic future thinking. Notably, specific errors
works that seek to explain how a common neural system sup- made when recollecting the past were also evident
ports our recollection of times past, imagination, and our when people engaged in predicting the future (for
attempts to predict the future. review, see Gilbert & Wilson, 2007).
If memory and imagination are intimately linked, it
“We predominantly stand in the present facing the future is natural to ask whether they are supported by the same
rather than looking back to the past.” neural structures. This has been examined in two ways:
—Suddendorf and Corballis (2007) one with a focus on the hippocampus, and the other
with an eye to an extended set of brain areas—includ-
Although not immediately intuitive, the idea that ing the medial and lateral prefrontal cortices, posterior
memory, imagination, and predicting what might cingulate, retrosplenial and lateral temporal cortices,
happen in the future are intimately linked is not new. and the medial temporal lobe (MTL)—often called the
Throughout the centuries, this notion has consistently “core network” for episodic memory and imagination
reemerged within philosophical, psychological, and (Buckner & Carroll, 2007; Spreng, Mar, & Kim, 2009).
contemporary work, along with the belief that the role Considering first the hippocampus, since the seminal
of recollection is to serve imagination and prediction work of Scoville and Milner (1957), the MTL, and par-
of the future. For instance, in 1798 Immanuel Kant ticularly the hippocampus, have been recognized as
noted that “Recalling the past (remembering) occurs playing a pivotal role in our ability to recollect past
only with the intention of making it possible to foresee experiences. This paper described the case of H.M.,
the future” (Kant, 2006, p. 79). In 1871, the White who underwent bilateral temporal lobectomy for the
Queen in Lewis Carroll’s Through the Looking Glass relief of intractable epilepsy, rendering him amnesic, or
astutely observed, “It’s a poor sort of memory that only unable to acquire new episodic memories. They noted
works backwards” (Carroll, 1994, ch. 5), while in 2006 “after the operation this young man could no longer
Suddendorf argued, “It is accurate prediction of the recognize the hospital staff nor find his way to the bath-
future, more so than accurate memory of the past room, and he seemed to recall nothing of the day-to-day
per se, that conveys adaptive advantage” (Suddendorf, events of his hospital life” (Scoville & Milner, 1957,
2006, p. 1007). p. 14). The case of H.M. precipitated 50 years of subse-
There is behavioral evidence supporting the connec- quent work examining the role of the hippocampus in
tion between memory and imagination of the future. memory (for reviews, see Corkin: 2002; Squire, 2004).
For instance, D’Argembeau and van der Linden (2004) Episodic memory, however, is not the only function
asked participants to mentally “reexperience” personal that has been ascribed to the hippocampus. In the
past events (episodic memory) or to “preexperience” 1970s, O’Keefe and Dostrovsky (1971) discovered cells
(episodic future thinking; Atance & O’Neill, 2001) in the rat hippocampus that displayed location-specific
possible future events that had or would occur in the firing (so-called “place cells”), while damage to the hip-
close or distant past or future. For the past and future, pocampus severely disrupted spatial navigation ability
temporally close events were associated with more sen- (Morris, Garrud, Rawlins, & O’Keefe, 1982). This evi-
sorial and contextual details and evoked stronger feel- dence prompted O’Keefe and Nadel (1978) to suggest
ings of reexperiencing (or preexperiencing) than the the hippocampus may provide the spatial scaffold for
606 memory
Figure 51.1 Example of an imagined scenario from Hass- participant who was age-, education-, and IQ-matched to P03.
abis, Kumaran, Vann, et al. (2007). The cue is shown at the Interviewer’s probing comments are in italics. Relevant back-
top, below which is an excerpt from P03, a patient with bilat- ground information is noted in square brackets.
eral hippocampal damage, followed by that of a control
and does not involve true visualization of an imagined a set of autobiographical memories and were later
scene, or engagement of the hippocampus (Mullally, asked to imagine novel events containing a combina-
Vargha-Khadem, & Maguire, 2014). In one study that tion of specific details taken from these episodic memo-
failed to observe imagination-based scene deficits in ries. The older adults continued to generate fewer
patients with hippocampal damage acquired in adult- episodic details for imagined events, suggesting that
hood (Squire et al., 2010), the patients were in fact not recollection difficulties alone cannot explain their
amnesic, showing instead only mild and nonsignificant impaired scene/event imagination (see also Addis,
memory deficits when compared with a control group Roberts, & Schacter, 2011; Gaesser, Sacchetti, Addis, &
(Maguire & Hassabis, 2011; Mullally, Hassabis, & Schacter, 2011; Romero & Moscovitch, 2012).
Maguire, 2012). This suggests that an impaired scene Co-existing episodic memory and scene/event imagi-
imagination profile may only occur in the context of nation impairments have also been noted in a number
adult-acquired hippocampal damage and concomitant, of other populations. Williams et al. (1996) reported
severe memory disturbance. that suicidal, depressed patients’ recollection of the
past and simulation of the future lacked specific details
Scene Imagination Deficits in Other Populations and appeared “over-general” relative to controls, while
Scene/event imagination deficits have also been D’Argembeau, Raffard, & van der Linden (2008) found
observed in older adults and in other patient groups; that patients with schizophrenia generated significantly
populations in which hippocampal function and other fewer episodic details for both past and future events.
neural components of the core network are known to Interestingly, hippocampal atrophy has been docu-
be compromised (for review, see Schacter, Addis, & mented in depression (Bremner et al., 2000), schizo-
Buckner, 2008). In one study, Addis, Wong, and Schacter phrenia (Herold et al., 2012), and in the aging brain
(2008) noted deficits in future simulations in older (Driscoll & Sutherland, 2005), suggesting that damage
adults. In a subsequent study, Addis, Musicaro, Pan, to this brain region may be a critical factor underlying
and Schacter (2010) sought to investigate whether the these disparate cognitive impairments.
impairment could simply be an expression of an epi- Looking beyond the hippocampus, there have been
sodic memory deficit whereby elderly adults simply few neuropsychological studies focussed on the role of
“recast” entire remembered events into the future. brain areas outside the MTL in imagination of fictitious
Using a recombination paradigm (see also Addis, Pan, or future scenes. In one such study, Berryhill, Picasso,
Vu, Laiser, & Schacter, 2009) participants first provided Arnold, Drowos, and Olson (2010) tested patients with
608 memory
each other and with a context (Mullally, Intraub, & occurs in relation to scenes and not to single, isolated
Maguire, 2012) are also challenges for this view. objects (Gottesman & Intraub, 2002), a dissociation
that mirrors the imagination dichotomy observed in
Scene Construction Theory A third account is amnesic patients (Hassabis, Kumaran, Vann, et al.,
scene construction theory (SCT; Hassabis & Maguire, 2007). Critically, however, BE depends upon an intact
2007, 2009; Maguire & Mullally, 2013), which was origi- ability to construct scenes. Thus, patients with bilateral
nally proposed following the observation that patients hippocampal damage and amnesia, who are unable to
with hippocampal damage and amnesia are unable to construct scenes, should be unable to form this
imagine scenes. In contrast to the constructive episodic extended representation and therefore fail to commit
simulation hypothesis, which had a broader focus on the BE memory error. This would result in a situation
the core network, SCT attempts to account specifically where amnesic patients display superior performance
for the role of the hippocampus in scene imagination, relative to healthy controls, and is exactly what Mullally,
episodic memory, and spatial navigation. In essence, Intraub, and Maguire (2012) recently demonstrated on
SCT proposes that the hippocampus primarily acts to a variety of BE paradigms (see figure 51.2).
facilitate the construction of atemporal scenes, and in These results enabled Mullally, Intraub, and Maguire
doing so allows the event details of episodic memories (2012) to conclude that the patients’ attenuated BE
and imagined future experiences a foundation upon could not be attributed to memory impairment, since
which to reside. In this way, hippocampal-dependent they paradoxically outperformed the controls on a
scene construction processes are held to underpin and range of memory tasks. Therefore, in this context,
support episodic memory, predicting the future, spatial impaired memory did not lead to impaired scene con-
navigation, and perhaps even dreaming and mind- struction. Instead, impaired scene construction actually
wandering, with the addition of self-related and tempo- led to better memory, thus successfully separating these
ral processing implemented via the recruitment of two processes.
other regions within the core network. These data also suggest that a function of the hip-
SCT thus places scenes at the center of hippocampal pocampus is the implicit and continuous prediction of
information processing, underpinning (but not wholly the upcoming environment; that is, the hippocampus
responsible for) critical functions such as episodic is continually constructing scenes, extrapolating beyond
memory, prospection, and spatial navigation. This has the boundaries of our current field of view. A recent
intuitive appeal—for most people, recalling the past, neuroimaging study (Chadwick, Mullally, & Maguire,
thinking about the future, and planning how to get 2013), which investigated the role of the hippocampus
somewhere typically involve imagining scenes. This also in BE in control participants, supports this hypothesis.
resonates with the patients’ experiences of trying to Specifically, they found robust activity in the hippocam-
imagine scenes: “There is no scene in front of me here. pus during the presentation of the original scene stimu-
It’s frustrating because I feel like there should be. I feel lus. Significantly, this activity was observed only on trials
like I’m listening to the radio instead of watching it on where participants later committed the BE error. This
the TV. I’m trying to imagine different things happen- suggests that the hippocampus is involved early at the
ing but there’s no visual scene opening out in front of initial stage of the BE effect where the predictive scene
me”; “It’s hard trying to get the space, it keeps getting extension process (attenuated in the hippocampal
squashed” (Mullally, Intraub, & Maguire, 2012). The patients) is hypothesized to occur. The BE data (Chadwick
appeal of the SCT is that it offers a unified account of et al., 2013; Mullally, Intraub, & Maguire, 2012), in
why such a wide range of seemingly disparate functions addition to the original scene construction findings
are impaired following hippocampal damage. (Hassabis, Kumaran, Vann, et al., 2007; Hassabis, Kumaran,
Further recent evidence appears to place scene con- & Maguire, 2007; Mullally, Intraub, & Maguire, 2012),
struction at the heart of hippocampal processing. support the idea that the primary function of the hip-
Boundary extension (BE; Intraub & Richardson, 1989) pocampus may not be mnemonic (Maguire & Mullally,
is a ubiquitous cognitive phenomenon whereby we erro- 2013; see also Graham, Barense, & Lee, 2010) but may
neously remember seeing more of a scene than was instead be to predict the nature of the world beyond
present in the sensory input, and occurs because when the immediate sensorium (Bar, 2011).
we view a scene, we implicitly extrapolate beyond the In conclusion, we believe that metacognitive, cogni-
borders to form an extended representation of that tive, neuropsychological, and neuroimaging evidence
scene. In the absence of the original visual input, this leaves in no doubt the close ties between episodic
extended scene is misremembered instead of the origi- memory, scene imagination, and predicting the future.
nal input, causing a memory error. Of note, BE only An understanding of this relationship is still in its
infancy. Interestingly, electrophysiological studies docu- ACKNOWLEDGMENT E.A.M. is supported by the Wellcome
menting preplay (e.g., Johnson & Redish, 2007) add to Trust. Note that this chapter is an edited version of an article
that appeared in The Neuroscientist: Mullally, S. L., and Maguire,
this evolving picture by hinting at animal parallels in
E.A. (2014). Memory, imagination and predicting the future:
predicting what might occur in the future. Further A common brain mechanism? The Neuroscientist, 20, 220–234.
studies in humans will need to move beyond describing
this relationship to focus on the mechanisms involved,
the precise functions of each area within the core REFERENCES
network, and the connectivity between them. In this
Addis, D. R., Musicaro, R., Pan, L., & Schacter, D. L.
way, we are confident that the next five years will hasten (2010). Episodic simulation of past and future events in
important new insights into this question that has older adults: Evidence from an experimental recombina-
intrigued through the ages. tion task. Psychol Aging, 25(2), 369–376.
610 memory
Addis, D. R., Pan, L., Vu, M. A., Laiser, N., & Schacter, D. Cooper, J. M., Vargha-Khadem, F., Gadian, D. G., &
L. (2009). Constructive episodic simulation of the future Maguire, E. A. (2011). The effect of hippocampal damage
and the past: Distinct subsystems of a core brain network in children on recalling the past and imagining new experi-
mediate imagining and remembering. Neuropsychologia, ences. Neuropsychologia, 49(7), 1843–1850.
47(11), 2222–2238. Corkin, S. (2002). What’s new with the amnesic patient HM?
Addis, D. R., Roberts, R. P., & Schacter, D. L. (2011). Age- Nat Rev Neurosci, 3, 153–160.
related neural changes in autobiographical remembering D’Argembeau, A., Raffard, S., & van der Linden, M. (2008).
and imagining. Neuropsychologia, 49(13), 3656–3669. Remembering the past and imagining the future in schizo-
Addis, D. R., Wong, A. T., & Schacter, D. L. (2007). Remem- phrenia. J Abnorm Psychol, 117(1), 247–251.
bering the past and imagining the future: Common and D’Argembeau, A., & van der Linden, M. (2004). Phenome-
distinct neural substrates during event construction and nal characteristics associated with projecting oneself back
elaboration. Neuropsychologia, 45(7), 1363–1377. into the past and forward into the future: Influence of
Addis, D. R., Wong, A. T., & Schacter, D. L. (2008). Age- valence and temporal distance. Conscious Cogn, 13(4),
related changes in the episodic simulation of future events. 844–858.
Psychol Sci, 19(1), 33–41. D’Argembeau, A., & van der Linden, M. (2006). Individual
Andelman, F., Hoofien, D., Goldberg, I., Aizenstein, O., & differences in the phenomenology of mental time travel:
Neufeld, M. Y. (2010). Bilateral hippocampal lesion and a The effect of vivid visual imagery and emotion regulation
selective impairment of the ability for mental time travel. strategies. Conscious Cogn, 15(2), 342–350.
Neurocase, 16(5), 426–435. De Vito, S., & Della Sala, S. (2011). Predicting the future.
Andrews-Hanna, J. R., Reidler, J. S., Sepulcre, J., Poulin, Cortex, 47(8), 1018–1022.
R., & Buckner, R. L. (2010). Functional-anatomic fraction- Driscoll, I., & Sutherland, R. J. (2005). The aging hippo-
ation of the brain’s default network. Neuron, 65(4), campus: Navigating between rat and human experiments.
550–562. Rev Neurosci, 16(2), 87–121.
Atance, C. M., & O’Neill, D. K. (2001). Episodic future Gaesser, B., Sacchetti, D. C., Addis, D. R., & Schacter, D.
thinking. Trends Cogn Sci, 5(12), 533–539. L. (2011). Characterizing age-related changes in remem-
Banks, W. P., & Karam, S. J. (1996). Medico-psychological bering the past and imagining the future. Psychol Aging,
study of a memory disorder. Conscious Cogn, 5(1–2), 2–21. 26(1), 80–84.
Bar, M. (Ed.). (2011). Prediction in the brain: Using our past to Gilbert, D. T., & Wilson, T. D. (2007). Prospection: Experi-
generate a future. Oxford, UK: Oxford University Press. encing the future. Science, 317(5843), 1351–1354.
Bartlett, F. C. (1932). Remembering: A study in experimental Gottesman, C. V., & Intraub, H. (2002). Surface construal
and social psychology. Cambridge, UK: Cambridge University and the mental representation of scenes. J Exp Psychol
Press. Human, 28(3), 589–599.
Berryhill, M. E., Picasso, L., Arnold, R., Drowos, D., & Graham, K. S., Barense, M. D., & Lee, A. C. (2010). Going
Olson, I. R. (2010). Similarities and differences between beyond LTM in the MTL: A synthesis of neuropsychological
parietal and frontal patients in autobiographical and and neuroimaging findings on the role of the medial tem-
constructed experience tasks. Neuropsychologia, 48(5), poral lobe in memory and perception. Neuropsychologia,
1385–1393. 48(4), 831–853.
Botzung, A., Denkova, E., & Manning, L. (2008). Experi- Hassabis, D., Kumaran, D., & Maguire, E. A. (2007). Using
encing past and future personal events: Functional neuro- imagination to understand the neural basis of episodic
imaging evidence on the neural bases of mental time travel. memory. J Neurosci, 27(52), 14365–14374.
Brain Cogn, 66(2), 202–212. Hassabis, D., Kumaran, D., Vann, S. D., & Maguire, E. A.
Bremner, J. D., Narayan, M., Anderson, E. R., Staib, L. H., (2007). Patients with hippocampal amnesia cannot imagine
Miller, H. L., & Charney, D. S. (2000). Hippocampal new experiences. Proc Natl Acad Sci USA, 104(5), 1726–1731.
volume reduction in major depression. Am J Psychiatry, Hassabis, D., & Maguire, E. A. (2007). Deconstructing epi-
157(1), 115–118. sodic memory with construction. Trends Cogn Sci, 11(7),
Buckner, R. L. (2010). The role of the hippocampus in pre- 299–306.
diction and imagination. Annu Rev Psychol, 61, 27–48. Hassabis, D., & Maguire, E. A. (2009). The construction
Buckner, R. L., & Carroll, D. C. (2007). Self-projection and system of the brain. Philos Trans R Soc Lond B Biol Sci,
the brain. Trends Cogn Sci, 11(2), 49–57. 364(1521), 1263–1271.
Burgess, N., Maguire, E. A., & O’Keefe, J. (2002). The Herold, C. J., Lässer, M. M., Schmid, L. A., Seidl, U.,
human hippocampus and spatial and episodic memory. Kong, L., Fellhauer, I., … Schröder, J. (2012). Hippo-
Neuron, 35(4), 625–641. campal volume reduction and autobiographical memory
Carroll, L. (1994). Through the looking glass. London: deficits in chronic schizophrenia. Psychiatry Res, 211(3),
Penguin. Originally published 1871. 189–194.
Chadwick, M. J., Mullally, S. L., & Maguire, E. A. (2013). Hurley, N. C., Maguire, E. A., & Vargha-Khadem, F. (2011).
The hippocampus extrapolates beyond the view in scenes: Patient HC with developmental amnesia can construct
An fMRI study of boundary extension. Cortex, 49(8), future scenarios. Neuropsychologia, 49(13), 3620–3628.
2067–2079. Intraub, H., & Richardson, M. (1989). Wide-angle memo-
Cohen, N. J., & Eichenbaum, H. (1991). The theory that ries of close-up scenes. J Exp Psychol Learn Mem Cogn, 15(2),
wouldn’t die: A critical look at the spatial mapping theory 179–187.
of hippocampal function. Hippocampus, 1(3), 265–268. Johnson, A., & Redish, A. D. (2007). Neural ensembles in
Cohen, N. J., & Eichenbaum, H. (1993). Memory, amnesia, and CA3 transiently encode paths forward of the animal at a
the hippocampal system. Cambridge, MA: MIT Press. decision point. J Neurosci, 27(45), 12176–12189.
612 memory
VII
LANGUAGE
AND ABSTRACT
THOUGHT
Chapter 52 AMUNTS AND CATANI 619
59 GLIMCHER 683
Introduction
PETER HAGOORT
ABSTRACT Anatomical models of language include a core left Core language regions
perisylvian network between Wernicke’s region in the tem-
poral lobe and Broca’s region in the frontal lobe. In addi- Methods and theoretical approaches to anatomically
tion, a number of other cortical areas and subcortical nuclei
based definitions of Broca’s region have changed
contribute to language functions. This extended language
network comprises the inferior parietal cortex (i.e., throughout the history of its study, resulting in the
Geschwind’s region), the anterior temporal lobe, the ante- proposal of many different parcellation schemes (for a
rior insula, the presupplementary motor area (preSMA), review, see Amunts & Zilles, 2012). On the surface,
and the lateral occipitotemporal cortex (i.e., visual word Broca’s region corresponds to the pars opercularis and
form area, VWFA). At the subcortical level, the thalamus,
triangularis of the inferior frontal gyrus. Cytoarchitec-
basal ganglia, and cerebellum are reciprocally connected to
cortical language regions and modulate their activity. In this tonically, the pars opercularis and triangularis are
chapter, we describe the anatomy of the language cortical largely occupied by areas 44 and 45, respectively. Their
regions as defined by their neuronal composition and layer- most distinct feature is the presence of very large pyra-
ing (i.e., cytoarchitectonics) and the receptor distribution midal cells in deep layer III. These two areas differ from
for the main neurotransmitters of the central nervous each other for the thickness of layer IV, well developed
system (i.e., receptorarchitectonics). The principal associa-
tion, projection, and commissural language pathways in area 45, but thinner and invaded by pyramidal cells
described by recent diffusion tractography studies will also from neighboring layers III and V in area 44. Layer IV
be reviewed. A particular focus of this chapter is to charac- is known to process, among others, input from the
terize the anatomical interindividual variability and its rele- thalamus.
vance to function in the normal brain and in language Probabilistic cytoarchitectonic maps of areas 44 and
disorders.
45 have been generated using quantitative and statisti-
cal criteria for delineating their borders (figure 52.1A;
Amunts et al., 1999). These maps reveal important ana-
Language development in humans occurred as a tomical features of Broca’s region. Intersubject variabil-
direct consequence of the functional specialization of ity of the overall surface and extent of these two areas
cortical areas located in the temporal (i.e., Wernicke’s is evident in the healthy human brain (Amunts et al.,
region) and frontal lobes (i.e., Broca’s region) of the 2004). In addition, a large portion of these two areas is
left hemisphere. These areas are primarily dedicated buried in the depths of the sulci, which prevents any
to language, and for this reason are identified as core possibility of predicting their exact extent and location
language regions. Other more bilaterally distributed solely on the basis of the surface anatomy of the inferior
cortical areas connect to these left-lateralized core frontal gyrus (figure 52.1A), which is in itself highly
regions and form an extended language network. In variable among people (Ono, Kubik, & Abernathey,
this chapter, we aim to introduce the reader to 1990).
current approaches used to map the main language One advantage of probabilistic maps is that they offer
areas as defined by cytoarchitectonics, receptorarchi- a quantitative approach to cortical localization in
tectonics, and their networks as defined by in vivo patients with brain lesion and in functional activation
diffusion tractography. Although our approach is studies (see www.jubrain.fz-juelich.de; Amunts et al.,
fundamentally anatomical, we refer to possible func- 2004; Fischl et al., 2007). For example, it has been
tional correlates and associated symptoms whenever shown that the maximum probability map for area 44
possible. is more likely to overlap with activations generated by
Figure 52.1 Cortical mapping of Broca’s region based on 44 and 45. In addition, the borders of the two areas are not
post-mortem cytoarchitectonic and receptorarchitectonic always well identified by the main sulci of this region: inferior
analysis. (A) Top, surface view of the probabilistic maps for frontal sulcus (ifs); horizontal ramus of the lateral fissure
Brodmann’s area 44 (BA44) and 45 (BA45) derived from (hrlf); ascending ramus of the lateral fissure (arlf); precentral
quantitative cytoarchitectonic measurements of 10 individual sulcus (prcs); diagonal sulcus (ds), (data from Amunts et al.,
brains. The probabilistic maps are intended to quantify the 2004). (B) Further segregation of Broca’s region based on
interindividual variability of the areas that compose Broca’s neurotransmitter receptors distribution. This analysis can
region, where “hot” tones correspond to a high anatomical reveal unique characteristics of subregions that may share
overlap, and “cold” tones to higher variability. This interindi- similar cytoarchitectonic morphology. Area 44v and op8, for
vidual variability can be appreciated in the bottom panel, example, have a similar density of GABAA or alternately
where the left and right maps for BA44 (red) and BA45 GABA(A) receptors but different density of cholinergic mus-
(yellow) are reported for two representative brains. Note, for carine receptors M1. (See color plate 44.)
example, the difference in the left-right asymmetry of BA
tasks involving syntactic processing (Friederici & Kotz, structural and functional connectivity of language net-
2003) and lexical decision (Heim, Eickhoff, Ischebeck, works (Heim, Eickhoff, & Amunts, 2011). These are
& Amunts, 2007), whereas area 45 is more involved in only a few applications of many. An important aspect to
semantic tasks (Amunts et al., 2004). The cytoarchitec- consider is that functional activations often do not cor-
tonic maps can also be used as seed regions to study respond to the cytoarchitectonic subdivision of Broca’s
region into two areas when language-related activations the posterior aspect of the superior temporal gyrus and
are superimposed on probabilistic maps. Indeed, often part of the middle temporal gyrus and is involved in
activations overlap only in part with these areas and multiple aspects of language processing (Binder et al.,
extend to other neighboring areas (Indefrey et al., 2000; Hickok & Poeppel, 2007). Cytoarchitectonically,
2001). Several reasons may explain the lack of an exact Wernicke’s region includes areas 22 and 42 and part of
correspondence between functional activations and area 37 (Aboitiz & Garcia, 1997), with further segmen-
cytoarchitectonic maps, including methodological tations identified using probabilistic cytoarchitectonic
shortcomings of both techniques, which may limit our and receptorarchitectonic mapping. The superior tem-
ability to capture the complex segregation of Broca’s poral gyrus, for example, is occupied by three distinct
region. areas (figure 52.3). Area Te1, the primary auditory
The application of novel receptorarchitectonic analy- cortex, occupies a large portion of the Heschl gyrus
sis to cortical areas (Zilles & Amunts, 2009) shows that (Morosan et al., 2001). Area Te2 is partially located
Broca’s region is indeed more complex than previously on the most lateral aspect of the Heschl gyrus and on
assumed. Amunts et al. (2010) analyzed the receptor the opercular surface of the superior temporal gyrus,
distribution of different neurotransmitter systems (glu- whereas area Te3 occupies approximately the posterior
tamate, GABA, serotonin, acetylcholine) in serial sec- two-thirds of the free surface of the superior temporal
tions of the posterior frontal cortex. The receptor gyrus (Morosan et al., 2005) and differs from its neigh-
analysis showed that not only do areas 44 and 45 differ boring areas for the prominent size and high density of
in their receptor distribution profile, but each area can pyramidal neurons in layer IIIc and a high cellular
be further subdivided into multiple areas (figure 52.1B). density in layer V. With respect to receptor architecture,
These subareas show a similar receptorarchitectonic Te3 is characterized by a low density of muscarinic M2
pattern to areas 44 and 45, which arguably suggests receptors (Morosan et al., 2005). Te3 comprises further
their involvement in language-related processes. Inter- subdivisions (Rivier & Clarke, 1997; Wallace, Johnston,
estingly, ventral area 6 has a pattern that is more similar & Palmer, 2002) that have some correspondence with
to areas 44 and 45 than to area 47, which suggests a findings from functional activation studies (Friederici
more distant structural and possibly functional related- & Kotz, 2003).
ness between areas of the same gyrus (i.e., areas 44, 45, Mesgarani and colleagues (2014) have recently shown
and 47 are parts of the inferior frontal gyrus) than that the superior temporal gyrus responds selectivity to
between areas lying in different gyri (areas 44 and 45 distinct phonetic features of spoken language and con-
in the inferior frontal gyrus and area 6 in the precentral tains the acoustic-phonetic representation of speech. In
gyrus; figure 52.2). their comment to this paper, Grodzinsky and Nelken
The anatomical correlates of Wernicke’s region are (2014) emphasized that speech representation in the
more ambiguously defined than those of Broca’s region. experiment was dominated by abstract and linguistically
In its modern definition, Wernicke’s region occupies defined features (Grodzinsky & Nelken, 2014).
Te1.0
Te2
Te1.1
Te2
Te3
Figure 52.3 Cytoarchitectonic of the language areas in the superior temporal region (Morosan et al., 2005).
Other cortical regions In the mesial aspect of the superior frontal gyrus, area
6 and, in part, posterior areas 8 and 32 form an impor-
In addition to Broca’s and Wernicke’s regions, other tant language region that participates in planning, initia-
cortical areas are relevant to speech and language. tion, and monitoring of speech (Paulesu, Frith, &
Geschwind’s region in the inferior parietal lobule Frackowiak, 1993; Penfield & Roberts, 1959). This region,
involves area 40 in the supramarginal gyrus and area 39 which corresponds to the anterior cingulate cortex and
in the angular gyrus (Catani et al., 2005). A more recent preSMA, has also been characterized using modern cyto-
parcellation of this region into seven cytoarchitectonic architectonic or receptorarchitectonic mapping in the
areas has been proposed, with five subdivisions covering human and macaque brains (Geyer et al., 1998; Pal-
area 40 and two subdivisions covering area 39 (Caspers omero-Gallagher, Mohlberg, Zilles, & Vogt, 2008). The
et al., 2006). Of the five anterior subdivisions, the three preSMA is involved in action monitoring, distinguishing
most anterior ones show a receptor architecture similar self from others’ actions, and low-level aspects of mental-
to that of area 44 in the frontal lobe (Caspers et al., izing (Lombardo et al., 2010; Yoshida et al., 2011). The
2013). Geschwind’s region is part of an extended anterior insula is highly connected to the frontal oper-
network that links core language regions to areas culum and Broca’s region (Catani et al., 2012; Cerliani
involved in memory, semantic knowledge, and social et al., 2012) and is often activated in language-produc-
cognition (Binder et al., 2009; Jacquemot & Scott, 2006; tion tasks (Dronkers, 1996). In the ventral occipitotem-
Meyer, Obleser, Anwander, & Friederici, 2012; Vilberg poral region, the visual word form area is specialized for
& Rugg, 2008). processing written strings (Cohen et al., 2000).
6
1-
3-
46
10
40 39 White matter pathways
44
45 43
41 Language areas are reciprocally connected through a
19 18
11 47 42
42
22 17 complex system of association and commissural tracts
38 21 37 whose anatomy has been studied in humans with post-
mortem blunt dissections and more extensively with
20
diffusion imaging tractography (figure 52.5A).
The arcuate fasciculus is a dorsal perisylvian associa-
A
MD tion tract connecting Wernicke’s, Geschwind’s, and
IN
VLp Broca’s regions. Within the arcuate fasciculus, two par-
VA LP LD allel pathways have been distinguished: the medial,
Mid
VLm
direct pathway connecting Wernicke’s with Broca’s
VLa CM Pul region (i.e., the arcuate fasciculus sensu strictu or long
VPl
VPm segment); and the indirect pathway, consisting of an
VPi MGN anterior segment that links Broca’s to Geschwind’s
LGN
region and a posterior segment between Geschwind’s
Figure 52.4 Thalamocortical connections of the language and Wernicke’s region (Catani et al., 2005). López-
areas. A: anterior group; LGN: lateral geniculate nucleus; Barroso et al. (2013) showed that performance in word
LP: lateroposterior; MD: mediodorsal; MGN: medial genicu- learning correlates with microstructural properties and
late nucleus; Mid: midline group; Pul: pulvinar; VA: ventral
strength of functional connectivity of the direct connec-
anterior; VLa: ventral lateral anterior; VPi: ventral posterior
inferior; VPl: ventral posterior lateral; VPm: ventral posterior tions between Broca’s and Wernicke’s regions. This
medial; VLp: ventral lateral posterior. (Modified from Catani study demonstrates that our ability to learn new words
& Thiebaut de Schotten, 2012.) relies on an efficient and fast communication between
Broca’s
region Wernicke’s
A
Anterior region
temporal Geschwind’s
Arcuate fasciculus (long segment) Frontal aslant tract lobe region
Visual word
Arcuate fasciculus (anterior segment) Uncinate fasciculus
form area
Arcuate fasciculus (posterior segment) Inferior longitudinal fasciculus
Figure 52.5 Language networks visualized with diffusion indicate higher degree of transcallosal connections. Note the
tractography. (A) Association pathways of left hemisphere reduced interhemispheric connectivity for most of the lan-
connecting the main language regions. (B) Density of callosal guage regions. (See color plate 45.)
cortical projections based on diffusion imaging. “Hot” tones
auditory temporal and motor frontal regions. Schulze, fronto-occipital fasciculus provides direct connections
Vargha-Khadem, and Mishkin (2012) suggested that between occipital (and perhaps posterior temporal
the absence of these connections in nonhuman pri- cortex) and frontal cortex in the human brain (Forkel
mates might explain our unique ability to learn new et al., 2012). The relevance of this fasciculus to reading,
words. The long segment is also important for syntactic writing, and other aspects of language remains to be
processing (Tyler et al., 2011; Wilson et al., 2011) and established (Duffau et al., 2005; Forkel et al., 2012).
words repetition (Parker Jones et al., 2014). Within the The frontal aslant tract is a newly described pathway
indirect pathway, the anterior and posterior segments connecting Broca’s region with medial frontal areas
have different roles in reading (Thiebaut de Schotten including preSMA and cingulate cortex (Catani et al.,
et al., 2014), phonological and semantic processing 2012; Lawes et al., 2008). Medial regions of the frontal
(Binder et al., 2009; Newhart et al., 2012), verbal lobe facilitate speech initiation through direct connec-
working memory ( Jacquemot & Scott, 2006), and prag- tion to the pars opercularis and triangularis of the infe-
matic interpretation (Hagoort, 2013). rior frontal gyrus. Patients with lesions to these areas
Language areas of the temporal and frontal lobes are present with various degrees of speech impairment
also interconnected by a set of ventral longitudinal from a total inability to initiate speech (i.e., mutism) to
tracts (Catani & Mesulam, 2008). The inferior longitu- mild altered fluency (Catani et al., 2012; Naeser et al.,
dinal fasciculus carries visual information from occipi- 1989). The frontal aslant tract is damaged in patients
tal and posterior temporal areas to the anterior temporal with the nonfluent/agrammatic form of primary pro-
lobe (Catani, Jones, Donato, & Ffytche, 2003) and plays gressive aphasia (Catani et al., 2013).
an important role in visual object recognition, reading, The frontal operculum is connected to the insula
and linking object representations to their lexical labels. through a system of short U-shaped fronto-insular
The uncinate fasciculus connects the anterior temporal tracts (Catani et al., 2012; Cerliani et al., 2012). Direct
lobe to the orbitofrontal region and part of the inferior insular inputs to Broca’s region from the insula
frontal gyrus and may play an important role in lexical provide visceral and emotional information for
retrieval, semantic associations, and naming (Catani speech-output modulation according to internal states.
et al., 2013). The uncinate fasciculus is severely damaged Lesions to these insular connections may result in
in those patients with the semantic variant of primary motor aprosodia (e.g., flat intonation) and apraxia of
progressive aphasia (Catani et al., 2013). The inferior speech (Dronkers, 1996).
females
14
males
N. of subjects
10
40%
30% 30%
Group 1 (~60%) 6
strong left lateralization
10%
2
5%
0
Group 1 Group 2 Group 3
Lateralization pattern
C
100
post-stroke
80
70
–10 –5 0 5 10
Group 3 (~20%)
bilateral, symmetrical Right long segment (index size)
Figure 52.6 Distribution of the pattern of lateralization of 2007). (C) Recovery of language six months after stroke is
the long segment in (A) three groups and (B) differences in correlated with the volume of the right long segment (Forkel
lateralization between genders (modified from Catani et al., et al., 2014).
comments on the manuscript. Michel Thiebaut de Schotten Amunts, K., Schleicher, A., Ditterich, A., & Zilles, K.
kindly provided the data set for figure 52.5. (2003). Broca’s region: Cytoarchitectonic asymmetry and
developmental changes. J Comp Neurol, 465, 72–89.
Amunts, K., Weiss, P. H., Mohlberg, H., Pieperhoff, P.,
Eickhoff, S., Gurd, J., Shah, J. N., Marshall, J. C.,
REFERENCES
Fink, G. R., & Zilles, K. (2004). Analysis of the neural
Aboitiz, F., & Garcia, G. L. (1997). The evolutionary origin mechanisms underlying verbal fluency in cytoarchitectoni-
of language areas in the human brain. A neuroanatomical cally defined stereotaxic space—the role of Brodmann’s
perspective. Brain Res Rev, 25, 381–396. areas 44 and 45. NeuroImage, 22, 42–56.
Aboitiz, F., Scheibel, A. B., Fisher, R. S., & Zaidel, E. Amunts, K., & Zilles, K. (2012). Architecture and organiza-
(1992). Fiber composition of the human corpus callosum. tional principles of Broca’s region. Trends Cogn Sci, 16,
Brain Res, 598, 143–153. 418–426.
Amunts, K., Lenzen, M., Friederici, A. D., Schleicher, A., Binder, J. R., Desai, R. H., Graves, W. W., & Conant, L. L.
Morosan, P., Palomero-Gallagher, N., & Zilles, K. (2009). Where is the semantic system? A critical review and
(2010). Broca’s region: Novel organizational principles and meta-analysis of 120 functional neuroimaging studies. Cereb
multiple receptor mapping. PLoS Biol, 8, e1000489. Cortex, 19, 2767–2796.
Amunts, K., Schleicher, A., Bürgel, U., Mohlberg, H., Binder, J. R., Frost, J. A., Hammeke, T. A., Bellgowan,
Uylings, H. B. M., & Zilles, K. (1999). Broca’s region P. S. F., Springer, J. A., Kaufman, J. N., & Possing, E. T.
revisited: Cytoarchitecture and intersubject variability. (2000). Human temporal lobe activation by speech and
J Comp Neurol, 412, 319–341. nonspeech sounds. Cereb Cortex, 10, 512–528.
ABSTRACT Speech research has typically focused on processing basic units from which information on a finer (faster)
single speech events, that is, vowels, syllables, or words. A dif- temporal scale can be extracted (Ghitza, 2011).
ferent approach focuses on connected speech. This work Based on linguistic, psychophysical, and physiological
points to new mechanisms for understanding human speech
perception and production in the context of neurophysiology considerations, it has been proposed that speech is
as well as in the context of comparative and evolutionary analyzed in parallel at multiple time scales (Boemio,
studies. The concept at the center of this research program Fromm, Braun, & Poeppel, 2005; Poeppel, 2003;
concerns rhythms—in the acoustic signal, the neural activity, Poeppel, Idsardi, & van Wassenhove, 2008). Both local-
and the motoric output systems. There is a remarkable to-global and global-to-local analyses are carried out
correspondence between average durations of speech or
speech-like expressions and the frequency ranges of cortical
concurrently (multitime resolution processing). The
oscillations. Here we summarize a multitime-resolution principal motivations for this hypothesis are twofold.
hypothesis about the perception of connected speech, capital- First, one single, short temporal integration window
izing on the role that neuronal oscillations play in processing that underpins hierarchical processing—that is, increas-
speech. We then examine the notion of the speech rhythm ingly larger analysis units as one ascends the processing
from a comparative perspective, demonstrating that lip-
hierarchy—fails to account for the spectral and tempo-
smacking behavior in macaques shares crucial features with
speech. The human and nonhuman primate data jointly ral sensitivity of the auditory system and is hard to
make a strong case for the coordination of neural and vocal reconcile with behavioral performance. Second, the
rhythms in communication. computational strategy of analyzing information on
multiple scales is widely used in engineering and bio-
logical systems, and the neuronal infrastructure exists
Across languages, speech typically exhibits a 3–8 Hz to support multiscale computation (Buzsaki & Draguhn,
rhythm (Chandrasekaran, Trubanova, Stillittano, Caplier, 2004). We conjecture that speech is chunked into seg-
& Ghazanfar, 2009; Crystal & House, 1982; Elliot & ments of roughly featural or phonemic length. In paral-
Theunissen, 2009; Greenberg, Carvey, Hitchcock, & lel, there is a “global” analysis that yields coarse,
Chang, 2003; Malecot, Johonson, & Kizziar, 1972), syllabic-scale inferences about speech and that subse-
reflected in the envelope of the speech waveform. This quently refines segmental analysis.
slow energy modulation corresponds roughly to the The speech signal contains events of different dura-
syllabic sequence (or syllabic “chunking”) of speech. tions: short energy bursts and formant transitions occur
Syllabic structure as reflected by the speech envelope is over 10–80 ms, whereas syllabically carried information
perceptually critical because it signals speaking rate, occurs over 150–300 ms. The processing of both types
carries stress and tonal contrasts, and can be viewed as of events is compatible with a hierarchical model
the carrier of the linguistic (question, statement, etc.) (smaller acoustic units are concatenated into larger
or affective (happy, sad, etc.) prosody of an utterance units like syllables), a parallel model (both temporal
(Rosen, 1992). Consequently, high sensitivity to enve- units are extracted independently and then combined),
lope structure and dynamics is critical for successful or a reverse hierarchical model (coarser temporal anal-
perception. Disrupting this rhythm significantly reduces ysis precedes/guides finer analysis). Although there is
intelligibility (Drullman, Festen, & Plomp, 1994; Elliot no decisive evidence, at the behavioral level, the data
& Theunissen, 2009; Ghitza & Greenberg, 2009; Saberi favor the parallel or reverse hierarchy models. There is
& Perrott, 1999; Shannon, Zeng, Kamath, Wygonski, & some independence in the processing of long (slow
Ekelid, 1995; Smith, Delgutte, & Oxenham, 2002), as modulation) and short (fast modulation) units. For
does disrupting the visual component arising from instance, speech is intelligible when it is first segmented
mouth and facial movements (Vitkovitch & Barber, into units up to 60 ms, and these local units are tempo-
1996). Thus, the speech rhythm parses the signal into rally reversed (Saberi & Perrott, 1999). This observation
optimized for different types of processing. Finally, parsing speech at the segmental scale, and right audi-
neuronal oscillations might relate in a principled way tory cortex for parsing speech at the syllabic time scale.
to temporal integration constants of different sizes. The The AST hypothesis accounts for various psychophys-
proposal holds that there exist hemispherically asym- ical and neuroimaging results that show the left tempo-
metric distributions of neuronal ensembles with pre- ral cortex responds better to many aspects of rapidly
ferred shorter versus longer integration constants; modulated auditory content, while right temporal
these cell groups “sample” the input on different time cortex responds better to slowly modulated signals,
scales: left auditory cortex has a relatively higher including music, voices, and other sounds (Zatorre,
proportion of short (gamma) integrating ensembles, Belin, & Penhune, 2002). Systematic differences in
right auditory cortex has more long (theta) integrating integration windows between left and right auditory
neuronal ensembles (figure 53.1B). As a consequence, cortices explain speech functional asymmetry by a
left-hemisphere auditory cortex is better equipped for better sensitivity of left auditory cortex to information
A B C
Lip-smack Chewing Hyoid
Upper lip 4 Tongue 3 Tongue
3 2
2
1
Displacement (cm)
Figure 53.3 (A) The anatomy of the macaque monkey vocal (C) Arrow schematics show the direction of significant influ-
tract as imaged with cineradiography. The key vocal tract ence from each structure onto the other two as measured by
structures are labeled: the lips, tongue, and hyoid. (B) Time- the partial directed coherence analysis of signals such as those
displacement plot of the tongue, inter-lip distance, and in B. (Figure reprinted with permission from Ghazanfar et al.,
hyoid for one exemplar each of lip-smacking and chewing. 2012, with permission from Elsevier.)
A C
12 n = 11
10
Total viewing time [s]
B 8
10 Hz
Lip-smack rhythm
6
6 Hz
2
3 Hz
0
0 500 1000 6Hz 3Hz / 10Hz
Time [ms]
Figure 53.4 (A) Example frame sequence from a video clip natural rate (6 Hz). (C) Total viewing times in seconds for
showing an avatar face producing a lip-smacking gesture. Lip- individual subjects (lines) and grand total (mean and stan-
smacking is characterized by regular cycles of vertical jaw dard error). All but one subject showed a preference for the
movement, often involving a parting of the lips. (B) Synthetic avatar with biological lip-smack rate. (Figure reprinted with
lip-smack rates were faster (10 Hz) or slower (3 Hz) than the permission from Ghazanfar et al., 2013.)
ABSTRACT The neural framework for human language and aspects of syntactic, semantic, and morphological
communication is analyzed in terms of two intersecting but processing, suggests major limitations to this classic
evolutionarily distinguishable neurobiological systems: a left-
approach to language and the brain. The Broca-Wer-
hemisphere frontotemporal system required for core syntac-
tic functions, integrated with a bihemispheric system shared nicke diagram captures (imperfectly) one important
with our primate relatives that supports sound to meaning aspect of the neural substrate for language function—
mapping and semantic and pragmatic interpretation. This the key role of the left-hemisphere network—but it
view of language as a dynamic coalition of interacting brain obscures another, equally important one. This is the
systems provides the framework for a cross-linguistic examina- role of bihemispheric systems and processes, where
tion (in English, Polish, and Arabic) of the way in which basic
generative systems for inflectional and derivational word for- both left and right hemispheres work together to
mation are distributed across these two systems and whether provide the fundamental underpinnings for human
(and how) different languages vary in these respects. communicative processes. A more fruitful approach to
human language and communication requires an
extended neurobiological framework, where these
The ability to communicate using language is funda- capacities are supported by intersecting but evolution-
mental to the distinctive and remarkable success of the arily and functionally distinguishable neural systems.
modern human. It is this capacity that separates us most This leads to a view of language as a dynamic coalition
decisively from our primate cousins, despite all that we of interacting brain systems, where we ask how core
have in common across species as intelligent social pri- linguistic and communicative functions are distributed
mates. A major scientific challenge for the cognitive across these systems and whether this distribution of
neurosciences is to understand exactly this relationship. functions varies between languages.
What is the neurobiological context in which human In this chapter we will develop this framework as a
language and communication have emerged, and what basis for examining how a core property of human
are the special human properties that make language language, dealing with words and their internal struc-
itself possible? ture, is neurally instantiated and how this varies across
Scientific (and popular) thinking about this relation- a contrastive sample of different languages. This aspect
ship has been dominated by our strong phenomeno- of language, known as morphology, is built around
logical experience, as speakers and listeners, that we the morpheme—the minimal meaning-bearing linguistic
command and inhabit a unified social communication unit—and the ways in which different morphemes—
system built around language. In neuroscientific terms, typically a stem morpheme (e.g., sad, jump) and a bound
this has long been expressed in the form of a single, grammatical morpheme (e.g., –ness, -ed)—combine to
central language system built around a left-hemisphere create new surface forms (sadness, jumped). A basic dis-
cortical network, linking critical areas in the left poste- tinction is made between inflectional and derivational
rior temporal lobe (“Wernicke’s area” in figure 54.1) processes, where inflectional morphology subserves
and in the left inferior frontal lobe (“Broca’s area” in primarily grammatical functions, while derivational
figure 54.1). This classic concept, established through morphology is associated with the creation and repre-
the pioneering research of the nineteenth-century neu- sentation of new words in the language. As we will see
rologists Paul Broca and Carl Wernicke, reflects the below, these different aspects of morphological func-
selective effects of left-hemisphere brain damage on tion can be linked to broader, neurobiologically based
basic grammatical functions involved in language com- distinctions between left-hemisphere systems essential
prehension and language production. for combinatorial morphosyntax, and bilateral systems
Recent research in our laboratories (and others), underpinning lexical representation and semantic
focusing on the neural systems supporting different interpretation.
generate comparable levels of competition, but none- not (jump, yard). Furthermore, the complex forms
theless show no signs of selective LH activation (see themselves are distributionally extreme, with typically
figure 54.2). just one dominant affix. The regular {-ed} past tense
This pattern of results for English suggests a strong morpheme applies across the board to more than
distinction between inflectional morphology and deri- 10,000 English verbs, with a subset of about 160 idiosyn-
vational morphology, with inflectional morphemes cratic irregular forms, while the noun plural {-s} applies
selectively triggering decompositional, primarily syntac- to every English noun with just a handful of exceptions
tic linguistic processes, while derivational morphemes (geese, oxen, etc.). The language learning environment
apparently play their role as part of the whole-word induced by these distributional properties may lead to
semantic representations created for each derived sharper or different distinctions between inflected and
lexeme, preferentially engaging bilateral temporal uninflected forms, compared to languages with more
systems. The generality and the cross-linguistic applica- inflection classes and with no clear distinction between
bility of such an analysis needs to be carefully exam- regular and irregular forms. Where derivationally
ined, however, given the particular properties of English complex forms are concerned, these occur in English
morphology. in a much less decompositional processing environ-
Inflectional morphology in English is very reduced in ment than in languages where almost every surface
scope, and the majority of words in the language are form is morphologically complex (as in the Slavic and
produced as bare stems. This leads to a strong distribu- Semitic languages discussed below), and which will also
tional contrast between words that are overtly morpho- vary in the complexity and prevalence of the parsing
logically complex (jumped, yards, etc.) and those that are operations that these forms require.
ABSTRACT We review event-related brain potential (ERP) evi- previously gleaned from the context (e.g., “integra-
dence for the rapid deployment of information from semantic tion”; see Brown & Hagoort, 1993) or are instead medi-
memory and current context during language comprehen- ated by real-time predictive processing (Federmeier,
sion. We summarize studies showing how linguistic expectan-
cies can emerge at various levels as sentential context 2007).
accumulates. We detail various qualitatively different pro- For present purposes, prediction involves the activa-
cesses (as indexed by the N400, frontal and parietal late posi- tion of or information about likely upcoming stimuli,
tive components, and a frontal negativity) engaged when prior to their receipt, that plays a causal role in stimulus
predictions are or are not made—and are or are not met—as processing. Linguistic predictions need not be con-
people make sense of sentences.
scious and typically are not. Their representational
In making sense of language, people typically draw on content may be fine-grained and specific, for example,
diverse verbal and nonverbal information from past for a particular expected word or sound, or for seman-
experience and the current environment—that is, tic features or gist or other types of stimulus attributes
context. Although effects of contextual variables may such as grammatical structure. At a given point in time,
be clear, the causal mechanisms of context effects are there may be multiple (even incompatible) predictions
not. A long-standing and influential idea is that biologi- with varying strengths, all subject to dynamic revision.
cal information-processing systems routinely utilize Without unequivocal evidence of representational
contextual information to prepare in advance for what activation prior to the actual occurrence of some “pre-
might come (see Bar, 2011). The hypothesis that expec- dicted” input, facilitated processing can be attributed
tancies (prediction, anticipation, prospection) play a to nonpredictive factors such as increased ease of inte-
causal role in perception and cognition more generally gration of the input with the message-level representa-
is widespread in the cognitive neurosciences—for tion. Thus, determining when and how predictions, as
example, the role of lateral prefrontal cortex in the defined above, play a role in real-time language com-
“expectation of, and preparation for, anticipated events” prehension is nontrivial. ERPs have proven an espe-
(Fuster, 2001, p. 325), “perceptual predictions” (Zacks, cially useful tool for disentangling the contributions of
Speer, Swallow, Braver, & Reynolds, 2007, p. 273), and different language-processing mechanisms, including
the “prospective brain” (Schacter, Addis, & Buckner, prediction, to context effects. In particular, ERPs
2007). However, within the domain of language com- provide a multidimensional window into the conse-
prehension, the role of prediction has been more quences of lexicosentential (in)congruity, which have
controversial. been measured relatively early in time, as amplitude
Contextual variables impact real-time language pro- modulations of the N400 (see also P2 amplitude effects)
cessing, and sentential contexts in particular have been at and prior to the apprehension of critical stimuli, and
shown to modulate, for example, signal-to-noise thresh- in post-N400 positivities and negativities; we discuss
olds for speech recognition in background noise (Miller, each in turn.
Heise, & Lichten, 1951), the duration required for
visual word identification (Tulving & Gold, 1963), The N400
lexical decision times (Fischler & Bloom, 1979), eye
movements in reading (Ehrlich & Rayner, 1981), and The N400 refers to a relative negativity between 250 and
event-related brain potential (ERP) amplitudes in 600 ms in the ERP to a potentially meaningful item
word-by-word reading (Kutas & Hillyard, 1980; Lau, (e.g., spoken or written word, acronym, pseudoword,
Stroud, Plesch, & Phillips, 2006). At issue is whether picture, face, gesture, etc.). The N400 reflects the
these effects arise due to the post hoc match between summed postsynaptic activity of a highly distributed
information derived from a current stimulus and that brain network that includes higher-level perceptual
kutas, federmeier, and urbach: the “negatives” and “positives” of prediction 649
areas and multimodal processing and storage areas in thus likely strength of the prediction for—the unpre-
the left superior/middle temporal gyri and medial and sented BC. Moreover, in a visual half-field version, in
anterior temporal lobes of both hemispheres (Lau, which the category member was briefly (200 ms) flashed
Phillips, & Poeppel, 2008; van Petten & Luka, 2006). in the right or left visual field, this related anomaly
The N400 is especially large to nouns that do not fit N400 effect was limited to the right visual field (which
with the meaning of their preceding context compared biases processing to the left hemisphere). These results
to the contextually most predictable noun (e.g., “dog” underscore the impact of memory structure (in this
in “I take my coffee with cream and dog” versus “sugar”; case, by category-based featural similarity) on word pro-
Kutas & Hillyard, 1980). N400s are also present to all cessing in sentences and suggest the use of predictive
but the most highly predictable nouns, even when they processing mechanisms during normal comprehen-
are semantically congruent. All other factors (word fre- sion, mediated by the left hemisphere (Federmeier,
quency, concreteness, orthographic neighborhood size, 2007).
word class) held constant, N400 amplitude is inversely Metusalem et al. (2012) adapted this paradigm to test
correlated (r > .8) with an item’s cloze probability in the hypothesis that language comprehension is simi-
an offline paper-and-pencil completion task. For some larly influenced by event knowledge–based organiza-
researchers the N400 sensitivity to cloze probability tion of semantic memory. Participants read short
reflects the activity of a predictive language system, passages—for example, A huge blizzard ripped through
whereas for others it reflects variable resource needs for town last night. My kids ended up getting the day off from
a contextual integration operation. Of course, the N400 school. They spent the whole day outside building a big …—
could reflect both, and from attempts to address this continued by the BC (snowman) or by one of two types
issue have emerged novel ERP paradigms that have of linguistically unlicensed, less probable words either
informed the functional organization of semantic related to the general event (playing in the snow) acti-
memory and incremental language processing beyond vated by the passage (jacket) or not (towel). Both types
semantic memory access. of improbable words elicited larger N400s than the BC,
A case in point is the “related anomaly” paradigm but those to the linguistically unlicensed but event-
(Kutas & Hillyard, 1984; Kutas, Lindamood, & Hillyard, related continuations were reliably smaller (figure
1984). While all low cloze words that complete a highly 55.1A). These results point to the immediacy and incre-
constraining context elicit large N400s relative to the mentality of event knowledge use during language com-
high cloze best completions (BC), those for words prehension and to events as an organizing principle of
related to the meaning of the BC are reliably smaller. semantic memory.
For example, the N400 to “weights” is smaller than that Laszlo and Federmeier (2009) similarly employed the
to “collars,” both anomalous endings for “The barbells the related anomaly paradigm to show that the information
strongman lifted were very ____,” presumably because of brought online by sentence context affects processing
the relationship between “weights” and the BC (heavy), of inputs based on their orthography. Participants read
although the relationship was not controlled in early sentences such as “The genie was ready to grant his third
studies. and final” that were completed either by the BC (wish)
Federmeier and Kutas (1999a, 1999b) investigated or by an unexpected word, pseudoword, or illegal string
categorical relations. They used sentence pairs to estab- that was either an orthographic neighbor or not (figure
lish an expectation for a particular member of a cate- 55.1B, words only). Relative to nonneighbors, all neigh-
gory, which was violated in a third of the sentences by bors—regardless of their lexical status—showed facilita-
a member of a different category (between-category tion, that is, smaller N400s. This result is difficult to
violation) and in a third by a different category member explain via mere integrative processes (for another
(within-category violation). Although the two category- example, see Rommers, Meyer, Praamstra, & Huettig,
violation types were equally implausible on average, the 2013). Overall, these results show that context acts
within-category violation enjoyed greater perceptuo- through the structure of event knowledge, semantic
semantic-feature overlap with the expected category categories, and orthography, and further suggests that
exemplar (e.g., He caught the pass and scored another touch- the system can anticipate features of upcoming words
down. There was nothing he enjoyed more than a good game along these dimensions.
of football/baseball/monopoly.). This pattern was observed Arguably, stronger electrophysiological evidence for
for written words, spoken words, and line drawings prediction comes from ERP effects present before the
(Federmeier & Kutas, 2001; Federmeier, McLennan, de predicted item of interest. There are a handful of such
Ochoa, & Kutas, 2002). Critically, the N400 reduction studies with the same general design—a lead-in context
increased as a function of the cloze probability of—and that sets up an expectation for a particular noun,
preceded by a word (e.g., article, adjective) that either ERPs at the gender-marked adjectives in Dutch (also see
matches or mismatches the upcoming noun (e.g., in Otten, Nieuwland, & van Berkum, 2007; Otten & van
grammatical gender, morphology, or phonology). Criti- Berkum, 2008, 2009).
cally, either class of prenominal words could reasonably DeLong, Urbach, and Kutas (2005) argued for pre-
be followed by a congruent matching noun, and is not diction of phonological word forms based on differen-
a violation per se. As such, these should not elicit differ- tial ERPs to the indefinite articles a and an, which have
ing ERPs, unless the comprehender expects a noun that the same meaning but are predictive of the initial
is only consistent with one of the alternative prenomi- phoneme class of the subsequent word (words begin-
nal words. Several such investigations report differing ning with a consonant versus vowel sound, respectively).
ERPs to the noun-matching versus mismatching pre- Participants read sentences ranging in constraint from
nominal alternatives. low to high and continuing with a range of more or less
In a series of experiments, sentential contexts (spoken expected indefinite article-noun pairings; for example,
or written) in Spanish were designed to create an “Because it frequently rains in London, Nigel always carries
expectation for a noun of feminine or masculine gender …” for which an umbrella is highly expected and a news-
(verbal or depicted), occasionally “violated” by an paper, while plausible, is less so. There was a statistically
article of the other grammatical gender. The results reliable inverse correlation between N400 amplitudes
indicate that the language-processing system was antici- and not only the target nouns but also the articles
pating a word of a particular gender, as a differential (figure 55.1C). These data provide clear evidence of
ERP pattern was found for articles of the expected graded linguistic prediction at a phonological level. Yet
versus unexpected gender (Wicha, Moreno, & Kutas, another variant of this design indicates prediction
2003; Wicha, Moreno, & Kutas, 2004; Wicha, Bates, for semantically defined classes of words in Polish
Moreno, & Kutas, 2003). Using a similar logic, van (Szewczyk & Schriefers, 2013).
Berkum, Brown, Zwitserlood, Kooijman, & Hagoort Taken together, these studies point to the availability
(2005) inferred syntactic prediction from differential of a predictive language processer that activates word
kutas, federmeier, and urbach: the “negatives” and “positives” of prediction 651
features and forms in advance of their input. They dem- indicating that an anomaly—detected by the system
onstrate that linguistic expectancies at many levels, during the N400 time window—is being (explicitly)
from semantic and syntactic to orthographic and pho- noted by the participants, who are “reviewing the prior
nological, can emerge as sentential context accumu- context to determine what went wrong and if the
lates. The fact that the system is capable of making problem might be repaired” (Van Petten & Luka, 2012).
predictions, however, does not mean that it always can This view resembles accounts proffered for posterior
or will do so (e.g., there are hemispheric and age- late positive effects (P600s) coincident with syntactic
related differences in the tendency to predict; Feder- processing difficulties, including agreement errors,
meier, 2007). Nor will predictions, when made, always phrase structure violations, and garden path construc-
be correct. Indeed, whereas N400 amplitude varies with tions (e.g., Gouvea, Phillips, Kazanina, & Poeppel,
many lexicosemantic variables, its latency is relatively 2009).
stable, even though the availability of information and In some cases, semantic anomalies elicit posterior
processes necessary for prediction and incremental LPC effects without accompanying N400 effects. (i.e.,
comprehension are considerably more variable across indistinguishable N400s for expected vs. anomalous
time, as a function of the input, task demands, and words). For example, using sentences like “For break-
individual differences, among other factors. This, fast, the boys/eggs would only eat … .” Kuperberg,
together with the need to refine and revise initial inter- Sitnikova, Caplan, and Holcomb (2003) observed no
pretations when predictions are incorrect, would seem N400 differences but a larger posterior positivity to eat
to imply comprehension and prediction-related pro- following eggs than boys (figure 55.1D; see also Hoeks,
cesses post-N400. Next we discuss evidence for these. Stowe, & Doedens, 2004; Kim & Osterhout, 2005; van
Herten, Kolk, & Chwilla, 2005, for similar reports). A
Posterior late positivities: The late positive complex number of different accounts have been put forward
and semantic P600 for these so-called semantic P600 effects. Many inter-
pret the N400 facilitation (amplitude reduction) to the
Processing differences between expected and semanti- anomalies as indicating that the comprehension system
cally anomalous words are sometimes seen post-N400 was subject to a temporary semantic illusion—that is,
(500–900 ms), with anomalous words eliciting increased in contrast to cases wherein LPC and N400 effects co-
centroparietal positivity. This is often referred to as an occur, that the system initially erroneously interprets
effect on the late positive complex (LPC). Semantic the anomalous word as plausible. Several researchers
anomaly effects on the LPC seem to be more depen- have postulated that the N400 reflects a processing
dent on attention and less obligatory than N400 anomaly stage insensitive to certain types of (for example, com-
effects. Whereas N400 effects can be observed even binatorial) processes essential for appreciating the
during the attentional blink or under masking, these semantic implausibility of a given word in a particular
obliterate LPC effects (Luck, Vogel, & Shapiro, 1996; sentence position (e.g., Kuperberg, 2007). The P600 in
Misra & Holcomb, 2003). Diminished LPC effects have this case is taken to reflect processes involved in syntac-
been described in children (e.g., Juottonen, Revonsuo, tic revision or in aligning the conflicting interpreta-
& Lang, 1996) as well as in adults with schizophrenia tions arising in different processing streams. Others
(e.g., Koyama et al., 1994), whose N400 effects in the (Brouwer, Harmut, & Hoeks, 2012) argue convincingly
same conditions are similar to adult controls. There is against such “multistream” accounts. If the N400 is an
notable variability in the tendency to elicit LPC effects index of lexicosemantic retrieval (subject to preactiva-
to semantic anomalies, which does not seem to be tion) and not a direct reflection of contextual integra-
straightforwardly linked to stimulus factors such as con- tion (as elaborated in Kutas & Federmeier, 2011), then
textual constraint (Kos, van den Brink, & Hagoort, the lack of N400 effect in these cases could mean that
2012; Van Petten & Luka, 2012) or to individual differ- semantic access for the anomalous words was facili-
ences in working memory capacity (Kos et al., 2012). tated (for example, because of the semantic relation-
Even with similar stimulus and task conditions, only ship between breakfast, eggs, and eating in the above
about a third of published studies show an LPC post- example); critically, it does not (necessarily) mean that
N400 effect (Van Petten & Luka, 2012), perhaps the comprehension system was ever “fooled.” On this
because, in a given sample, only about half of the people view, the lack of N400 may reflect preactivation, and
seem to be eliciting such effects (Kos et al., 2012). the (semantic) P600 effect indicates that the system
The functional significance of LPC anomaly effects has detected the implausibility and that it is, parsimo-
remains underspecified and says little about predic- niously, functionally equivalent to LPCs following N400
tion per se. Typically, these effects are interpreted as anomaly effects.
kutas, federmeier, and urbach: the “negatives” and “positives” of prediction 653
The frontal positivity to prediction violations, instead, demonstrate how the latency, polarity, and scalp distri-
can be characterized most cleanly when equally unex- bution of ERP measures such as the N400 and late posi-
pected words (in the ideal case, the same lexical items; tivities, in suitable experimental designs, can shed light
Federmeier, Wlotko, De Ochoa-Dewald, & Kutas, 2007) on the mechanisms by which context variables impact
are compared as a function of contextual constraint. the time course of interpretation. Some provide strong
For example, the word log has matched, low-cloze prob- evidence for contextually supported and seemingly
ability in both contexts: graded predictive and incremental processes.
(1) He was cold most of the night and finally got up These studies are existence proofs that prediction
to get another … does occur in language processing. The project now is
(2) He fell on the floor after tripping on the … to characterize principles governing predictive pro-
cesses and the biological mechanisms at work. Of all the
However, context (1) leads readers to have strong contextually available information (verbal and nonver-
predictions for the alternative ending, blanket, whereas bal), which does the system use to generate predictions?
context (2) does not lead to consistent expectations for How fast do they come online, and does this vary with
an alternative word. Relative to (2) the ERP to (1) is any trait and/or state variables and environmental
characterized by increased prefrontal positivity between factors? Is prediction automatic and obligatory, or does
500 and 900 ms (figure 55.1F). it come online under only some conditions? When
The link between this frontal positivity and prediction- predictions are made, what kinds of information are
related processes is strengthened by its absence in most predicted, with what specificity, and what degree of
older adults (Federmeier, Kutas, & Schul, 2010; Wlotko strength? Do contextual variables constrain the space
et al., 2012), who similarly tend not to show other pre- of possible predictions to winner take all, with unsuc-
diction-related ERP patterns (Federmeier et al., 2002). cessful competitors squashed or even inhibited? If pre-
These effects can, however, be observed in a subset of dictions turn out to be incorrect, what is the cost (c.f.,
older adults with high verbal fluency (DeLong, Groppe, Smith & Levy, 2011)? Finally, what is the relation if any,
Urbach, & Kutas, 2012; Federmeier et al., 2010), who of prediction to the speed and depth of the interpreta-
also may show young-like N400 prediction effects (Feder- tions ultimately constructed? Clearly, much remains to
meier et al., 2002). Federmeier and colleagues have be discovered, and it seems likely that ERPs will con-
linked the frontal positivity to processes involved in tinue to play an important explanatory part.
message-level meaning construction in the face of failed
predictions. We speculate that this response may arise
from activity related to the basal forebrain cholinergic
system, which has been shown to be sensitive to the
REFERENCES
degree of “expected uncertainty” in the environment,
and which controls the balance of top-down versus bot- Avery, M. C., Nitz, D. A., Chiba, A. A., & Krichmar, J. L.
tom-up influence on processing, without changing the (2012). Simulation of cholinergic and noradrenergic mod-
ulation of behavior in uncertain environments. Front Comput
attentional set (Avery, Nitz, Chiba, & Krichmar, 2012; Yu
Neurosci, 6, 5.
& Dayan, 2005). Indeed, recent work has shown that the Bar, M. (2011). Predictions in the brain: Using our past to generate
use of top-down predictions in sentence processing, a future. New York, NY: Oxford University Press.
even within young adults who seem likely to be predict- Brouwer, H., Harmut, F., & Hoeks, J. (2012). Getting real
ing, is importantly modulated by the utility of those about semantic illusions: Rethinking the functional role of
the P600 in language comprehension. Brain Res, 1446,
predictions (Wlotko & Federmeier, 2011). For example
127–143.
“fooling” the predictive processing system by presenting Brown, C., & Hagoort, P. (1993). The processing nature of
unexpected synonyms of the best completion—that the N400: Evidence from masked priming. J Cogn Neurosci,
is, words whose semantics but not lexical form are 5, 34–44.
expected—eradicates the frontal positivity when partici- Coulson, S., King, J. W., & Kutas, M. (1998). ERPs and
pants are reading for comprehension (and thus care domain specificity: Beating a straw horse. Lang Cogn Proc,
13, 653–672.
most about semantics) but not when they are making Coulson, S., & Kutas, M. (2001). Getting it: Human event-
lexical decisions (and do care about word form). related brain response to jokes in good and poor compre-
henders. Neurosci Lett, 316, 71–74.
DeLong, K. A., Groppe, D. M., Urbach, T. P., & Kutas, M.
Progress and future directions (2012). Thinking ahead or not? Natural aging and anticipa-
tion during reading. Brain Lang, 121, 226–239.
With particular attention to the thorny issue of pre- DeLong, K. A., Urbach, T. P., & Kutas, M. (2005). Probabi-
dictive processing, the experiments reviewed herein listic word pre-activation during language comprehension
kutas, federmeier, and urbach: the “negatives” and “positives” of prediction 655
Otten, M., Nieuwland, M. S., & van Berkum, J. J. A. Van Petten, C., & Luka, B. J. (2006). Neural localization of
(2007). Great expectations: Specific lexical anticipation semantic context effects in electromagnetic and hemody-
influences the processing of spoken language. BMC Neuro- namic studies. Brain Lang, 97, 279–293.
sci, 8, 89. Van Petten, C., & Luka, B. J. (2012). Prediction during lan-
Otten, M., & van Berkum, J. (2008). Discourse-based word guage comprehension: Benefits, costs, and ERP compo-
anticipation during language processing: Prediction or nents. Int J Psychophysiol, 83, 176–190.
priming? Discourse Process, 45, 464–496. Van Petten, C., Kutas, M., Kluender, R., Mitchiner, M., &
Otten, M., & van Berkum, J. J. A. (2009). Does working McIsaac, H. (1991). Fractionating the word repetition
memory capacity affect the ability to predict upcoming effect with event-related potentials. J Cogn Neurosci, 3,
words in discourse? Brain Res, 1291, 92–101. 131–150.
Pineda, J. A., Foote, S. L., & Neville, H. J. (1989). Effects Wicha, N. Y., Bates, E. A., Moreno, E. M., & Kutas, M.
of locus coeruleus lesions on auditory, long-latency, event- (2003). Potato not pope: Human brain potentials to gender
related potentials in monkey. J Neurosci, 9, 81–93. expectation and agreement in Spanish spoken sentences.
Rommers, J., Meyer, A. S., Praamstra, P., & Huettig, F. Neuro Let, 346(3), 165–168.
(2013). The contents of predictions in sentence compre- Wicha, N. Y., Moreno, E. M., & Kutas, M. (2003). Expecting
hension: Activation of the shape of objects before they are gender: An event-related brain potential study on the role
referred to. Neuropsychologia, 51, 437–447. of grammatical gender in comprehending a line drawing
Ruchkin, D. S., Johnson, R., Jr., Grafman, J., Canoune, H., within a written sentence in Spanish. Cortex, 39(3), 483–508.
& Ritter, W. (1992). Distinctions and similarities among Wicha, N. Y., Moreno, E. M., & Kutas, M. (2004). Anticipat-
working memory processes: An event-related potential ing words and their gender: An event-related brain poten-
study. Brain Res Cogn Brain Res, 1, 53–66. tial study of semantic integration, gender expectancy, and
Rugg, M. D. (1990). Event-related brain potentials dissociate gender agreement in Spanish sentence reading. J Cogn Neu-
repetition effects of high- and low-frequency words. Mem rosci, 16(7), 1272–1288.
Cognit, 18, 367–379. Wilding, E. L., & Ranganath, C. (2012). Electrophysiologi-
Schacter, D. L., Addis, D. R., & Buckner, R. L. (2007). cal correlates of episodic memory processes. In S. Luck &
Remembering the past to imagine the future: The prospec- E. Kappenman (Eds.), The Oxford handbook of event-related
tive brain. Nat Rev Neurosci, 8, 657–661. potential components (pp. 373–396). New York, NY: Oxford
Smith, N. J., & Levy, R. (2011). Cloze but no cigar: The University Press.
complex relationship between cloze, corpus, and subjective Wlotko, E., & Federmeier, K. D. (2011). Flexible implemen-
probabilities in language processing. Paper presented at tation of anticipatory language comprehension mecha-
the Proceedings of the 33rd Annual Meeting of the Cogni- nisms. J Cogn Neurosci, Suppl. 1, 233.
tive Sciences Conference, Austin, TX. Wlotko, E. W., & Federmeier, K. D. (2012a). Age-related
Swaab, T. Y., Ledoux, K., Camblin, C. C., & Boudewyn, M. A. changes in the impact of contextual strength on multiple
(2011). Language-related ERP components. In S. J. Luck & aspects of sentence comprehension. Psychophysiology, 49,
E. S. Kappenman (Eds.), The Oxford handbook of event-related 770–785.
potential components (pp. 397–440). New York, NY: Oxford Wlotko, E. W., & Federmeier, K. D. (2012b). So that’s what
University Press. you meant! Event-related potentials reveal multiple aspects
Szewczyk, J. M., & Schriefers, H. (2013). Prediction in of context use during construction of message-level
language comprehension beyond specific words: An ERP meaning. NeuroImage, 62, 356–366.
study on sentence comprehension in Polish. J Mem Lang, Wlotko, E. W., Federmeier, K. D., & Kutas, M. (2012). To
68, 297–314. predict or not to predict: Age-related differences in the use
Tulving, E., & Gold, C. (1963). Stimulus information and of sentential context. Psychol Aging, 27, 975–988.
contextual information as determinants of tachistoscopic Yu, A. J., & Dayan, P. (2005). Uncertainty, neuromodulation,
recognition of words. J Exp Psychol, 66, 319. and attention. Neuron, 46, 681–692.
van Berkum, J. J. A., Brown, C. M., Zwitserlood, P., Zacks, J. M., Speer, N. K., Swallow, K. M., Braver, T. S., &
Kooijman, V., & Hagoort, P. (2005). Anticipating upcom- Reynolds, J. R. (2007). Event perception: A mind-brain
ing words in discourse: Evidence from ERPs and reading perspective. Psychol Bull, 133, 273–293.
times. J Exp Psychol Learn Mem Cogn, 31, 443–467.
van Herten, M., Kolk, H. H., & Chwilla, D. J. (2005). An
ERP study of P600 effects elicited by semantic anomalies.
Brain Res Cogn Brain Res, 22, 241–255.
ABSTRACT The human ability to communicate information is capture the fundamental multimodal complexity of the
not exclusive to speech—meaning can also be conveyed by human language faculty. Spoken languages also exploit
the hands in sign languages and in co-speech gestures. Tradi- the visual-gestural modality for expression and use ges-
tionally, the two have been studied separately, partially because
each has very different properties (e.g., co-speech gesture is tures accompanying speech, with the hands, face, and
integrated with the speech channel; sign language is an inde- body as articulators. For example, speakers can move
pendent, hierarchical system). This chapter brings together the fingers of an inverted V-hand in a wiggling manner
for the first time findings on the neural underpinnings of while saying “he walked across” or use bodily demon-
signs and co-speech gestures. Sign language and co-speech strations of reported actions as they tell narratives. Ges-
gestures both recruit the left hemisphere for production and
involve the right hemisphere for representation of spatial
tures contribute semantic, syntactic, and pragmatic
events. Sign, co-speech gesture, and speech also implicate information to the verbal part of an utterance. To be
similar neural substrates for comprehension: left inferior able to understand the neural infrastructure of our
frontal cortex, posterior superior temporal cortex, and bilat- language faculty as a whole, we need to take into
eral middle temporal gyri, as well as similar electrophysiologi- account the composite multimodal utterances of spoken
cal responses to semantic violations (N400) and syntactic
languages—that is, both speech and co-speech gestures.
violations (P600). This review underscores the fact that the
brain’s general infrastructure for language and communica- Similarly, mapping the neural architecture for manual
tion treats meaningful information coming from all channels gestures that form a hierarchical linguistic system in
and formats in highly similar ways. sign languages is also necessary for a general account
of the neural bases of the human language faculty.
Meaning can be conveyed by the hands in a variety of Because language on the hands has very different
ways: through signing (used by deaf communities), pan- properties for sign languages and for co-speech gesture,
tomimed actions and emblematic gestures (often used we discuss their neural underpinnings in separate sec-
in the absence of speech), or through gestures pro- tions. When appropriate, we highlight relevant parallels
duced while speaking (i.e., gesticulation). This chapter and differences in each section and situate their neural
focuses primarily on the first and the last phenomena, processing in the context of what we know for spoken
which form the endpoints of the gesture continuum language.
(McNeill, 1992). As one moves from gesticulation
toward sign language, idiosyncratic gestures used with The neurobiology of sign language
speech are replaced by conventionalized/pantomimic
gestures, linguistic properties increase, and the obliga- Sign languages exploit visual-spatial contrasts at the
tory use of speech declines. Unlike co-speech gesture, phonological level (e.g., locations on the body consti-
sign languages are complete linguistic systems, exhibit- tute contrastive units of form), at the morphological
ing structural and language-specific constraints at the level (e.g., temporal aspect is marked by distinct pat-
phonological, lexical, syntactic, and discourse levels. terns of movement), at the syntactic level (e.g., gram-
Traditionally, the world’s languages have been matical roles are distinguished through the use of
grouped into two classes, signed and spoken languages, locations in signing space), and at the discourse level
based on the modality through which communicative (e.g., topics can be associated with spatial locations).
messages are transmitted: visual-manual versus audi- Despite this dependence on visual-spatial processing,
tory-vocal. However, in the last decade it has become the left, rather than the right, hemisphere is critical for
clear that this simple modality distinction does not producing and comprehending sign language. Damage
Figure 56.1 Illustration of left-hemisphere activations when (2000). (C) Greater activation for deaf signers than for
deaf signers comprehend sign language. (A) Signed sen- hearing nonsigners for pseudosigns from Emmorey, Xu, &
tences vs. nonsense signs from Neville et al. (1998). (B) Signs Braun (2011). (D) Signed sentences vs. a still image of the
and pseudosigns vs. fixation baseline from Petitto et al. signer from MacSweeney et al. (2002). (See color plate 49.)
ABSTRACT Linguistic expressions are often underdetermined A: Hey, I wonder who ate some of the chocolates I was going
with respect to the meaning that they convey. That is why to give to Anne?
context is needed to establish the message that is intended to
B: Oh, I heard the kids in the kitchen earlier.
be communicated by the linguistic expressions that are used.
Sometimes context information is explicitly provided, but From A’s utterance, one may infer that (the parentheti-
often it relies on implicit background knowledge shared cals indicate some of the different categories of prag-
between speaker and listener. The context of communication
needs to be invoked in understanding and producing linguis-
matic inference referred to later):
tic codes. What is required, in addition to the linguistic code (1) A wants to know who is responsible for eating
itself, to determine the meaning of an expression is usually
some of the chocolates (speech act conditions).
referred to as pragmatics. The neurobiology of pragmatics is
an area that has come under investigation only recently. Here (2) A thinks B might know the answer to his implied
we discuss central aspects of pragmatics. On the basis of question, knows about the existence of the chocolates,
recent functional MRI (fMRI) studies, it has become clear and so on (audience design: utterances are formulated
that the theory of mind network is involved in making the and adapted for specific listeners).
step from coded meaning to what is often called “speaker (3) A thinks that not all of the chocolates were eaten
meaning,” that is the intended message for the listener. Event-
related potential studies have investigated whether pragmatic (scalar implicature from the use of some).
processing is superimposed on and follows in time the com- Likewise, from B’s utterance in the context of A’s ques-
putation of coded meaning or, alternatively, whether context
tion much further information can be inferred:
information is immediately integrated with lexical, syntactic,
and semantic sources of information. On the whole, the evi- (4) B doesn’t know for sure who ate some of the
dence supports the immediacy assumption, suggesting that chocolates, but B is nevertheless trying to provide some
linguistic and nonlinguistic context information is immedi-
ately used for utterance interpretation. However, quite some
kind of answer (adjacency pair constraint: answers
individual variation is found in different aspects of pragmatic should follow questions; Grice’s maxim of relevance).
processing. (5) B knows or guesses that A thinks the chocolates
were in the kitchen (bridging inference, audience
Neurobiological models of language not only need to design).
address the circuitry that is crucial for encoding and (6) B is suggesting that perhaps the kids ate some of
decoding the content of an utterance, but they also the chocolates (indirect speech act, conversational
need to specify the neural infrastructure for inferring implicature), via additional premises like “kids tend to
what the speaker intended to communicate by uttering do naughty things” (common ground: implicit back-
a sentence (i.e., speaker meaning; Grice, 1989; Noveck ground knowledge shared between speakers and
& Reboul, 2008). This is what we broadly refer to as listeners).
neuropragmatics (Bambini, 2010; Bara, 2010). We will
first discuss what is covered by the term pragmatics and None of these inferences are logical deductions from
why it is crucial in an account of human communica- what is said. They can only be derived by making addi-
tion and language. This is followed by an overview of tional assumptions about the purposes and intentions
recent studies on the neural basis of pragmatics. with which we engage in verbal interaction and about
common understandings about how we should use lan-
Pragmatics guage. For example, other things being equal, we
assume that people will recognize our speech acts
Pragmatics is the study of the way in which context, (questions, requests, assertions, accusations, etc.) even
including the discourse, beliefs, and inferences of par- though they are often not coded directly—here A’s I
ticipants, contributes to the meanings of utterances wonder who is not syntactically a question, but rather
(see, e.g., Levinson, 1983; Sperber & Wilson, 1987). specifies a precondition for asking one (the basis for
The mechanisms involved are diverse, but they give rise inference 1 above), which A thinks should be enough
to substantial inferences that are not actually coded in for B to recognize the intention (inference 2 above). A
what is said. Consider the following interchange: doesn’t seem to think all the chocolates have been
meaning. Next to core areas for retrieving lexical infor- (Levinson, 2000) or, alternatively, are linked with pro-
mation from memory and unification of the lexical cessing effort and increased processing time, as is
building blocks in producing and understanding mul- claimed to follow from the relevance theory of Sperber
tiword utterances, other brain networks are needed to and Wilson (1995; cf. Noveck & Reboul, 2008). The
realize language-driven communication to its full area in which the alternative accounts are tested most
extent. explicitly is that of scalar implicatures. For example, the
quantifier some, although logically equivalent to all, is
pragmatically often interpreted as some but not all (see
The immediacy of pragmatic processing above). The speaker is assumed to use some for a reason,
which is to convey information that would not be pro-
A central issue of debate is to what extent pragmatic vided by all (Grice’s maxim of quantity). The question
inferences are generated automatically and by default is whether scalar implicatures are computed by default,
ABSTRACT Cultural evolution has emerged as a key source of that the cultural evolution of language provides a solu-
explanation for the emergence of complex linguistic struc- tion to this problem, indicating how we can explain the
ture in the human lineage. In this chapter, we argue that the
close fit between the structure of language and the
cultural evolution of language has been shaped by nonlinguis-
tic constraints deriving from the human brain. By analogy to mechanisms employed for acquiring and using lan-
reading, novel cortical networks for acquiring and using lan- guage. We then review recent proposals about neuronal
guage are suggested to have emerged through the cultural recycling and how they may provide a neural founda-
recycling of preexisting neural substrates. These language tion for the cultural evolution of language by analogy
networks inherited the structural properties and limitations with a human skill that we know is the product of cul-
of their component cortical circuits. In support for this per-
spective on the neurobiology of language, we discuss evidence tural evolution: reading. Finally, we discuss some of the
regarding the multifunction nature of Broca’s area—often implications for the neurobiology of language, by high-
considered to be a canonical language region—and the dis- lighting specific nonlinguistic neural substrates that
tributed nature of lexicosemantic representations. We con- provide the bases upon which language networks
clude by noting that more research is needed to explore how emerge during development.
the cultural evolution perspective may provide new insights
into the neurobiology of language.
A solution to the logical problem of language
Research on language evolution aims to answer some evolution: Language shaped by the brain
of the most fundamental questions about the nature of
our linguistic abilities: Why is language the way it is, and The acquisition of language is subject to a number of
how did it come to be that way? Fueled by theoretical biological, species-specific constraints. After all, only
constraints derived from recent advances in the brain humans have language; no other animal communica-
and cognitive sciences, the past couple of decades have tion system comes close to the complexity and diversity
seen an explosion of research on language evolution. of forms we see in human language (e.g., Evans &
This research was initially prompted by Pinker and Levinson, 2009). A key question is, however, whether
Bloom’s (1990) groundbreaking article arguing for the these biological constraints necessarily have to be spe-
natural selection of biological structures dedicated to cific to language or whether they may be broader in
language. The new millennium, however, has seen a nature, deriving from constraints on nonlinguistic
shift toward explaining language evolution in terms of neural mechanisms that have been pressed into use in
cultural evolution rather than biological adaptation. language.
Nonetheless, although the cultural evolution of lan- A long-standing influential approach is to assume
guage has had a substantial impact on the cognitive that language acquisition is constrained by a Universal
sciences, it has received relatively little attention within Grammar (UG): a genetic language-specific neural
cognitive neuroscience (though see, e.g., Arbib, 2010; system analogous to the visual system (e.g., Maynard-
Deacon, 1997, for exceptions). Smith & Szathmáry, 1997; Pinker, 1997). As such, UG
In this chapter, we outline how the cultural evolution provides a possible explanation for the close fit between
of language may be consistent with recent thinking the structure of language and how it is acquired and
about the cognitive neuroscience of language. First, we used. But the idea of linguistically driven biological
discuss the logical problem of language evolution faced adaptations as the origin of a genetically specified UG
by theories proposing biological adaptations for arbi- faces a logical problem of language evolution (Christiansen
trary features of language. As an alternative, we argue & Chater, 2008). UG is meant to characterize a set of
ABSTRACT Just over a decade ago, neurobiologists knew almost early 1990s, movement scholars had begun to turn
nothing about the neural mechanisms of voluntary choice. toward older economic theories like expected utility
Today, the basic outlines of the human and monkey systems
theory (von Neumann & Morgenstern, 1944) in an
for making decisions are now clearly beginning to emerge.
Speaking broadly, it is now known that a set of frontal cortical effort to understand this process. Unlike those studying
and basal ganglia store and represent the idiosyncratic values perceptual decision making who approached choice
we place on all kinds of things. This value system, a complex from the realm of sensation (Newsome, Britten, &
of many areas, synthesizes a single common representation of Movshon, 1989; Shadlen, Britten, Newsome, & Movshon,
the values of all of the many choice options that we face at 1996), these scholars found themselves trying to under-
any given moment. These so-called subjective values are then
sent to a set of frontal and parietal areas that actually perform stand the idiosyncratic “preferences” that guide human
the choice process. This chapter provides an outline of the decision making from a cognitive neuroscientific point
human and monkey value and choice systems. of view. This led them to search for hidden internal
representations that might guide decision-making pro-
cesses within the central nervous system.
Over the course of the past decade, enormous progress
Economists had a long history of developing theories
has been made toward understanding the basic mecha-
that specified the minimally complex internal represen-
nism by which the human brain makes choices. This
tation of preferences that could, in principle, account
review focuses on what is known about the mechanisms
for a given set of observed decisions (Houthakker, 1950;
we employ when choosing between goods or options
McFadden, 2005), and this group of cognitive neurosci-
that have different intrinsic values to us. This kind of
entists took this as a starting point (Platt & Glimcher,
decision making is known within neuroscientific circles
1999). They hypothesized that economic theories might
as value-based decision making. Contemporary studies
define the internal variables used by the nervous system
suggest that we can think of value-based decision
to guide choice. What resulted from that hypothesis was
making as reflecting two sequentially arranged neural
a new discipline at the borders of neuroscience, psy-
mechanisms: a valuation mechanism that learns, stores,
chology, and economics: neuroeconomics. This chapter
and retrieves the values of goods or actions under con-
reviews one of the central threads in that discipline,
sideration, and a choice mechanism that takes the output
which is often called the standard model.
of the valuation circuit as its starting point and gener-
ates a choice from among those options. This segrega-
tion of the decision-making system into two neat An overview of the standard model for value-based
components is part pedagogy and part reality (Padoa- decision making
Schioppa, 2011), but it serves as a starting point for
understanding how value-based choice arises in the The Choice Circuit We begin our overview by exam-
brain. ining the neural circuits that actually make choices,
Value-based studies of decision making in neurosci- once an internal representation of the values of the
ence emerged originally from studies of movement options under consideration has already been expressed
control. Scholars of movement control had long been with the nervous system. For an economist, it is natural
interested in how animals choose what movement to to assume that decision makers hold some “internal”
make: if I can look at (or reach toward) any point in representation of the subjective values of the goods or
extra-personal space, how do I select one movement actions under consideration. This internal representa-
from the set of all of those possible movements? By the tion, if it is to support logically consistent comparisons,
One interesting feature of the map is that the repre- largest peak, sometimes called a winner-take-all computa-
sentation of each of the three valuable movements is tion (Edelman & Keller, 1996; Maass, 2000; van Gisber-
quite broadly distributed. This reflects the fact that gen, van Opstal, & Tax, 1987).
adjacent neurons are strongly connected to one another How then does the growing, largest peak trigger the
and that the strength of that connection falls off as a selected movement? Some of these topographic maps
function of distance (Gilbert & Wiesel, 1979). A feature include a biophysical threshold that makes that possi-
not immediately obvious in this reconstruction is that ble. In the superior colliculus, which receives topo-
each neuron in the map is also connected in an inhibi- graphically mapped connections from LIP, when the
tory way to other neurons. The exact pattern of these firing rates of neurons at any one location on the map
inhibitory connections varies from area to area, but for experience firing rates over about 100 Hz, these neurons
our purposes let us consider these inhibitory connec- change state and burst at about 1,000 Hz for a roughly
tions to be universal; each neuron is connected with a fixed period of time (Hall & Moschovakis, 2004;
fixed level of inhibition to every other neuron in the van Opstal & van Gisbergen, 1989). This very high rate
topography (Haider, Hausser, & Carandini, 2013; Lee, completely suppresses all other movement-triggering
Helms, Augustine, & Hall, 1997). The peaks of activity, activity on that map, and then passes on to influence
thus, in some sense, “fight” with each other for control the movement-control circuits of the brainstem.
of the map. It is important to understand, however, that it is not
one small brain map that controls all eye movement–
How is the argmax operation performed? related decision making. A cascade of reciprocally
connected maps, as shown in figure 59.3, performs
It should be obvious that choosing, for a network like this function. The specializations of these maps are a
this, amounts to identifying the peak that is highest on subject of intense current inquiry and uncertainty, but
maps like the one portrayed in figure 59.2, and then we do know about a few key specializations. We know,
passing that information on to the circuits that generate for example, that the last of the maps in the eye-
movements. Studies of brain slices (Ozen, Helms, & movement cascade is the superior colliculus, and we
Hall, 2003) suggest that when the monkey chooses, the know that it is only this map on which the biophysical
strength of all of the short-range excitatory connections threshold mechanism operates (Hall & Moschovakis,
in the topographic map increases. As short-range exci- 2004).
tation grows over all of the map, the largest peak
becomes stronger and thus more effective at suppress- How does behavioral stochasticity arise?
ing its neighbors. As those neighbors are suppressed,
they become less effective at suppressing the largest We know that neuronal firing rates are in fact quite
peak. The result is a self-reinforcing growth of the stochastic. If we were to direct two inputs to the LIP
several important ways. The most important of these and the ventral striatum that combines the outputs of
ways is that subjective value signals always correlate with many antecedent brain areas onto a single common
and predict choice behavior—even when that choice scale appropriate for direct comparison and choice
behavior is incompatible with a utility-based theory; (Fehr & Rangel, 2011; Glimcher, 2011; Kable & Glimcher,
these neural signals appear to be much more closely 2009; and see Wallis & Rushworth, 2013, for an impor-
related to behavior than they are to traditional eco- tant extension of this approach). Brain activity in the
nomic theories. medial prefrontal cortex and striatum clearly reveals
the idiosyncratic values people place on goods, actions,
The medial prefrontal cortex and the striatum or rewards. One can predict how people will trade off
delays to future rewards, different kinds of rewards,
While some important debate about this point remains, social rewards, even co-operation from measurements
nearly all models of choice thus now posit the existence of activity in these two areas (Levy & Glimcher, 2012).
of a utility-like signal in the medial prefrontal cortex So it is natural to hypothesize that the choice circuits
62 OCHSNER 719
66 MENDE-SIEDLECKI AND
TODOROV 767
ABSTRACT The complex social and emotional cognition char- a foundation for this inquiry, describing the basic learn-
acteristic of human experience relies upon more basic forms ing mechanisms through which we assign negative value
of learning that enable us to assign affective value to stimuli.
to environmental stimuli, the means by which we
Our neuroscientific understanding of these associative learn-
ing processes has grown out of decades of detailed research control the expression of these learned associations,
in animal models using Pavlovian conditioning paradigms. In and the neurocircuitry underlying these processes.
this chapter, we begin by reviewing this foundational research, The survival of any organism depends on its ability to
describing the basic learning mechanisms through which we detect and evade environmental threat. Cues and con-
assign negative value to environmental stimuli. We then texts that signal danger must be learned rapidly so that
examine various means by which we control the expression
of these learned fear associations. We focus specifically on the serious threats can be anticipated with few actual
functional architecture underlying four distinct types of regu- encounters. These memories should also be long-
latory processes: extinction, active coping, cognitive emotion lasting. A perilous situation might be infrequent, but an
regulation, and reconsolidation. In each section, we review organism should not need to learn about a previous
what is known about the neurocircuitry of the regulatory threat anew. Across a range of species, the neurocir-
method from the nonhuman animal literature, as well as from
studies in humans.
cuitry governing fear learning represents an evolution-
arily conserved solution to these shared environmental
The growth of cognitive neuroscience over the last few demands, enabling the rapid formation of persistent
decades has brought a dramatic increase in the study fear memories. The expression of these learned fears
of affective and social cognition. The foundation of diverts the organism’s cognitive, behavioral, and physi-
much of this work lies in research in animal models, ological resources toward the detection and response
which has yielded a detailed neuroscientific account of to threat. While this recruitment of resources confers
the associative mechanisms through which we evaluate robust adaptive benefit in the face of danger, it also
and anticipate salient environmental events. Through means that fear expression in the absence of imminent
recent work examining more complex social and emo- threat is costly. In humans, excessive fear that persists
tional cognition, it has become increasingly clear that even in safe contexts is a cardinal feature of anxiety
these higher-order processes recruit a common under- disorders. Thus, the failure to regulate the expression
lying neural circuitry as these basic forms of evaluative of fear in accordance with the true presence of environ-
learning. On one hand, this convergence is intuitive. mental threat may be a critical factor underlying vulner-
Associative learning imbues our world with emotional ability to psychopathology.
significance, rendering objectively neutral stimuli In our daily lives, we flexibly control the expression
ominous or alluring, evoking thoughts and feelings, of fear in multiple ways. We may recognize that a stimu-
and motivating our actions and decisions. Much of this lus that previously predicted threat is perfectly safe in
emotional experience stems from social interaction. a different setting (e.g., hearing the growl of a bear
Conversely, social cognition is intrinsically emotional, while in the forest versus in the zoo). We may take
as our thoughts about other people are laden with actions to make a potentially unpleasant situation less
learned evaluations. On the other hand, these overlap- daunting (e.g., bringing a friend to a party where we
ping substrates of both basic and complex learning and will know few others). Sometimes simply changing the
behavior present a vexing puzzle, challenging scientists way we think about a dreaded future event is sufficient
to understand how these shared neural circuits give rise to disarm our fear (e.g., the vaccine shot will hurt, but
to the profound complexity of human social and emo- it’s quick and necessary to stay well). Finally, recent
tional experience. In this chapter, we will attempt to lay research suggests that experiences that violate our
hartley et al.: the cognitive neuroscience of fear and its control 697
negative expectations may, under certain conditions, predict an aversive unconditioned stimulus (US). In
directly overwrite this invalidated information. order for the CS to evoke a conditioned fear response,
In this chapter, we examine the functional architec- it must temporally coincide with the US, and it must
ture underlying the acquisition and control of fear. We carry information (i.e., reduce uncertainty) about
begin with a brief discussion of the neurocircuitry of the timing and likelihood of US delivery (Gallistel,
fear acquisition and then explore four types of regula- 2003; Pavlov, 1927; Rescorla, 1988). Through this learn-
tory processes that can be utilized to control fear: ing, a CS can evoke an array of behavioral and physio-
extinction, active coping, cognitive regulation, and logical reactions that comprise a preparatory response
reconsolidation. An overview of the basic circuitry dis- to predicted threat. In rodents, commonly studied con-
cussed is depicted in figure 60.1. ditioned responses (CRs) include freezing as well as
changes in heart rate, respiration, and potentiated startle
Fear acquisition responses. In humans, CRs include changes in skin con-
ductance, potentiated startle, and pupil dilation.
Humans and animals alike use environmental cues to Across species, the amygdala is a key locus for the
determine the presence or absence of danger. These acquisition, consolidation, and expression of Pavlovian
cues can acquire their emotional valence through a fear memory. In rodents, these mnemonic processes are
process of Pavlovian learning, which occurs when a pre- typically studied using auditory fear conditioning, in
viously neutral conditioned stimulus (CS) comes to which a tone CS comes to predict an electrical shock
Figure 60.1 Neurocircuitry of fear learning and regulation: connections are activated, inhibiting fear expression. Contex-
the lateral nucleus (LA) of the amygdala receives afferent tual modulation of extinction expression is mediated by
sensory information regarding the conditioned stimulus (CS) projections from the hippocampus to the vmPFC and/or
and unconditioned stimulus (US) relationship and is the site amygdala. During active coping (circuit denoted in blue),
of storage of the fear memory. The LA and basal nuclei (BA) information from the LA is routed not to the CE, which drives
are interconnected and both project to the central nucleus fear expression, but to the BA, which in turn projects to the
(CE), which has outputs to brainstem and hypothalamic striatum. The IL/vmPFC supports the learning active coping
regions that control the expression of the conditioned responses and mediates the subsequent inhibition of fear
response (CR). Following conditioning, the prelimbic (PL) expression outside of the instrumental learning context.
region of the ventromedial prefrontal cortex (vmPFC) is acti- During cognitive regulation, the dorsolateral prefrontal
vated following CS presentations and drives the expression of cortex (dlPFC) regulates fear expression through projections
conditioned fear. During fear-extinction learning and consoli- to the vmPFC, which in turn inhibits amygdala activity (circuit
dation, connections are established between the infralimbic denoted in green). Reconsolidation (denoted by red X)
(IL) subregion of the vmPFC and the inhibitory interca- diminishes conditioned fear expression through alteration of
lated (ITC) cell masses, which inhibit activity in the CE the original CS-US association stored in the LA. (See color
(circuit denoted in orange). During extinction recall, these plate 51.)
hartley et al.: the cognitive neuroscience of fear and its control 699
Pantazopoulos, Caldera, Pantazopoulos, & Paré, 2005; cortical gray matter in this region (Hartley, Fischl, &
Likhtik et al., 2008; Royer, Martina, & Paré, 1999). In Phelps, 2011; Milad et al., 2005) have been shown to
contrast to the IL, the neighboring prelimbic subregion correlate with fear extinction recall. BOLD activation
(PL) of mPFC opposes extinction, driving the expres- in the dorsal anterior cingulate cortex in fear-
sion of fear CRs (Corcoran & Quirk, 2007; Sierra- conditioning studies commonly increases in response
Mercado et al., 2011; Sotres-Bayon, Sierra-Mercado, to CS presentation. Cortical thickness in this region and
Pardilla-Delgado, & Quirk, 2012). During extinction CS-evoked BOLD activation correlate with fear expres-
training, CS-evoked responses in PL correlate with the sion (Milad, Quirk, et al., 2007), motivating the pro-
subsequent failure of extinction recall (Burgos-Robles, posal that the dorsal anterior cingulate cortex may
Vidal-Gonzalez, & Quirk, 2009), suggesting that the represent a human homologue of the rodent prelimbic
interplay between IL and PL cortical regions deter- region. As in the rodent, the human hippocampus
mines the success or failure of extinction learning. appears to play a critical role in the context-dependent
While cued fear extinction depends on the interplay expression of fear and extinction learning. Experimen-
of the mPFC and amygdala, information about whether tal paradigms in which fear and extinction learning are
the current context signals threat or safety is encoded associated with visually distinct contextual stimuli have
by the hippocampus, which can also mediate the observed hippocampal BOLD activation during con-
competition between functional fear inhibition and text-specific extinction retrieval (Kalisch et al., 2006;
excitation circuits to ensure that fear extinction is Milad, Wright, et al., 2007), and the magnitude of this
adaptively expressed (Maren, Phan, & Liberzon, 2013). signal change correlated positively with activation in the
The ventral hippocampus projects directly to inter- vmPFC (Milad, Wright, et al., 2007). The absence of
neurons in the PL, allowing inhibition of the principle context-dependent fear reinstatement in patients with
cells that promote conditioned fear expression (Sotres- hippocampal lesions (LaBar & Phelps, 2005) comports
Bayon et al., 2012). This region also directly activates with parallel findings in experimental lesion studies in
BA fear neurons (Herry et al., 2008). Thus the connec- rodents (Wilson, Brooks, & Bouton, 1995).
tivity of the hippocampus enables the facilitation of
either extinction learning or fear expression depend- Individual Variation and Clinical Implications of
ing on contextual information. The influence of Extinction Learning Extinction learning and reten-
context on fear expression is mediated by the dorsal tion are impaired in a number of anxiety-related disor-
hippocampus. Inactivation of the dorsal hippocampus ders (Graham & Milad, 2011; Lissek, Pine, & Grillon,
prevents the contextual renewal of extinguished fears 2006; Milad et al., 2009), suggesting that attenuated
in a novel context (Corcoran & Maren, 2004). Addition- fear extinction may confer psychiatric vulnerability.
ally, when fear conditioning, extinction, and recall However, the factors governing an individual’s ability to
occur in the same place, dorsal hippocampal inactiva- form and retain fear extinction memories are not
tion prior to recall prevents the fear recovery (Corcoran presently well understood. Extinction learning and
& Maren, 2004). Together, these data suggest that the retention appear to be relatively stable individual char-
dorsal hippocampus is sensitive to the relationship acteristics (Bush, Sotres-Bayon, & LeDoux, 2007;
between the current context and the predicted US, Fredrikson, Annas, Georgiades, Hursti, & Tersman,
driving the expression of learned fear in ambiguous 1993; Zeidan et al., 2012), and recent research has
environments. begun to clarify how individual variation in genetic
Functional neuroimaging studies in humans suggest background, sex, and developmental stage modulate
that the circuitry governing fear extinction in rodents fear extinction.
is largely conserved across species. The BOLD signal in Fear conditioning is substantially heritable (Hettema,
a subgenual region of the ventromedial prefrontal Annas, Neale, Kendler, & Fredrikson, 2003), likely
cortex (vmPFC), a putative homologue of the rodent reflecting the interactive contribution of multiple genes
infralimbic region, increases across the course of extinc- that influence fear learning and retrieval processes.
tion learning, whereas activation in the amygdala shows Convergent evidence in both humans and genetically
a corresponding decrease (Phelps, Delgado, Nearing, modified mice suggests that genetically mediated reduc-
& LeDoux, 2004). This inverse pattern of activation in tion in the expression of brain-derived neurotrophic
the vmPFC and amygdala is also observed when extinc- factor impairs fear extinction learning (Soliman et al.,
tion memories are retrieved (Kalisch et al., 2006; Milad, 2010), whereas reduction in the expression of the sero-
Wright, et al., 2007; Phelps et al., 2004), and both the tonin transporter selectively impairs extinction recall
magnitude of the vmPFC BOLD signal change (Milad, (Hartley et al., 2012; Wellman et al., 2007). Genetic
Wright et al., 2007) as well as the thickness of the variation in the dopaminergic (Lonsdorf et al., 2009;
hartley et al.: the cognitive neuroscience of fear and its control 701
reactive behavior with a proactive one that allows them suppress conditioned fear, this structure appears to be
to limit their contact with an aversive CS. EFF is divided a final common pathway for the inhibition of innate
into three phases: an initial CS-US fear-conditioning physiological and behavioral reactions that constitute
phase, an instrumental phase, in which the animal can fear expression.
inactivate the CS by performing a specific behavior Few studies in humans have probed the neural mech-
(e.g., rearing), and a test phase, in which a retrieval CS anisms underlying active coping. One recent functional
that cannot be instrumentally controlled is presented. MRI (fMRI) study in which subjects could avoid a cued
Animals that successfully learn to terminate the aversive shock by pressing a key observed heightened correla-
CS through instrumental action show far less freezing tion of activity within the amygdala and the ventral stria-
and more active behavior during the test compared to tum in the avoidance condition (Delgado et al., 2009),
yoked controls, for whom the unreinforced CSs experi- consistent with the role of this pathway in the shift
enced during the instrumental phase are equivalent to from passive to active responses to fear (Amorapanth,
an extinction session (Cain & LeDoux, 2007). Lesions LeDoux, & Nader, 2000). In another study in which
of LA and BA, but not CE, disrupt the acquisition of subjects had to avoid a virtual predator that would ulti-
EFF (Amorapanth et al., 2000). Instrumental learning mately deliver shock (Mobbs et al., 2009), activity in the
is thought to depend upon direct BA projections to the vmPFC increased during avoidance phase, consistent
striatum, a known substrate of goal-directed action with a role for the IL in supporting active avoidance in
(Cain & LeDoux, 2008). This is intriguing, because rodents (Moscarello & LeDoux, 2013). A recent behav-
while lesion or inactivation of LA and CE disrupt ioral study examining the effects of avoidance learning
fear conditioning (LeDoux, Cicchetti, Xagoraris, & on subsequent fear conditioning indicates that, as in
Romanski, 1990; LeDoux, Iwata, Cicchetti, & Reis, 1988; rodents (Baratta et al., 2007), experiences of instru-
Wilensky et al., 2006), pretraining lesions of BA have mental control can diminish the subsequent expression
no impact on Pavlovian learning (Anglaga-Figueroa & of conditioned fear (Hartley et al., 2013), suggesting a
Quirk, 2005). This double dissociation between the CE mechanism by which active coping may promote psy-
and BA suggests that instrumental and Pavlovian learn- chological resilience (Maier & Watkins, 2010). Collec-
ing are processed through distinct amygdalar pathways tively, this work suggests that in environments that
emanating out from LA. Inactivation of the CE pro- afford the ability to escape or disarm a present threat
motes active avoidance behavior in animals that previ- through action, inhibiting passive fear expression in
ously showed no evidence of instrumental learning favor of proactive coping responses can have lasting
(Lazaro-Munoz, LeDoux, & Cain, 2010), suggesting beneficial consequences.
that these nuclei may mediate a direct competition
between passive and active responses to threat. Cognitive regulation
Actively responding to an aversive CS also recruits the
infralimbic mPFC (IL) pathway, which is implicated in Studies of active coping investigate how overt, instru-
suppressing learned fear during extinction. Experi- mental behaviors can diminish fear expression.
ments using a signaled active-avoidance paradigm, in However, another means by which humans can actively
which an action performed during the CS prevents US control fear is to change their thoughts. Theories of
delivery, suggest that inactivation of the IL slows avoid- emotion highlight how the interpretation or appraisal
ance learning and increases the mutually incompatible of an event can influence the emotional response
conditioned freezing response. Crucially, active avoid- (Scherer, 2005). Although a few cognitive techniques
ance causes an IL-dependent attenuation of condi- have been shown to diminish fear, most studies examine
tioned freezing both in and out of the training strategies that emphasize reinterpreting the emotional
environment, suggesting that instrumental memory significance of the event (see Ochsner & Gross, 2008,
yields a generalizable inhibition of conditioned fear for a review and chapter 62 in this volume). By actively
(Moscarello & LeDoux, 2013). This lasting effect of manipulating this appraisal process, fear or negative
instrumental control is consistent with evidence from affect can be diminished.
alternative paradigms suggesting that active control In a typical study of cognitive emotion regulation,
over aversive situations yields plastic changes in the IL participants are presented negative emotional scenes
that diminish subsequent conditioned fear expression and asked to reinterpret the events depicted in the
(Baratta et al., 2007; Baratta et al., 2009) and buffer the scene in such a way as to reduce their negative affective
subsequent response to even uncontrollable stressors response, such as imagining that a scene depicting a
at later time (Amat et al., 2005, 2006). Because both bloody wound is fake. This type of active reappraisal has
active coping and extinction learning recruit the IL to been shown to be effective at reducing negative affect
hartley et al.: the cognitive neuroscience of fear and its control 703
all long-term memories go through a protein synthesis– in humans, its effects do not extend to multiple mea-
dependent consolidation period, not all instances of sures of conditioned fear (Kindt, Soeter, & Vervliet,
retrieval trigger reconsolidation. In rodent studies of 2009), suggesting it may target the expression of spe-
Pavlovian fear learning, retrieval involves the presenta- cific fear-output measures as opposed to the fear
tion of a reminder stimulus that reactivates the original memory itself (see Schiller & Phelps, 2011, and
memory. Reconsolidation only occurs if this stimulus Lonergan, Olivera-Gigueroa, Pitman, & Brunet, 2013,
somehow violates expectations, suggesting that this for reviews). However, introducing extinction training
process functions to introduce new information into after fear-memory retrieval in humans has been shown
the existing memory. If the retrieval stimulus in a CS-US to persistently diminish the expression of fear for up
pairing is identical to those presented during training, to a year (Schiller et al., 2009) and reduce subsequent
protein-synthesis inhibitors injected into LA have no amygdala BOLD responses (Agren et al., 2012). It is
effect on the later expression of fear CRs (Díaz-Mataix, proposed that, in contrast to standard extinction that
Ruiz Martinez, Schafe, LeDoux, & Doyère, 2013; Wang, inhibits the fear memory via the vmPFC, extinction
de Oliveira Alvares, & Nader, 2009)—suggesting that training during the reconsolidation window serves to
no reconsolidation occurred. However, if retrieval is update and modify the original fear memory. A recent
prompted by the CS or the US presented alone fMRI study compared BOLD responses during standard
(Díaz-Mataix, Debiec, LeDoux, & Doyère, 2011), or extinction training and extinction during reconsolida-
by a US presented at a novel time during the CS tion. Although the extinction procedure was identical,
(Debiec, Díaz-Mataix, Bush, Doyère, & Ledoux, 2010; extinction during reconsolidation failed to engage the
Díaz-Mataix et al., 2013; Wang et al., 2009), then pro- vmPFC and reduced amygdala-vmPFC connectivity,
tein-synthesis inhibitors injected into LA strongly dimin- relative to standard extinction (Schiller et al., 2013).
ish fear CRs during a subsequent test. These results are consistent with rodent studies suggest-
Reconsolidation may be triggered by a mismatch ing that timing extinction training to coincide with
between the reminder and the original training, but to reconsolidation induces plasticity in the LA that may
change the fear memory an additional manipulation of alter the original fear memory (Monfils et al., 2009;
reconsolidation is necessary. As outlined above, recon- Clem & Huganir, 2010), thus reducing the need to
solidation can be disrupted with a pharmacological inhibit it.
agent, but it also possible to change the memory by It is important to note that the strength of initial
introducing new information into the existing memory training can impose a temporary boundary condition
during reconsolidation. Research on reconsolidation- on reconsolidation. With strong training in rodents,
extinction boundaries demonstrates that the effective- presentation of the CS alone cannot trigger reconsoli-
ness of extinction is strongly increased if it occurs dation in the first week after learning, but can do so
during the reconsolidation window, when the memory one month later. In comparison, when initial training
is in a temporarily labile state. Following a single is weaker, the CS alone triggers reconsolidation just 24
retrieval CS with an extinction session 10 or 60 minutes hours after conditioning (Monfils et al., 2009; Wang
later leads to a more lasting reduction of conditioned et al., 2009). Intriguingly, reconsolidation can be trig-
fear than extinction sessions 6 or 24 hours after the gered 24 hours after strong training if the US occurs
retrieval CS, when the reconsolidation window is pre- at an earlier than expected time during the CS (Díaz-
sumably closed (Monfils, Cowansage, Klann, & LeDoux, Mataix et al., 2013). In order to be able to translate this
2009). Crucially, this manipulation is relatively insen- potentially exciting technique to clinical interventions,
sitive to renewal, reinstatement, and spontaneous future research will need to more fully illuminate the
recovery (Monfils et al., 2009). While reconsolidation- potential constraints of fear memory reconsolidation—
extinction boundaries are an area of open inquiry including the relationship between strength of training,
(for review, see Auber, Tedesco, Jones, Monfils, & the age of memory, and the precise mismatch between
Chiamulera, 2013), these data suggest that reconsolida- the reminder and learning.
tion can be used to reduce fear without the use of
potentially toxic drugs, and underscore the idea that old Conclusion
memories can be altered with a well-timed intervention.
In humans, the ability to persistently target fear mem- In spite of the tremendous advances in human neuro-
ories without toxic pharmacological manipulations is science techniques in the last 20 years, our ability to
crucial for its potential translation to clinical disorders. investigate the specificity of neural circuits in humans
Although there is some evidence that a relatively safe is still relatively limited. It is because of these limitations
drug, propranolol, can influence fear reconsolidation that exploring the similarities of neural circuits across
hartley et al.: the cognitive neuroscience of fear and its control 705
Corcoran, K. A., & Quirk, G. J. (2007). Activity in prelimbic Haaker, J., Gaburro, S., Sah, A., Gartmann, N., Lonsdorf,
cortex is necessary for the expression of learned, but not T. B., Meier, K., … Kalisch, R. (2013). Single dose of
innate, fears. J Neurosci, 27(4), 840–844. L-dopa makes extinction memories context-independent
Debiec, J., Díaz-Mataix, L., Bush, D. E., Doyère, V., & and prevents the return of fear. Proc Natl Acad Sci USA,
Ledoux, J. E. (2010). The amygdala encodes specific 110(26), 2428–2436.
sensory features of an aversive reinforcer. Nat Neurosci, Hartley, C. A., Gorun, A., Reddan, M. C., & Phelps, E. A.
13(5), 536–537. (2013). Stressor controllability modulates fear extinction in
Delgado, M. R. (2009). Avoiding negative outcomes: Track- humans. Neurobiol Learn Mem. Advance online publication.
ing the mechanisms of avoidance learning in humans doi:10.1016/j.nlm.2013.12.003
during fear conditioning. Front Behav Neurosci, 3, 1–9. Hartley, C. A., Fischl, B., & Phelps, E. A. (2011). Brain
Delgado, M. R., Nearing, K. I., LeDoux, J. E., & Phelps, E. structure correlates of individual differences in the acquisi-
A. (2008). Neural circuitry underlying the regulation of tion and inhibition of conditioned fear. Cereb Cortex, 21(9),
conditioned fear and its relation to extinction. Neuron, 59, 1954–1962.
829–838. Hartley, C. A., McKenna, M. C., Salman, R., Holmes, A.,
Díaz-Mataix, L., Debiec, J., LeDoux, J. E., & Doyère, V. Casey, B., Phelps, E. A., & Glatt, C. E. (2012). Serotonin
(2011). Sensory-specific associations stored in the lateral transporter polyadenylation polymorphism modulates the
amygdala allow for selective alteration of fear memories. J retention of fear extinction memory. Proc Natl Acad Sci USA,
Neurosci, 31(26), 9538–9543. 109(14), 5493–5498.
Díaz-Mataix, L., Ruiz Martinez, R. C., Schafe, G. E., Herry, C., Ciocchi, S., Senn, V., Demmou, L., Müller, C., &
LeDoux, J. E., & Doyère, V. (2013). Detection of a tempo- Lüthi, A. (2008). Switching on and off fear by distinct
ral error triggers reconsolidation of amygdala-dependent neuronal circuits. Nature, 454(7204), 600–606.
memories. Curr Biol, 23(6), 467–472. Herry, C., Ferraguti, F., Singewald, N., Letzkus, J. J.,
Fanselow, M. S., & Lester, L. S. (1988). A functional behav- Ehrlich, I., & Lüthi, A. (2010). Neuronal circuits of fear
ioristic approach to aversively motivated behavior: Preda- extinction. Eur J Neurosci, 31(4), 599–612.
tory imminence as a determinant of the topography of Hettema, J. M., Annas, P., Neale, M. C., Kendler, K. S., &
defensive behavior. In R. C. Bolles & M. D. Beecher (Eds.), Fredrikson, M. (2003). A twin study of the genetics of fear
Evolution and learning (pp. 185–212). Hillsdale, NJ: Erlbaum. conditioning. Arch Gen Psychiatry, 60(7), 702.
Foa, E. B. (2011). Prolonged exposure therapy: Past, present, Hofmann, S. G., Schulz, S. M., Meuret, A. E., Moscovitch,
and future. Depress Anxiety, 28(12), 1043–1047. D. A., & Suvak, M. (2006). Sudden gains during
Fredrikson, M., Annas, P., Georgiades, A., Hursti, T., & therapy of social phobia. J Consult Clin Psychol, 74(4),
Tersman, Z. (1993). Internal consistency and temporal sta- 687–697.
bility of classically conditioned skin conductance responses. Johansen, J. P., Cain, C. K., Ostroff, L. E., & LeDoux, J. E.
Biol Psychol, 35(2), 153–163. (2011). Molecular mechanisms of fear learning and
Gallistel, C. R. (2003). Conditioning from an information memory. Cell, 147(3), 509–524.
processing perspective. Behav Process, 62(1–3), 89–101. Kalisch, R., Korenfeld, E., Klaas, S., Weiskopf, N.,
Goosens, K. A., & Maren, S. (2001). Contextual and auditory Seymour, B., & Dolan, R. (2006). Context-dependent
fear conditioning are mediated by the lateral, basal, and human extinction memory is mediated by a ventromedial
central amygdaloid nuclei in rats. Learn Mem, 8(3), prefrontal and hippocampal network. J Neurosci, 26(37),
148–155. 9503–9511.
Gourley, S. L., Kedves, A. T., Olausson, P., & Taylor, J. R. Kim, J. H., & Richardson, R. (2007). A developmental dis-
(2009). A history of corticosterone exposure regulates fear sociation of context and GABA effects on extinguished fear
extinction and cortical NR2B, GluR2/3, and BDNF. Neuro- in rats. Behav Neurosci, 121(1), 131–139.
psychopharmacology, 34, 7077–7160. Kim, J. H., Hamlin, A. S., & Richardson, R. (2009). Fear
Graham, B. M., & Milad, M. R. (2011). The study of fear extinction across development: The involvement of the
extinction: Implications for anxiety disorders. Am J Psychia- medial prefrontal cortex as assessed by temporary inac-
try, 168(12), 1255–1265. tivation and immunohistochemistry. J Neurosci, 29(35),
Graham, B. M., & Richardson, R. (2011). Fibroblast growth 10802–10808.
factor-2 alters the nature of extinction. Learn Mem, 18(2), Kindt, M., Soeter, M., & Vervliet, B. (2009). Beyond extinc-
80–84. tion: Erasing human fear responses and preventing the
Grillon, C. (2009). D-cycloserine facilitation of fear extinc- return of fear. Nat Neurosci, 12(3), 256–258.
tion and exposure-based therapy might rely on lower- Koolhaas, J. M., Korte, S. M., de Boer, S. F., van der
level, automatic mechanisms. Biol Psychiatry, 66(7), Vegt, B. J., van Reenen, C. G., Hopster, H., … Blokhuis,
636–641. H. J. (1999). Coping styles in animals: Current status in
Guastella, A. J., Lovibond, P. F., Dadds, M. R., Mitchell, behavior and stress-physiology. Neurosci Biobehav Rev, 23,
P., & Richardson, R. (2007). A randomized controlled trial 925–935.
of the effect of D-cycloserine on extinction and fear condi- LaBar, K. S., Gatenby, C., Gore, J. C., LeDoux, J. E., &
tioning in humans. Behav Res Ther, 45(4), 663–672. Phelps, E. A. (1998). Human amygdala activation during
Gunduz-Cinar, O., Macpherson, K. P., Cinar, R., Gamble- conditioned fear acquisition and extinction: A mixed trial
George, J., Sugden, K., Williams, B., & Holmes, A. fMRI study. Neuron, 20, 937–945.
(2012). Convergent translational evidence of a role for LaBar, K. S., LeDoux, J. E., Spencer, D. D., & Phelps, E. A.
anandamide in amygdala-mediated fear extinction, threat (1995). Impaired fear conditioning following unilateral
processing and stress-reactivity. Mol Psychiatry, 18(7), temporal lobectomy in humans. J Neurosci, 15(10),
813–823. 6846–6855.
hartley et al.: the cognitive neuroscience of fear and its control 707
Pattwell, S. S., Duhoux, S., Hartley, C. A., Johnson, D. return of fear in humans using reconsolidation update
C., Jing, D., Elliott, M. D., … Lee, F. S. (2012). Altered mechanisms. Nature, 463(7277), 49–53.
fear learning across development in both mouse and Schiller, D., & Phelps, E. A. (2011). Does reconsolidation
human. Proc Natl Acad Sci USA, 109(40), 16318–16323. occur in humans? Front Behav Neuro, 5, 24.
Pavlov, I. P. (1927). Conditioned reflexes. London: Routledge Sierra-Mercado, D., Padilla-Coreano, N., & Quirk, G. J.
& Kegan Paul. (2011). Dissociable roles of prelimbic and infralimbic cor-
Phelps, E. A., Delgado, M. R., Nearing, K. I., & LeDoux, J. tices ventral hippocampus and basolateral amygdala in the
E. (2004). Extinction learning in humans: Role of the amyg- expression and extinction of conditioned fear. Neuropsycho-
dala and vmPFC. Neuron, 43(6), 897–905. pharmacology, 36(2), 529–538.
Phelps, E. A., & LeDoux, J. E. (2005). Contributions of the Soliman, F., Glatt, C. E., Bath, K. G., Levita, L., Jones, R.
amygdala to emotion processing: From animal models to M., Pattwell, S. S., … Casey, B. J. (2010). A genetic variant
human behavior. Neuron, 48(2), 175–187. BDNF polymorphism alters extinction learning in both
Pigott, T. A. (2003). Anxiety disorders in women. Psychiatr mouse and human. Science, 327(5967), 863–866.
Clin North Am, 26(3), 621–672. Sotres-Bayon, F., & Quirk, G. J. (2010). Prefrontal control
Quirk, G. J., Russo, G. K., Barron, J. L., & Lebron, K. of fear: More than just extinction. Curr Opin Neurobiol,
(2000). The role of ventromedial prefrontal cortex in the 20(2), 231–235.
recovery of extinguished fear. J Neurosci, 20, 6225–6231. Sotres-Bayon, F., Sierra-Mercado, D., Pardilla-Delgado,
Rabinak, C. A., Angstadt, M., Sripada, C. S., Abelson, J. L., E., & Quirk, G. J. (2012). Gating of fear in prelimbic cortex
Liberzon, I., Milad, M. R., & Phan, K. L. (2013). Canna- by hippocampal and amygdala inputs. Neuron, 76(4),
binoid facilitation of fear extinction memory recall in 804–812.
humans. Neuropharmacology, 64, 396–402. Urry, H. L., van Reekum, C. M., Johnstone, T., Kalin, N.
Raczka, K. A., Mechias, M. L., Gartmann, N., Reif, A., H., Thurow, M. E., Schaefer, H. S., … Davidson, R. J.
Deckert, J., Pessiglione, M., & Kalisch, R. (2011). (2006). Amygdala and ventromedial prefrontal cortex are
Empirical support for an involvement of the mesostriatal inversely coupled during regulation of negative affect and
dopamine system in human fear extinction. Transl Psychia- predict the diurnal pattern of cortisol secretion among
try, 1(6), e12. older adults. J Neurosci, 26(16), 4415–4425.
Raio, C. M., Brignoni-Perez, E., Goldman, R., & Phelps, E. Walker, D. L., Ressler, K. J., Lu, K. T., & Davis, M. (2002).
A. (2014). Acute stress impairs the retrieval of extinction Facilitation of conditioned fear extinction by systemic
memory in humans. Neurobiol Learn Mem. Advance online administration or intra-amygdala infusions of D-cycloserine
publication. doi:10.1016/j.nlm.2014.01.015 as assessed with fear-potentiated startle in rats. J Neurosci,
Raio, C. M., Orederu, T. A., Palazzolo, L., Shurick, A. A., 22(6), 2343–2351.
& Phelps, E. A. (2013). Cognitive emotion regulation Wang, S. H., de Oliveira Alvares, L., & Nader, K. (2009).
fails the stress test. Proc Natl Acad Sci USA, 110(37), Cellular and systems mechanisms of memory strength as a
15139–15144. constraint on auditory fear reconsolidation. Nat Neurosci,
Repa, J. C., Muller, J., Apergis, J., Desrochers, T. M., Zhou, 12(7), 905–912.
Y., & LeDoux, J. E. (2001). Two different lateral amygdala Wellman, C. L., Izquierdo, A., Garrett, J. E., Martin, K.
cell populations contribute to the initiation and storage of P., Carroll, J., Millstein, R., … Holmes, A. (2007).
memory. Nat Neurosci, 4(7), 724–731. Impaired stress-coping and fear extinction and abnormal
Rescorla, R. A. (1988). Behavioral studies of Pavlovian con- corticolimbic morphology in serotonin transporter knock-
ditioning. Annu Rev Neurosci, 11, 329–352. out mice. J Neurosci, 27(3), 684–691.
Ressler, K. J., Rothbaum, B. O., Tannenbaum, L., Wilensky, A. E., Schaffe, G. E., Kristense, M. P., & LeDoux,
Anderson, P., Graap, K., Zimand, E., … Davis, M. (2004). J. E. (2006). Rethinking the fear circuit: The central nucleus
Cognitive enhancers as adjuncts to psychotherapy: Use of of the amygdala is required for the acquisition, consolida-
D-cycloserine in phobic individuals to facilitate extinction tion and expression of Pavlovian fear conditioning. J Neuro-
of fear. Arch Gen Psychiatry, 61(11), 1136–1144. sci, 26, 12387–12396.
Romanski, L. M., Clugnet, M. C., Bordi, F., & LeDoux, J. Wilson, A., Brooks, D. C., & Bouton, M. E. (1995). The role
E. (1993). Somatosensory and auditory convergence in the of the rat hippocampal system in several effects of context
lateral nucleus of the amygdala. Behav Neurosci, 107(3), in extinction. Behav Neurosci, 109(5), 828–836.
444–450. Yap, C. S., & Richardson, R. (2007). Extinction in the devel-
Royer, S., Martina, M., & Paré, D. (1999). An inhibitory oping rat: An examination of renewal effects. Dev Psychobiol,
interface gates impulse traffic between the input and output 49(6), 565–575.
stations of the amygdala. J Neurosci, 19, 10575–10583. Zeidan, M. A., Igoe, S. A., Linnman, C., Vitalo, A., Levine,
Schafe, G. E., Doyère, V., & LeDoux, J. E. (2005). Tracking J. B., Klibanski, A., … Milad, M. R. (2011). Estradiol
the fear engram: The lateral amygdala is an essential locus modulates medial prefrontal cortex and amygdala activity
of fear memory storage. J Neurosci, 25(43), 10010–10014. during fear extinction in women and female rats. Biol Psy-
Scherer, K. R. (2005). What are emotions? And how can they chiatry, 70(10), 920–927.
be measured? Soc Sci Inform, 44(4), 695–729. Zeidan, M. A., Lebron-Milad, K., Thompson-Hollands, J.,
Schiller, D., Kanen, J. W., LeDoux, J. E., Monfils, M. H., Im, J. J., Dougherty, D. D., Holt, D. J., & Milad, M. R.
& Phelps, E. A. (2013). Extinction during reconsolidation (2012). Test–retest reliability during fear acquisition and
of threat memory diminishes prefrontal cortex involve- fear extinction in humans. CNS Neurosci Ther, 18(4),
ment. Proc Natl Acad Sci USA, 110(50), 20040–20045. 313–317.
Schiller, D., Monfils, M. H., Raio, C. M., Johnson, D. C.,
LeDoux, J. E., & Phelps, E. A. (2009). Preventing the
ABSTRACT The ability to flexibly alter behavior in the service overstated (Semendeferi, Lu, Schenker, & Damasio,
of future goals is one of the key evolutionary adaptations that 2002). Instead, it has been suggested that, rather than
has enabled humankind to flourish. Self-regulation refers to overall size, the human PFC demonstrated increased
a set of mental processes for overriding impulses, selectively
attending to goal-relevant information, and monitoring
white matter connectivity (i.e., Schoenemann, Sheehan,
thoughts and behavior for signs of failure. Although self- & Glotzer, 2005; although see Barton & Venditti, 2013).
regulation is of fundamental importance to an individual’s Regardless of whether the human PFC shows evi-
success, failures of self-regulation are common. In this chapter dence of specialized morphological enlargement, that
we focus on the role of exposure to tempting cues, negative humans possess a unique capacity for planning and
emotions, and limited cognitive resources in bringing about
self-regulation compared to all other animals appears
self-regulation failure. We review evidence from neuropsy-
chology and functional neuroimaging that successful self- irrefutable.
regulation depends on the interaction between brain Or does it? Just how unique is the human capacity for
structures in the prefrontal cortex involved in representing self-regulation? Comparative psychological research on
goals and directing attention away from goal-irrelevant stimuli nonhuman primates and other animals has generally
and cortical and subcortical structures involved in represent-
found that, with the exception of some domesticated
ing the value of rewards during decision making. When the
balance between these countervailing systems is disrupted as animals (e.g., Miller, Pattison, DeWall, Rayburn-Reeves,
a result of overwhelming impulses, negative affect, or defi- & Zentall, 2010), most nonhuman animals display
cient top-down control, self-regulation failure ensues. remarkably poor ability to inhibit prepotent responses
in order to obtain later, larger rewards (i.e., delay of
More than any other species, humans are especially gratification). For example, most nonhuman animals,
talented at controlling their own behavior in order to including many primates, will tolerate delays of only a
follow their goals and abide by rules and laws. Outside few seconds before consuming a desired food item
of certain psychiatric and neurological conditions, even (Green, Myerson, Holt, Slevin, & Estle, 2004; Ramseyer,
the most impulsive human is still leaps and bounds Pelé, Dufour, Chauvin, & Thierry, 2006). There are,
more capable of not blowing off work, of not eating however, some rare exceptions. For instance, great apes
everything in the pantry, or of not stealing their spouse’s and some species of birds will tolerate delays as long as
food than are the most precocious of nonhuman pri- a few minutes provided the expected reward greatly
mates. Despite humankind’s enormous advantage in exceeds the value of the currently available item (e.g.,
this domain, successful self-regulation remains difficult Beran, 2002; Dufour, Wascher, Braun, Miller, & Bugnyar,
and failures are common (Baumeister & Heatherton, 2012). Given the studies discussed above, it appears that
1996; Wagner & Heatherton, in press). How is it, then, outside of certain primates and species of birds, evi-
that humans are capable of inhibiting urges and pushing dence of self-regulation-like behavior is sparse among
aside temptations in order to pursue their goals? nonhuman animals. Why is it that humans have evolved
One commonly held view is that humans have evolved this complex capacity to self-regulate?
specific mental faculties that allow for superior plan-
ning and behavioral flexibility. These adaptations under- The importance of self-regulation for human
lie humans’ apparent superiority at self-regulation, and social groups
it has been hypothesized that they arose from a dispro-
portionate amount of cortical expansion of the prefron- Unlike many animals, humans display a prolonged
tal cortex (PFC) over the course of human evolution period of development and are unable to care for them-
(Rilling, 2006). Given the known role of the PFC in selves for the first decade of life. Being a member of a
self-control, it was reasonable to assume that the brain social group brings with it tremendous advantages,
would show specific structural changes to support these from sharing the burden of child rearing to cooperative
putative cognitive adaptations. However, recent work hunting and food sharing (Buss & Kenrick, 1998). As
suggests that relatively larger size of the PFC in humans human safety and survival has long depended on living
compared to nonhuman primates may have been in groups, it has been suggested that humans have a
ABSTRACT The question of how cognition—and cognitive Many Star Trek episodes depict the struggle between
control in particular—interacts with emotion has been of cognitively controlled Spock and emotionally impulsive
long-standing interest in popular culture and scientific
Kirk as they argue, cajole, and sometimes outright
inquiry. This chapter presents a fresh take on their interac-
tion, offered as a counterpoint to popular theories suggesting battle each other in a seesawing war to prove which
that emotional processes and higher-level cognitive processes, human faculty—cognitive control or emotion—is
such as those supporting self-control, are antagonists that supreme. This model of the cognition-emotion rela-
compete for control over behavior, with the rise of one leading tionship embodies a key assumption common to both
to the fall of the other. While such “seesaw” models can be lay and scientific theories: namely, that cognitive
very useful for explaining specific classes of phenomena, they
may not be able to account for the full range of interactions control is inherently antagonistic to emotion (see
between affective and cognitive processes of which humans figure 62.1).
are capable—and on which they depend on a daily basis. The Is this seesaw view of the cognitive control–emotion
alternative account of their relationship is motivated by a mix relationship sufficient to explain all possible ways in
of old-school appraisal theory and contemporary cognitive which control and emotion interact? If it is not, what
neuroscience data, whereby top-down cognitive control mech-
anisms play an integral role in at least four aspects of our
other kinds of models of their relationship might we
emotional lives: generating emotions, reporting on and consider?
understanding our own emotions, understanding the emo- The goal of this chapter is to begin addressing these
tions of others, and regulating our emotions. This account questions. Toward that end, it is divided into three
suggests that cognitive and affective processes are intimately parts. The first reviews a handful of salient exemplars
intertwined and provides a foundation for understanding
of the seesaw model, using them as foils for considering
core functional relationships among multiple kinds of behav-
ior and for asking new kinds of questions about them. what other kinds of cognition-emotion models might
be useful. The second presents research from my lab
If there is one thing humans are fond of it is dichoto- and others that suggests a particular alternative view of
mies. Up versus down. Left versus right. Good versus the control-emotion relationship. Here, we will con-
bad. Such dichotomies are simple and easy to under- sider examples of emotion generation, self-reports, per-
stand and apply. They typically have traction because ception, and regulation that either are not spoken to
they capture something important about the world. by seesaw models or are inconsistent with (at least
One dichotomy critically important for psychology is strong versions of) them. In the third and last section,
the distinction between cognition and emotion. we will consider one potential model that fits these data
Many lay and scientific accounts of their relationship and its implications for theoretical approaches to
focus on one form of cognition—cognitive control, or the understanding cognition, emotion, and future research
ability to engage in deliberate and reasoned thinking— on their relationship.
and the way in which it stifles emotional responses. This
relationship is epitomized by the relationship between Seesaw models of cognitive control and emotion
the two central characters of my favorite 1960s televi-
sion series: Star Trek. On one hand there is Spock, Historically, the idea that cognition and emotion seesaw
science officer on the Starship Enterprise. Spock is a back-and-forth with one another dates back at least to
Vulcan, and Vulcans have spent centuries eliminating Descartes’s ideas about the dichotomy between passion
base emotional impulses in favor of higher-order cogni- and reason (Damasio, 1994) and carry forward in con-
tive control and reasoning. On the other hand, there is temporary ideas about development (Somerville &
James T. Kirk, the all-too-human starship captain. Kirk Casey, 2010), moral judgment (Greene et al., 2001),
is the embodiment of impulsive emotion, driven as he and a range of other phenomena. For present
is by a pulsating libido and an overweening machismo. purposes, it is germane to review a few examples that
ABSTRACT Fears of the dark, sadness following peer rejection, mood disorders during this developmental window
and disappointment from losing a job are examples of emo- (Beesdo, Knappe, & Pine, 2009).
tions that we experience throughout our lives. The intensity
of these emotions can vary by age and across individuals. In
this chapter, we examine the development of two aspects of The emotional landscape of adolescence
emotional behavior: emotional reactivity and emotional regu-
lation. Emotional reactivity can be defined as the mental pro- Emotional reactivity and regulation are influenced by a
cesses that support the detection and assignment of salience number of factors—some transient and developmen-
to environmental inputs. Emotion regulation processes are tally regulated, others stable within an individual
those that would support a modification of an emotional
response most commonly assessed in terms of the reduction (Somerville, Jones, & Casey, 2010). A large body of
of an initial emotional response. Although these emotional psychological research has focused on whether differ-
processes shift in important ways from infancy to adulthood, ences in behavioral and emotional reactivity early in life
we focus on changes that take place during transitions into are indicative of stable, persistent, life-long emotional
and out of adolescence. reactivity patterns. Classic work by Ainsworth and
Bowlby on attachment (Ainsworth & Bowlby, 1991),
Kagan on behavioral inhibition (Kagan, Snidman,
A 2-year-old’s temper tantrum and an adolescent’s oscil- Arcus, & Reznick, 1994), Mischel on delay of gratifica-
lating mood swings illustrate how emotional processes tion (Mischel, Shoda, & Peake, 1988) and Rothbart
may change in important ways with development. (Rothbart, 2007) on temperament suggests that emotion
Through human development, we develop the ability reactivity and regulation can reflect trait-like tendencies
to filter the constant stream of environmental inputs early in life that statistically predict long-term mental
and determine which are important and which hold and physical health and illness.
emotional significance. Emotional responses to this These emotional tendencies vary by age and environ-
information affect our actions and how attention is mental factors. For example, increasing environmental
deployed to them. In this chapter, we present evidence demands (e.g., stress) placed on an individual can exac-
for developmental changes in emotional reactivity and erbate emotional reactivity, while the development of
emotional regulation. We provide examples of how regulatory abilities can help to quiet emotions (Mischel
positive and negative emotional cues engage cognitive & Baker, 1975; Posner & Rothbart, 2007). Environmen-
processes that support the detection, appraisal, and tal challenges and stressors are thought to reach a nor-
regulation of emotions. mative peak during adolescence, when the individual
Two terms used throughout this chapter are emo- transitions from dependence on the parent to relative
tional reactivity and emotion regulation. Emotion reac- independence while regulatory abilities are still devel-
tivity may be defined as the mental processes that oping—a period of life Hall described as ridden with
support the detection and assignment of salience to emotional upheaval, negative mood, and chronic stress
negative or positive information and our reaction to (Arnett, 1999; Hall, 1904).
that information. Emotion regulation processes are those The period of adolescence begins around the onset
that support a modification of an emotional response of puberty and ends ambiguously when an individual
(Gross & Thompson, 2009), most commonly assessed achieves adult-like levels of independence. The adoles-
in terms of the reduction of an initial emotional cent faces unique challenges with more independence
response. Although these emotional processes shift in than ever before, but also more responsibilities at home,
important ways from infancy to adulthood, we focus on school, and in social interactions (Larson, 2001; Spear,
behavioral and brain changes that take place during 2000). The social landscape takes on a heightened
adolescence, given the high prevalence of anxiety and importance, while individuals strive to identify with
somerville and casey: emotional reactivity and regulation across development 731
increasingly complex and salient social groups (Pfeifer termed ventral striatum in humans). These regions
& Peake, 2012; Somerville, 2013). Simultaneously, phys- play a critical role in reinforcement learning, integrat-
ical development mediated by hormonal changes of ing motivational signals, and facilitating instrumental
puberty results in widespread physiological shifts that behavior (Casey et al., 2000; Delgado, 2007; Pasupathy
can be unpredictable, intense, and stressful (Peper & & Miller, 2005; Schultz, Tremblay, & Hollerman, 1998).
Dahl, 2013; Sisk & Zehr, 2005). These changes are sup- The amygdala and ventral striatum are often charac-
ported by self-reports of more frequent, intense, and terized as “negative” and “positive” emotion centers,
rapid fluctuations in negative and positive emotions by respectively. However, the role of these structures in
adolescents relative to children or adults (Larson, emotional processing is not straightforwardly valence-
Moneta, Richards, & Wilson, 2002; Rutter, Graham, specific (Levita et al., 2009; Lindquist, Wager, Kober,
Chadwick, & Yule, 1976). Bliss-Moreau, & Barrett, 2012; Pagnoni, Zink, Montague,
The challenges faced by the adolescent are not spe- & Berns, 2002). Rather, the amygdala and ventral stria-
cific to humans but observed across species, as evi- tum are thought to play complementary roles in detect-
denced by similar increases in interactions with peers, ing and learning about salience based on associative
novelty seeking, and reward sensitivity during this properties and prediction error, respectively (Li, Schil-
developmental phase. These behaviors are thought to ler, Schoenbaum, Phelps, & Daw, 2011). Dense, hierar-
have evolved to serve adaptive functions (Spear & chical striatal-prefrontal cortical projections (“loops”)
Varlinskaya, 2010) related to successful mating and comprise reciprocal circuits that integrate motivational
attainment of resources necessary for survival. We high- signals with context-specific and regulatory demands to
light significant changes in brain maturation and func- ultimately guide motivated behavior (Haber, Kim,
tion that parallel these behavioral changes. Mailly, & Calzavara, 2006; Haber & Knutson, 2010).
somerville and casey: emotional reactivity and regulation across development 733
et al., 2012), and that the ventral striatum is significantly with complete abandonment. This control of emotions
more reactive to a wide range of emotional facial dis- is an important societal function, since emotions can
plays after puberty’s onset relative to before (Pfeifer disrupt other goal-directed activity through wide-ranging
et al., 2011). Together, these studies suggest that emo- mechanisms—from fundamental ways in which emotion
tional facial expressions are assigned higher salience in captures our attention (Vuilleumier, 2005), to the dis-
adolescents than in younger and older individuals, ruptive effect of high physiological arousal on cognitive
perhaps reflecting their social communication value performance (Yerkes & Dodson, 1908), to the sociocul-
during a time of life characterized by social reorienta- tural display rules that can motivate us to suppress the
tion (Crone & Dahl, 2012; Nelson, Leibenluft, McClure, expression of our true feelings (Gross & Levenson,
& Pine, 2005). 1993). As a result, human emotion is ubiquitously
Numerous studies have begun to examine emotion subject to regulation attempts that vary from implicit
reactivity in adolescents to the receipt or loss of mon- and automatic to intentional and effortful.
etary incentive or prizes. These studies show that the Emotion regulation is accomplished through interac-
magnitude (Ernst et al., 2005; Galvan et al., 2006; van tions between emotion circuitry and regions of the
Leijenhorst et al., 2010) and temporal sustainment brain that are capable of exerting regulation or control
(Fareri, Martin, & Delgado, 2008) of responses in the over emotional responses. Given the continuing devel-
ventral striatum are greater in adolescents than in chil- opment of cognitive control capabilities (de Luca et al.,
dren and adults (figure 63.1B). Importantly, this pattern 2003; Huizinga, Dolan, & van der Molen, 2006) through-
has been recently extended in human imaging to out youth, one could hypothesize that regulatory influ-
contexts of primary rewards such as juice (Galvan & ences operate with reduced efficiency during childhood
McGlennen, 2013), indicating that even fundamental and adolescence compared to adulthood.
rewards may elicit heightened value signaling during Here we focus on the developmental trajectories of
adolescence. However, like the amygdala findings, results three types of emotion regulation: forms of regulation
do not always support an account of adolescent striatal whereby participants must disengage from emotional
hypersensitivity (see Bjork et al., 2004, and Galvan, material in order to accomplish a secondary task, pro-
2010). cesses whereby emotional information is reappraised to
A complementary approach examines the extent to reduce its affective impact, and emotion regulation that
which participants’ neural activity reflects computa- is the result of associative learning.
tional learning signals that are thought to reflect dopa-
minergic neuronal activity (Schultz, Dayan, & Montague, Disengagement from Emotional Material The
1997). These studies examine how the expectations of need to disengage from emotional aspects of stimuli in
receiving a reward based on prior outcomes and its order to complete a secondary task is a form of emotion
value (e.g., low or high magnitude) impact future regulation that draws on cognitive and attention control
choices. Cohen and colleagues (2010) showed that resources. One task that has been used to elucidate
compared to children and adults, adolescents respond developmental emotion-cognition interactions is the
more quickly to stimuli that have been previously associ- emotional go/no-go task, where participants must
ated with a high (relative to low) reward value and show follow instructions to respond to (or withhold a response
greater ventral striatal activity to unexpected reward. to) stimuli that vary in appetitive or aversive salience.
This heightened positive prediction error and striatal Adolescents have more difficulty than children and
activity has been suggested as a possible mechanism for adults withholding a response toward an emotionally
increased risky decisions during adolescence. Alterna- arousing happy face relative to emotionally neutral
tively, van den Bos, Crone, and colleagues (van den Bos faces, suggesting that the appetitive salience associated
et al., 2011) have shown neural representation of pre- with happy faces selectively disrupts their impulse
diction errors to be similar across age. According to control (Somerville, Hare, & Casey, 2011). Hare and
their work, the functional connectivity between the colleagues demonstrated that adolescents are slower to
ventral striatum and prefrontal cortex is what is chang- respond to fearful faces (Hare et al., 2008), suggesting
ing as a function of age. As such, developmental changes that negative emotional content is more disruptive to
in behavior during adolescence may reflect how learn- their cognitive performance during adolescence as
ing signals guide behavior and expectations (van den well. The incidental presence of emotionally negative
Bos et al., 2009, 2011). images reduces young adolescents’ performance during
a go/no-go task more than in children or older indi-
Emotion Regulation in Adolescence In our daily viduals (Cohen-Gilbert & Thomas, 2013), likely by cap-
lives, we rarely allow ourselves to experience emotions turing attentional and cognitive resources (Grose-Fifer,
somerville and casey: emotional reactivity and regulation across development 735
this chapter contribute to our understanding of neural
responses to positive and negative emotional informa-
tion and how they may influence adolescent behavior
differentially from that of children and adults. These
findings are significant for informing the developmen-
tal trajectory of psychopathologies like anxiety and
mood disorders that emerge during this developmental
period.
During adolescence, we believe that unique patterns
of subcortical-cortical interactions impact emotional
processes at the psychological level. This developmental
neurobehavioral pattern underscores the conceptual-
ization of development as not only linear in nature, but
representing complex trajectories as well. Behaviorally,
emotional tendencies seem to shift during adolescence
in a way that sensitizes them to emotional information
differently than during younger or older ages. At the
neural level, nonlinear recruitment, structural changes,
and connectivity challenge assumptions that develop-
mental processes “fill in” slowly and steadily. This
pattern reflects the multitude of changes that take place
in adolescence, ranging from hormonal, physical, and
psychosocial that adolescents must navigate with less
“regulatory input” from authority figures. Tuning of
brain circuitry underlying emotion regulation reflects
Figure 63.2 A lack of extinction learning and retention of normative, experience-dependent maturation that
extinction memory is observed in adolescent humans (top) results from adolescents interacting with their environ-
and mice (bottom). P23, “child”; P29, “periadolescent.” (Repro- ments. Thus, the differences reported here should not
duced with permission from Pattwell et al., 2012.)
be pathologized or interpreted as adolescent deficien-
cies, but rather as normal developmental progressions,
comparable in magnitude across ages, suggesting that much like a child learning to walk.
conditioned learning is evident very early in life. There is a growing recognition that adolescence is a
However, the extent to which humans and rodents time of life associated with numerous preventable
express extinction learning varies nonlinearly across health risks that result from dysregulation of emotional
development. Specifically, adolescents show a reduction responses, including the deleterious outcomes of risk
in the expression of extinction learning relative to both taking, experimentation with drugs and alcohol, rela-
children and adults (figure 63.2; Pattwell et al., 2012). tional aggression, the emergence of mood anxiety dis-
In this way, adolescents do not seem to be able to capi- orders, and suicide, to name a few. Despite the
talize on learned extinction cues. Interestingly, this fundamental importance of this developmental phase,
effect has a nonlinear developmental course: younger the majority of neurobehavioral findings reported here
and older individuals show stronger extinction than have emerged within the past decade. The degree to
adolescents. which adolescent neurodevelopment research has
recently achieved recognition as an important, yet
Conclusion underspecified, topic of study within the emotion and
emotion regulation research disciplines is truly excit-
The work highlighted in this chapter illustrates the com- ing. With a greater volume of comprehensive and
plexity of emotional processes and their development. careful empirical research, this area will continue to
Increasing daily challenges associated with striving for advance toward the ultimate goal of translating neuro-
independence, paired with heightened signaling of sub- developmental findings in ways that improve adolescent
cortical regions during adolescence, can exacerbate mental and physical health.
emotions, while the development of regulatory abilities
supported by refinement of cortical connections can ACKNOWLEDGMENTS Supported by NIMH grants R00 MH087813
regulate emotions. Collectively, the studies presented in (LHS) and P50 MH 079513 (BJC).
somerville and casey: emotional reactivity and regulation across development 737
Larson, R. W. (2001). How U.S. children and adolescents science perspective on the process and its relation to psy-
spend time: What it does (and doesn’t) tell us about their chopathology. Psychol Med, 35, 163–174.
development. Curr Dir Psychol Sci, 10(4), 160–164. Ochsner, K. N., & Gross, J. J. (2005). The cognitive control
Larson, R. W., Moneta, G., Richards, M. H., & Wilson, S. of emotion. Trends Cogn Sci, 9(5), 242–249.
(2002). Continuity, stability, and change in daily emo- Pagnoni, G., Zink, C. F., Montague, P. R., & Berns, G. S.
tional experience across adolescence. Child Dev, 73(4), (2002). Activity in human ventral striatum locked to errors
1151–1165. of reward prediction. Nat Neurosci, 5(2), 97–98.
Laviola, G., Pascucci, T., & Pieretti, S. (2001). Striatal Pasupathy, A., & Miller, E. K. (2005). Different time courses
dopamine sensitization to D-amphetamine in periadoles- of learning-related activity in the prefrontal cortex and
cent but not adult rats. Pharmacol Biochem Behav, 68, striatum. Nature, 433, 873–876.
115–124. Pattwell, S. S., Duhoux, S., Hartley, C. A., Johnson,
LeDoux, J. E. (2000). Emotion circuits in the brain. Annu Rev D. C., Jing, D., Elliott, M. D., … Yang, R. R. (2012).
Neurosci, 23, 155–184. Altered fear learning across development in both mouse
Levita, L., Hare, T. A., Voss, H. U., Glover, G., Ballon, D. and human. Proc Natl Acad Sci USA, 109(40), 16318–16323.
J., & Casey, B. J. (2009). The bivalent side of the nucleus Peper, J. S., & Dahl, R. E. (2013). The teenage brain: Surging
accumbens. NeuroImage, 44(3), 1178–1187. hormones—brain-behavior interactions during puberty.
Li, J., Schiller, D., Schoenbaum, G., Phelps, E. A., & Curr Dir Psychol Sci, 22(2), 134–139.
Daw, N. D. (2011). Differential roles of human striatum Pfeifer, J. H., Masten, C. L., Moore, W. E., Oswald, T. M.,
and amygdala in associative learning. Nat Neurosci, 14(10), Mazziotta, J. C., Iacoboni, M., & Dapretto, M. (2011).
1250–1252. Entering adolescence: Resistance to peer influence, risky
Lindquist, K. A., Wager, T. D., Kober, H., Bliss-Moreau, behavior, and neural changes in emotion reactivity. Neuron,
E., & Barrett, L. F. (2012). The brain basis of emotion: A 69(5), 1029–1036.
meta-analytic review. Behav Brain Sci, 35(3), 121–143. Pfeifer, J. H., & Peake, S. J. (2012). Self-development: Inte-
Luna, B., Garver, K. E., Urban, T. A., Lazar, N. A., & grating cognitive, socioemotional, and neuroimaging per-
Sweeney, J. A. (2004). Maturation of cognitive processes spectives. Dev Cogn Neurosci, 2(1), 55–69.
from late childhood to adulthood. Child Dev, 75(5), Phelps, E. A., Delgado, M. R., Nearing, K. I., & LeDoux, J.
1357–1372. E. (2004). Extinction learning in humans: Role of the amyg-
Luna, B., Padmanabhan, A., & O’Hearn, K. (2010). What dala and vmPFC. Neuron, 43(6), 897–905.
has fMRI told us about the development of cognitive Phelps, E. A., & LeDoux, J. E. (2005). Contributions of the
control through adolescence? Brain Cogn, 72(1), 101–113. amygdala to emotion processing: From animal models to
Maren, S., & Quirk, G. J. (2004). Neuronal signalling of fear human behavior. Neuron, 48(2), 175–187.
memory. Nat Rev Neurosci, 5(11), 844–852. Phillips, R. G., & LeDoux, J. E. (1992). Differential contribu-
McClure, E. B., Monk, C. S., Nelson, E. E., Zarahn, E., tion of the amygdala and hippocampus to cued and con-
Leibenluft, E., Bilder, R. M., … Pine, D. S. (2004). A textual fear conditioning. Behav Neurosci, 106, 274–285.
developmental examination of gender differences in brain Pine, D. S., Grun, J., Zarahn, E., Fyer, A., Koda, V., Li, W.,
engagement during evaluation of threat. Biol Psychiatry, … Bilder, R. M. (2001). Cortical brain regions engaged by
55(11), 1047–1055. masked emotional faces in adolescents and adults: An fMRI
McRae, K., Gross, J. J., Weber, J., Robertson, E. R., study. Emotion, 1(2), 137–147.
Sokol-Hessner, P., Ray, R. D., … Ochsner, K. N. (2012). Posner, M. I., & Rothbart, M. K. (2007). Research on atten-
The development of emotion regulation: An fMRI study of tion networks as a model for the integration of psychologi-
cognitive reappraisal in children, adolescents and young cal science. Annu Rev Psychol, 58, 1–23.
adults. Soc Cogn Affect Neurosci, 7(1), 11–22. Rothbart, M. K. (2007). Temperament, development, and
Milad, M. R., & Quirk, G. J. (2002). Neurons in medial pre- personality. Curr Dir Psychol Sci, 16(4), 207–212.
frontal cortex signal memory for fear extinction. Nature, Rubia, K., Smith, A. B., Woolley, J., Nosarti, C., Heyman,
420(6911), 70–74. I., Taylor, E., & Brammer, M. J. (2006). Progressive
Mischel, W., & Baker, N. (1975). Cognitive transformations increase of frontostriatal brain activation from childhood
of reward objects through instructions. J Pers Soc Psychol, to adulthood during event-related tasks of cognitive control.
31(2), 254–261. Hum Brain Mapp, 27, 973–993.
Mischel, W., Shoda, Y., & Peake, P. K. (1988). The nature of Rutter, M., Graham, P., Chadwick, O. F. D., & Yule, W.
adolescent competencies predicted by preschool delay of (1976). Adolescent turmoil: Fact or fiction? J Child Psychol
gratification. J Pers Soc Psychol, 54(4), 687. Psychiatry, 17, 35–56.
Monk, C. S., McClure, E. B., Nelson, E. E., Zarahn, E., Schafe, G. E., Nader, K., Blair, H. T., & LeDoux, J. E.
Bilder, R. M., Leibenluft, E., & Pine, D. S. (2003). (2001). Memory consolidation of Pavlovian fear condition-
Adolescent immaturity in attention-related brain engage- ing: A cellular and molecular perspective. Trends Neurosci,
ment to emotional facial expressions. NeuroImage, 20, 24(9), 540–546.
420–428. Schultz, W., Dayan, P., & Montague, P. R. (1997). A neural
Moore, W. E., Pfeifer, J. H., Masten, C. L., Mazziotta, substrate of prediction and reward. Science, 275(5306),
J. C., Iacoboni, M., & Dapretto, M. (2012). Facing puberty: 1593–1599.
Associations between pubertal development and neural Schultz, W., Tremblay, L., & Hollerman, J. R. (1998).
responses to affective facial displays. Soc Cogn Affect Neurosci, Reward prediction in primate basal ganglia and frontal
7(1), 35–43. cortex. Neuropharmacology, 37(4–5), 421–429.
Nelson, E. E., Leibenluft, E., McClure, E. B., & Pine, D. S. Silvers, J. A., McRae, K., Gabrieli, J. D. E., Gross, J. J.,
(2005). The social re-orientation of adolescence: A neuro- Remy, K. A., & Ochsner, K. N. (2012). Age-related differ-
somerville and casey: emotional reactivity and regulation across development 739
64 The Amygdala: Relations to Biologically
Relevant Learning and Development
DYLAN G. GEE AND PAUL J. WHALEN
ABSTRACT The amygdala plays a critical role in learning about contributions to biologically relevant learning during
cues in the environment that predict biologically relevant classical conditioning (see Davis & Whalen, 2001), neu-
outcomes and informing our behavioral responses in anticipa- roimaging studies have documented a largely similar
tion of these outcomes. Consistent with classic principles of
learning theory, the amygdala is particularly responsive to
role for the human amygdala. For example, functional
uncertainty, which enhances learning. Much of the extant neuroimaging studies have demonstrated contributions
research on the human amygdala has employed facial expres- of the amygdala during classical conditioning para-
sions to elucidate its contributions to emotional learning. In digms with a variety of CS-US contingencies, including
addition to their role in nonverbal communication, facial colored squares predicting electric shock (LaBar,
expressions can be considered conditioned stimuli based on
Gatenby, Gore, LeDoux, & Phelps, 1998).
their reinforcement history in prior social situations. Through
interactions between the amygdala and prefrontal cortex These findings demonstrate in a clear manner that
(PFC), bottom-up and top-down processing shape this social the human amygdala is involved in learning about envi-
learning. Stronger amygdala-prefrontal connectivity begets ronmental cues that predict biologically relevant out-
better behavioral outcomes, and disrupted cross-talk between comes as well as informing behavioral responses in
these regions underlies emotion dysregulation in healthy and
anticipation of these outcomes (LeDoux & Schiller,
clinical populations. Moreover, the amygdala and its connec-
tions with PFC undergo dynamic changes throughout devel- 2009). Research in humans has extended this initial
opment, which likely contribute to developmental changes in work to show that the amygdala might function in a
emotional behavior. Given the neurodevelopmental nature of more general manner, facilitating all kinds of biologi-
many disorders and the widespread implication of amygdala- cally relevant learning. For example, the amygdala is
prefrontal circuitry in psychopathology, understanding how responsive to both aversive and appetitive CSs (Paton,
the amygdala changes in typical development and possible
disruptions in biologically relevant learning are both funda- Belova, Morrison, & Salzman, 2006). Data such as these
mental to treating psychopathology. The following chapter fit well with attentional hypotheses (Kapp, Whalen,
will detail how the amygdala contributes to biologically rele- Supple, & Pascoe, 1992) and seminal work demonstrat-
vant learning and gives rise to individual differences in emo- ing a critical role for the amygdala in associative changes
tional behavior, as well as how these processes change with observed in appetitive reward paradigms in the rat
development and risk for psychopathology.
(Gallagher & Holland, 1994). That is, amygdala activa-
tion to a tone that predicts shock is but a part of an
affective information-processing system that directs our
The amygdala and biologically relevant learning
attention to important events in the environment. The
The amygdala plays a central role in learning about amygdala functions as an orienting subsystem for the
biologically relevant events, which can signal the rela- rest of the brain, which allows us to respond appropri-
tive safety or danger of a given environment. In the ately to those events (Davis & Whalen, 2001; Gallagher
framework of classical conditioning, the amygdala & Holland, 1994). In alerting other systems at times
responds to biologically relevant events that require no when it would be advantageous to gather information
previous learning (i.e., unconditioned stimuli, or USs, (i.e., learn), the amygdala supports adaptive functions
that are inherently emotionally salient, such as a shock), that cross the categorical boundaries of constructs
as well as the events that predict these biologically rel- such as motivation, emotion, vigilance, attention, and
evant events (i.e., conditioned stimuli, or CSs, which cognition.
begin as neutral but take on emotional salience through
conditioning). This form of learning depends on the The Role of Uncertainty in Biologically Rele-
amygdala in both humans and nonhuman animals (see vant Learning Derived from principles of learning
LeDoux, 1996). While nonhuman animal studies still theory, the idea that there is more to learn in the face
comprise the majority of our knowledge about amygdala of uncertainty is fundamental to how the amygdala
between the amygdala and the PFC corresponds to which involves the suppression of a previously learned
one’s ability to regulate one’s emotions in this way. pairing between a CS and US (Quirk, 2002; Rescorla,
In a study of emotion regulation, functional connec- 2001). The inhibition of CS-US associations depends
tivity between the amygdala and mPFC increased during critically on top-down regulatory input from the mPFC
reappraisal, and the strength of this connectivity was to the basolateral amygdala (Milad & Quirk, 2002). For
associated with participants’ self-report of how effective example, in rodents, electric stimulation of the mPFC
they were at regulating their emotions (Banks et al., resulted in the inhibition of conditioned responses. In
2007). A selective increase in the functional coupling humans, increased vmPFC activation and structural
of the amygdala with the vmPFC and dorsolateral PFC volume have been associated with successful extinction
during emotion regulation has also been reported (Erk (Hartley, Fischl, & Phelps, 2011; Phelps, Delgado,
et al., 2010). Similarly, numerous studies have demon- Nearing, & LeDoux, 2004). These results suggest that
strated increased prefrontal activity and concomitant emotion regulation and extinction rely on overlapping
decreased amygdala activity during successful emotion neural mechanisms (Delgado, Nearing, LeDoux, &
regulation (e.g., Ochsner et al., 2002; Wager, Davidson, Phelps, 2008), consistent with the fact that both pro-
Hughes, Lindquist, & Ochsner, 2008), highlighting the cesses involve reevaluating biologically relevant stimuli
importance of amygdala-prefrontal interactions in suc- (Quirk & Beer, 2006). These structural and functional
cessful top-down control of emotion. Supporting the findings highlight the importance of amygdala-mPFC
idea that stronger amygdala-prefrontal connectivity interactions for the regulation and inhibition necessary
begets better emotion regulation, stronger amygdala- for extinction learning and memory.
mPFC functional coupling has also been associated with Amygdala-prefrontal interactions further support the
greater amygdala habituation (Hare et al., 2008) and resolution of ambiguity in our environment. Unlike
lower anxiety (e.g., Gee et al., 2013; Pezawas et al., fearful faces, surprised faces do not predict the valence
2005). Although correlational measures cannot inform of an unknown eliciting event such that they can be
the directional nature of regional influences, negative subjectively interpreted as either positive or negative
(inverse) amygdala-prefrontal coupling has been theo- (Neta, Norris, & Whalen, 2009). Individual differences
rized to reflect top-down regulation. in valence judgments of surprised faces correspond to
Functional interactions between the amygdala and distinct patterns of brain activity involving the amygdala
mPFC also support the extinction of fear conditioning, and vmPFC (Kim, Somerville, Johnstone, Alexander, &
negative coupling and reduced amygdala reactivity with facilitate adaptive behavior such as stronger emotion
age may provide a neurobiological basis for develop- regulation, changes in amygdala-prefrontal circuitry
mental improvements in emotion regulation. may also render individuals vulnerable in certain stages
of development or with risk for psychopathology. The
Conclusion coming decades will be an exciting time for research
on affective neuroscience, as we better understand how
Emotion, whether typical or in the context of psycho- the amygdala interacts with a broader brain circuitry to
pathology, can be conceived as a constant interplay give rise to what we currently call emotion. This line of
between an organism’s reactions to biologically relevant research will play a critical role in detailing how amyg-
stimuli (bottom-up processing) and its attempts to mod- dala circuitry facilitates adaptive behavior, contributes
ulate these responses (top-down processing). A wealth to developmental changes, and influences risk for
of animal and human studies demonstrate the central psychopathology.
role of the amygdala and its connections with PFC in
these processes. Across the life span, amygdala function
and connectivity undergo dynamic changes, which REFERENCES
likely support changes in learning and social behavior Adolphs, R., Tranel, D., Damasio, H., & Damasio, A. (1994).
that are fundamental to development. For example, Impaired recognition of emotion in facial expressions fol-
lowing bilateral damage to the human amygdala. Nature,
learning about the relative safety or danger of a given
372(6507), 669–672.
environment plays a critical role in key developmental Adolphs, R., Tranel, D., Damasio, H., & Damasio, A. R.
transitions, such as when the organism leaves the nest (1995). Fear and the human amygdala. J Neurosci, 15(9),
or enters adolescence. Moreover, it is through amygdala 5879–5891.
contributions in development that faces take on predic- Aggleton, J. P., Burton, M. J., & Passingham, R. E. (1980).
tive value through experiences in the social world, Cortical and subcortical afferents to the amygdala of the
rhesus monkey (Macaca mulatta). Brain Res, 190(2), 347–368.
building the reinforcement history that gives rise to Amaral, D. G., & Price, J. L. (1984). Amygdalo-cortical pro-
individual differences in neural and behavioral jections in the monkey (Macaca fascicularis). J Comp Neurol,
responses. While these neurodevelopmental changes 230(4), 465–496.
ABSTRACT Accumulating evidence suggests that social cogni- Theories of emotion: Form and function
tive processing and emotion are integrally intertwined. This
link encompasses early sensory processes and high-order, con- Darwin (1872) was the first scientist to speculate about
ceptual processing, and likely stems from the adaptive, func-
tional role that emotions fulfill in regulating behavior toward the importance of emotional expression in humans. He
biologically relevant goals. In this chapter, we discuss the link suggested that emotional expression serves a functional
between sensory and socioemotional processes from converg- role in regulating behavior in response to salient envi-
ing perspectives, including shared neural substrates empha- ronmental events and in signaling important social
sizing critical roles of the amygdala, ventromedial prefrontal information to others. More recently, Shariff and Tracy
cortex, and insular cortex; commonalities in connection with
chemosensory, visual, and motor systems; and bidirectional
(2011) suggest that early in our evolutionary lineage,
interactions of perceptual and socioemotional processes. We emotions served as cues (e.g., for action) but were later
discuss the evolutionary origins of socioemotional systems, exploited as signals (e.g., communicating one’s emo-
and emphasize their function in biasing behavior, suggesting tional state to others). Theoretical classifications of the
that complex emotional and social cognition neurobehavioral range of human emotion impact and shape our under-
programs originate and continue to share a deep functional
standing of the functional role of emotional processes
integration with simpler basic sensory and perceptual
processes. in the brain. Many models of emotion focus on the
form or content of emotion experiences and emphasize
the importance of classifying distinct subjective emotion
Traditional models of cognition view higher processes experiences (e.g., Ekman, 1999).
such as reasoning and decision making as separate Other researchers argue that our best chance at
from the workings of dedicated affective neural understanding emotion experience will be borne of
systems, but growing evidence supports the notion of attempts to understand their adaptive functions.
an integral link between affective and cognitive pro- LeDoux (2012) argues for conceptualizing the range of
cesses (e.g., for reviews, see Pessoa, 2009; Phelps, phenomena typically under the umbrella of emotions
2006). Of particular relevance in the present chapter is based on how the underlying physiological processes
the relationship between emotion and social cogni- serve the pursuit of motivational drives for optimizing
tion. Social processes and behavior appear to be heavily survival. He writes: “By focusing on the subjective state,
influenced by emotional states and abilities, and emo- emotion theories tend to gloss over the underlying
tional experiences may serve an important function as details of emotional processing for the sake of converg-
social signals (Adolphs, 2002, 2003). In this chapter, we ing on a single word that symbolizes diverse underlying
review the neuroanatomy of socioemotional systems, states mediated by different kinds of circuits” (LeDoux,
and in particular discuss the relationship between early 2012, p. 655). For instance, aggression may be moti-
sensory processes and higher-order knowledge regard- vated by an array of underlying motivational drives,
ing social and emotional information. We emphasize including personal survival and drives to reproduce. In
the importance of sensory and motor processes in each case, the same emotional experience might be
understanding information in socioemotional contexts induced by different, adaptively functional motivations.
and argue that these lower-level processes are critical By adopting this framework, LeDoux abstracts away
in shaping our understanding of emotional and social from the notion of basic emotions in favor of examining
events in the world. Rather than acting as mere way the function of sensory and motor processes in regulat-
stations for the relaying of sensory inputs and outputs ing behavior to optimize survival.
from higher-level neural processing, we argue that per- Farb, Chapman, and Anderson (2013) also argue that
ception and action are integral to socioemotional focusing on the function of emotions will be more
processing. informative for understanding the neural processes
ABSTRACT Historically, the amygdala’s role in face processing Single-unit recordings in nonhuman primates indicate
has been relegated to the domain of emotion, while a network that populations of neurons in IT cortex and superior
of posterior cortical regions is typically considered the core temporal sulcus (STS) respond selectively to faces, as
network involved in face processing. In this chapter, we
examine several decades of neurophysiological and neuroim-
compared to comparably complex object categories
aging work suggesting that the amygdala possesses face- (Bruce, Desimone, & Gross, 1981; Desimone, 1991;
selective properties, which can be observed in the absence of Desimone, Albright, Gross, & Bruce, 1984; Perrett,
emotional expressions. We also review a recent large-scale Rolls, & Caan, 1982; Perrett et al., 1984). These results
group analysis of functional MRI data from 215 participants have been widely and robustly replicated (Eifuku, de
in face-localizer tasks observing robust responses to faces in
Souza, Tamura, Nishijo, & Ono, 2004; Hasselmo, Rolls,
the amygdala, which were reliable over time and identifiable
in a majority of participants. Taken together, these data & Baylis, 1989; Kiani, Esteky, & Tanaka, 2005; Tsao,
suggest that the amygdala deserves consideration as a key Freiwald, Knutsen, Mandeville, & Tootell, 2003), gener-
node in the core face-processing network. alized to human subjects (Ojemann, Ojemann, &
Lettich, 1992), and confirmed in combination with
Successful navigation of our social universe requires an functional magnetic resonance imaging (fMRI; Tsao,
understanding of other social agents. Harmonious Freiwald, Tootell, & Livingston, 2006). More recent
interaction depends on accurate inferences regarding work has refined and revised theories on the neural
a rich compendium of social information—identity, architecture of face perception, suggesting that there
race, gender, age, emotions, mental states, and person- may be multiple face-selective patches (Freiwald & Tsao,
ality, just to name a few. Much of this information can 2012; Rajimehr, Young, & Tootell, 2009; Weiner & Grill-
be inferred from the human face. Behavioral research Spector, 2010, 2012). However, contemporaneous
shows that we are able to rapidly extract information research using identical methodology suggests that the
regarding identity (Grill-Spector & Kanwisher, 2005; Yip neural bases of face processing are not limited to pos-
& Sinha, 2002), race and gender (Cloutier, Mason, & terior cortical areas.
Macrae, 2005; Martin & Macrae, 2007), emotional
expression (Esteves & Öhman, 1993; Whalen et al.,
1998), and attractiveness (Locher, Unger, Sociedade, & Single-unit recordings of face-selective amygdala
Wahl, 1993; Olson & Marshuetz, 2005), and even make responses
inferences about personality such as aggressiveness
(Bar, Neta, & Linz, 2006) and trustworthiness (Willis & In 1979, Sanghera and colleagues set out to examine
Todorov, 2006; Todorov, Pakrashi, & Oosterhof, 2009) whether neurons in the amygdala respond to visual
based upon minimal exposure to faces. Moreover, these input, citing previous anatomical work describing pro-
inferences and evaluations have behavioral conse- jections from IT cortex to the amygdala (Herzog &
quences (Todorov, Mende-Siedlecki, & Dotsch, 2013). Hoesen, 1976; Jones & Powell, 1970). Indeed, of 1,754
Given the importance of faces, it is not surprising that neurons analyzed, 113 units in the lateral aspect of the
they have been the subject of an extensive study in amygdala were deemed responsive to visual stimuli.
psychology and cognitive neuroscience (Calder, Rhodes, Moreover, in the last paragraph of the paper, the authors
Johnson, & Haxby, 2011). wrote, “It should also perhaps be noted that nine of the
The initial groundwork for research on the neural amygdaloid neurons with visual responses were found
bases of face perception was laid in the 1970s and early to respond primarily to faces or photographs of faces”
1980s by researchers mapping the visual properties of (Sanghera, Rolls, & Roper-Hall, 1979, p. 624). Over the
inferior temporal (IT) cortex (e.g., Gross, Rocha- next 30 years, research confirming these initial findings
Miranda, & Bender, 1972; see also Gross, 1994, 2008). has steadily accumulated, leading to the conclusion that
mende-siedlecki and todorov: the role of the amygdala in face processing 767
there are neurons within the amygdala that display Neuroimaging evidence of the amygdala’s
robust selectivity for faces. face-selective properties
To begin with, a number of additional single-unit
recording studies of nonhuman primates have repli- There is a lack of complete convergence between the
cated Sanghera and colleagues’ findings (Leonard, single-unit recording research and human neuroimag-
Rolls, Wilson, & Baylis, 1985; Nakamura, Mikami, & ing work, with respect to which brain areas are ulti-
Kubota, 1992; Rolls, 1984), with later studies examining mately classified as face-selective. Functional MRI
the effects of novelty (Wilson & Rolls, 1993) as well as studies of human face perception have classically relied
identity and expression (Gothard, Battaglia, Erickson, upon paradigms employing functional localizers
Spitler, & Amaral, 2007) on face-selective responses (Kanwisher, McDermott, & Chun, 1997; McCarthy,
in the amygdala. Subsequently, these findings were Puce, Gore, & Allison, 1997), which allow researchers
corroborated by high-resolution fMRI work with non- to compare between blood oxygen level–dependent, or
human primates (Hoffman, Gothard, Schmid, & Logo- BOLD, responses to faces and to comparatively complex
thetis, 2007; Logothetis, Guggenberger, Peled, & Pauls, objects (e.g., houses). In this way, localizer studies and
1999), and generalized to humans in neurophysiologi- single-unit recording studies share the same fundamen-
cal work with patients undergoing treatment for epi- tal logic—subjects are presented with stimuli from
lepsy (Fried, MacDonald, & Wilson, 1997; Fried, various categories (e.g., faces, body parts, objects,
Cameron, Yashar, Fong, & Morrow, 2002; Kreiman, scrambled faces, etc.), and subsequently, researchers
Koch, & Fried, 2000; Mormann et al., 2008; Quiroga, test for units (areas of the brain or neurons, respec-
Reddy, Kreiman, Koch, & Fried, 2005; Rutishauser et tively) that respond preferentially to a given category.
al., 2011; Seeck et al., 1995; Steinmetz, 2008; Viskontas, Despite relatively similar approaches in terms of tasks
Quiroga, & Fried, 2009; see also Sato et al., 2012, for and stimuli, neuroimaging studies employing func-
similar findings using intracranial field potentials). tional localizers consistently identify only regions in
Across the majority of this work, several patterns posterior cortical areas, including the fusiform face
have emerged. First, based on single-unit recording area (FFA; Kanwisher et al., 1997; McCarthy et al., 1997;
work in both humans and nonhuman primates, it is Tong, Nakayama, Moscovitch, Weinrib, & Kanwisher,
clear that only a subset of neurons within the amygdala 2000), occipital face area (OFA; Gauthier, Skudlarski,
is truly selective for faces. A conservative estimate of Gore, & Anderson, 2000; Puce, Allison, Asgari, Gore, &
the number of face-selective amygdala neurons among McCarthy, 1996), and posterior superior temporal
the visually responsive neurons would likely lie some- sulcus (pSTS; Allison, Puce, & McCarthy, 2000; Puce
where between 5% and 10% (see Todorov, 2012), et al., 1996). Taken together, the FFA, OFA, and pSTS
though some authors have suggested that as many are considered to comprise a network of core regions
as 20% might display selectivity for whole faces involved in the perception of faces (Haxby & Gobbini,
(Rutishauser et al., 2011). Second, face-selective 2012; Haxby, Hoffman, & Gobbini, 2000; Said, Haxby,
responses in the amygdala are qualitatively different & Todorov, 2011).
from those in posterior cortical regions on at least a However, face-localizer studies do not typically clas-
few metrics. Direct comparison between recordings in sify the amygdala as a face-selective region. When con-
amygdala and STS suggest that face-selective responses sidered in the context of regions involved in face
in the amygdala have longer response latencies than perception, the amygdala is occasionally referenced as
those in STS and, moreover, generalize somewhat less a supporting node in the extended face network
across faces than those in STS (Leonard et al., 1985). (Haxby et al., 2000; Haxby & Gobbini, 2012), tasked
Third, face-selective neurons in the amygdala are less with processing emotional expressions. However, a
clustered, as compared to those in IT cortex and STS. number of human neuroimaging experiments have
Finally, it seems that no single facial characteristic or reported increased amygdala response to neutral
quality (e.g., species specificity, emotional expression, faces (for example, Dubois et al., 1999; Fitzgerald,
identity, familiarity, etc.) can completely account for Angstadt, Jelsone, Nathan, & Phan, 2006; Johnstone
the amygdala’s responses to faces. For example, et al., 2005; Kesler-West et al., 2001; van der Gaag,
Leonard and colleagues observed that only four face- Minderaa, & Keysers, 2007; Wright & Liu, 2006).
selective neurons in the amygdala (out of 19 tested) While some researchers have argued that the
responded preferentially to faces with emotional “extended” aspects of the face-processing network
expressions, while Gothard and colleagues observed deserve more attention and prominence (Ishai,
that 64% of face-selective amygdala neurons responded 2008), this perspective has faced its share of criticism
to both emotional expression and identity. (Wiggett & Downing, 2008).
mende-siedlecki and todorov: the role of the amygdala in face processing 769
Figure 66.1 Group analysis of functional localizer data from inference analysis, showing activation across the NeuroSynth
215 participants, with NeuroSynth validation. Panels A database for the term “face.” These maps indicate the consis-
through C depict brain regions showing greater BOLD tency of activation for the term. Panels G through I show the
responses to face vs. control stimuli (p < .05, corrected), results of a reverse inference analysis. These maps indicate
including bilateral amygdala. We used NeuroSynth—a meta- the relative selectivity of activation for the term (see Yarkoni
analytic tool capable of synthesizing results from a database et al., 2011, for details). (This figure originally appeared in
of neuroimaging studies—to externally validate these data. Mende-Siedlecki, Verosky, Turk-Browne, & Todorov, 2013.)
Panels D through F represent the results of a forward (See color plate 55.)
bilateral amygdala, and right pSTS. Next, for each ROI amygdala, right FFA, left FFA, right pSTS, and SC. A
and for each participant, we extracted voxel-by-voxel principle components analysis identified two distinct
parameter estimates from the Faces > Controls con- sources of variance in the reliability of face-selective
trast, for both runs of the localizer task, and computed responses. On one hand, cortical regions (right and left
the Pearson correlation between the two runs. On FFA, right dlPFC) loaded strongly onto the first princi-
average, across the 119 multirun participants, face- pal component, while on the other hand, subcortical
selective responses in the amygdala were reliable over regions (right and left amygdala, SC) loaded strongly
time, and this reliability was significantly greater onto the second principal component. (We note that
than zero; the average cross-run reliability was also the right pSTS loaded equally onto both principal com-
computed for an anatomically defined control region ponents, with relatively weak strength.) These first two
in A1. components accounted for approximately 50% of the
Moving forward, we sought to explore systematic total variance.
covariances of these multivoxel reliability correlations Finally, we performed a series of psychophysical inter-
across the network of face-selective ROIs. We observed action (PPI) analyses aimed at isolating face-specific
that the right amygdala and right FFA were most highly patterns of functional connectivity between our face-
intercorrelated with those of other ROIs, based upon selective ROIs and other brain areas. We placed seeds
correlations of cross-run reliability scores. Specifically, in right and left amygdala and observed enhanced con-
reliability in right amygdala correlated with left nectivity with bilateral fusiform, primary visual cortex,
mende-siedlecki and todorov: the role of the amygdala in face processing 771
Behrmann, 2013; Catani, Jones, Donato, & Ffytche, et al., 1985; Nakamura et al., 1992; Sanghera et al.,
2003; Gschwind, Pourtois, Schwartz, van de Ville, & 1979), the face-selective regions observed in posterior
Vuilleumier, 2012; Pugliese et al., 2009). To exert its cortical areas seem to be much more homogenously
excitatory effects on sensory regions in the cortex, the composed of face-selective neurons (Tsao et al., 2006).
amygdala transmits through the nucleus basalis of A better understanding of the fundamental differences
Meynert, which maintains direct cholinergic projec- between cortical and subcortical aspects is critical
tions to the cortex (Kapp, Whalen, Supple, & Pascoe, for building a comprehensive neural model of face
1992; Whalen, 1998). Taken together, this profile of processing.
connectivity suggests that the amygdala is uniquely Convergent findings across multiple methods, para-
suited to rapidly process faces that are in some way digms, and species populations have robustly identified
behaviorally relevant and subsequently decrease the responses in the amygdala that are preferential for
sensory threshold in cortical regions (in this case, faces. Although additional investigations are needed to
primary visual areas), accelerating more fine-grained characterize the exact nature and purpose of the amyg-
aspects of face processing. dala’s contributions to face processing, there is ample
In identifying the amygdala’s precise contributions to evidence to suggest that the amygdala deserves consid-
face processing, future work must inevitably confront eration as a core face-selective region. The populations
additional open questions—with likely interrelated of face-selective neurons found within the amygdala are
answers. For example, standard fMRI’s spatial resolu- tasked with detecting and processing one of the most
tion makes it difficult to determine which amygdala important stimuli we social creatures can perceive—the
nuclei are responsible for the face-selective responses face.
we observed. The amygdala comprises many individual
subnuclei with varying functionality and connectivity REFERENCES
(Aggleton, 2000; Freese & Amaral, 2009), and with
Adolphs, R. (2010). What does the amygdala contribute to
dissociable response patterns in the context of face
social cognition? Ann NY Acad Sci, 1191, 42–61.
processing (Hoffman et al., 2007; Kim, Somerville, Aggleton, J. P. (2000). The amygdala: A functional analysis.
Johnstone, Alexander, & Whalen, 2003; Whalen et al., Oxford, UK: Oxford University Press.
2001). Diffusion tensor imaging may potentially provide Allison, T., Puce, A., & McCarthy, G. (2000). Social percep-
a more exact specification of the source of face-selective tion from visual cues: Role of the STS region. Trends Cogn
responses in the amygdala (Bach, Behrens, Garrido, Sci, 4, 267–278.
Amaral, D. G., Behniea, H., & Kelly, J. L. (2003). Topo-
Weiskop, & Dolan, 2011; Saygin, Osher, Augustinack, graphic organization of projections from the amygdala to
Fischl, & Gabrieli, 2011). Moreover, a precise account the visual cortex in the macaque monkey. J Neurosci, 118,
of which amygdala subnuclei contribute to face process- 1099–1120.
ing may help to reconcile the present data with work Amaral, D. G., Price, J. L., Pitkänen, A., & Carmichael,
on the amygdala’s role in processing emotional expres- S. T. (1992). Anatomical organization of the primate amyg-
daloid complex. In J. P. Aggleton (Ed.), The amygdala: Neu-
sions (Morris et al., 1996; Whalen et al., 1998; Winston, robiological aspects of emotion, memory, and mental dysfunction
O’Doherty, & Dolan, 2003; Yang et al., 2002) and social (pp. 1–66). New York, NY: Wiley-Liss.
evaluations of faces (Said, Baron, & Todorov, 2009; Avidan, G., Hadj-Bouziane, F., Liu, N., Ungerleider, L., &
Said, Dotsch, & Todorov, 2010; Todorov & Engell, 2008; Behrmann, M. (2013). Selective dissociation between core
Todorov, Said, Oosterhof, & Engell, 2011; Winston, and extended regions in the face processing network in
congenital prosopagnosia. Cereb Cortex. Advance online
O’Doherty, Kilner, Perrett, & Dolan, 2007).
publication. doi:10.1093/cercor/bht007
Furthermore, future work must assess the degree to Bach, D. R., Behrens, T. E., Garrido, L., Weiskop, N., &
which face-selective responses in the amygdala differ Dolan, R. J. (2011). Deep and superficial amygdala nuclei
from those in posterior cortical regions like FFA and projections revealed in vivo by probabilistic tractography.
pSTS. While our data categorize the amygdala as face- J Neurosci, 31, 618–623.
Bar, M., Neta, M., & Linz, H. (2006). Very first impressions.
selective based on the same metrics and criteria
Emotion, 6, 269–278.
by which we would classify the FFA or pSTS as face- Bruce, C., Desimone, R., & Gross, C. G. (1981). Visual prop-
selective, the patterns of neuronal selectivity underlying erties of neurons in a polysensory area in superior temporal
these responses may differ considerably. The primate sulcus of the macaque. J Neurophysiol, 46, 369–384.
literature provides evidence for this divergence. While Calder, A. J., Rhodes, G., Johnson, M. H., & Haxby, J. V.
face-selective neurons in the amygdala are likely to be (2011). The Oxford handbook of face perception. Oxford, UK:
Oxford University Press.
found intermixed with other neurons that do not Catani, M., Jones, D. K., Donato, R., & Ffytche, D. H.
display face-selective properties and, moreover, may not (2003). Occipito-temporal connections in the human
even be selective for any class of visual stimuli (Leonard brain. Brain, 126, 2093–2107.
mende-siedlecki and todorov: the role of the amygdala in face processing 773
to primate and non-primate faces. J Neurophysiol, 94, responsive to faces in the temporal cortex: Studies of func-
1587–1596. tional organization, sensitivity to identity and relation to
Kim, H., Somerville, L. H., Johnstone, T., Alexander, A., perception. Hum Neurobiol, 3, 197–208.
& Whalen, P. J. (2003). Inverse amygdala and medial pre- Pessoa, L. (2005). To what extent are emotional stimuli pro-
frontal cortex responses to surprised faces. NeuroReport, 14, cessed without attention and awareness? Curr Opin Neuro-
2317–2322. biol, 15, 188–196.
Kreiman, G., Koch, C., & Fried, I. (2000). Category-specific Puce, A., Allison, T., Asgari, M., Gore, J. C., & McCarthy,
visual responses of single neurons in the human medial G. (1996). Differential sensitivity of human visual cortex to
temporal lobe. Nat Neurosci, 3, 946–953. faces, letterstrings, and textures: A functional magnetic
LaBar, K. S., Gitelman, D. R., Mesulam, M. M., & Parrish, resonance imaging study. J Neurosci, 16, 5205–5215.
T. B. (2001). Impact of signal-to-noise on functional MRI Pugliese, L., Catani, M., Ameis, S., Dell’Acqua, F., Thie-
of the human amygdala. NeuroReport, 12, 3461–3464. baut de Schotten, M., Murphy, C., … Murphy, D. G.
Leonard, C. M., Rolls, E. T., Wilson, F. A. W., & Baylis, (2009). The anatomy of extended limbic pathways in
G. C. (1985). Neurons in the amygdala of the monkey Asperger syndrome: A preliminary diffusion tensor imaging
with responses selective for faces. Behav Brain Res, 15, tractography study. NeuroImage, 47, 427–434.
159–176. Quiroga, R. Q., Reddy, L., Kreiman, G., Koch, C., & Fried,
Lerner, Y., Singer, N., Gonen, T., Weintraub, Y., Cohen, O., I. (2005). Invariant visual representation by single neurons
Rubin, N., & Ungerleider, L. G. (2012). Feeling without in the human brain. Nature, 435, 1102–1107.
seeing? Engagement of ventral, but not dorsal, amygdala Rajimehr, R., Young, J. C., & Tootell, R. B. H. (2009).
during unaware exposure to emotional faces. J Cogn Neuro- An anterior temporal face patch in human cortex, pre-
sci, 24, 531–542. dicted by macaque maps. Proc Natl Acad Sci USA, 106,
Locher, P., Unger, R., Sociedade, P., & Wahl, J. (1993). At 1995–2000.
first glance: Accessibility of the physical attractiveness ste- Rolls, E. T. (1984). Neurons in the cortex of the temporal
reotype. Sex Roles, 28, 729–743. lobe and in the amygdala of the monkey with responses
Logothetis, N. K., Guggenberger, H., Peled, S., & Pauls, selective for faces. Hum Neurobiol, 3, 209–222.
J. (1999). Functional imaging of the monkey brain. Nat Rossion, B., Hanseeuw, B., & Dricot, L. (2012). Defining
Neurosci, 2, 555–562. face perception areas in the human brain: A large-scale
Martin, D., & Macrae, C. N. (2007). A face with a cue: factorial fMRI face localizer analysis. Brain Cogn, 79,
Exploring the inevitability of person categorization. Eur 138–157.
J Soc Psychol, 37, 806–816. Rutishauser, U., Tudusciuc, O., Neumann, D., Mamelak,
McCarthy, G., Puce, A., Gore, J. C., & Allison, T. (1997). A. N., Heller, A. C., Ross, I. B., … Adolphs, R. (2011).
Face-specific processing in the human fusiform gyrus. Single-unit responses selective for whole faces in the human
J Cogn Neurosci, 9, 605–610. amygdala. Curr Biol, 21, 1654–1660.
Mende-Siedlecki, P., Verosky, S. C., Turk-Browne, N. B., & Said, C. P., Baron, S., & Todorov, A. (2009). Nonlinear
Todorov, A. (2013). Robust selectivity for faces in the amygdala response to face trustworthiness: Contributions
human amygdala in the absence of expressions. J Cogn Neu- of high and low spatial frequency information. J Cogn Neu-
rosci, 25(12), 2086–2106. rosci, 21, 519–528.
Mormann, F., Kornblith, S., Quiroga, R. Q., Kraskov, A., Said, C. P., Dotsch, R., & Todorov, A. (2010). The amygdala
Cerf, M., Fried, I., & Koch, C. (2008). Latency and and FFA track both social and non-social face dimensions.
selectivity of single neurons indicate hierarchical process- Neuropsychologia, 48, 3596–3605.
ing in the human medial temporal lobe. J Neurosci, 28, Said, C. P., Haxby, J. V., & Todorov, A. (2011). Brain systems
8865–8872. for assessing the affective value of faces. Philos Trans R Soc
Morris, J. S., Frith, C. D., Perrett, D. I., Rowland, D., Lond B Biol Sci, 336, 1660–1670.
Young, A. W., Calder, A. J., & Dolan, R. J. (1996). A dif- Sander, D., Grafman, J., & Zalla, T. (2003). The human
ferential neural response in the human amygdala to fearful amygdala: An evolved system for relevance detection. Rev
and happy facial expressions. Nature, 383, 812–815. Neurosci, 14, 303–316.
Nakamura, K., Mikami, A., & Kubota, K. (1992). Activity of Sanghera, M. F., Rolls, E. T., & Roper-Hall, A. (1979).
single neurons in the monkey amygdala during perfor- Visual response of neurons in the dorsolateral amygdala of
mance of a visual discrimination task. J Neurophysiol, 67, the alert monkey. Exp Neurol, 63, 61–62.
1447–1463. Santos, A., Mier, D., Kirsch, P., & Meyer-Lindenberg, A.
Ojemann, J. G., Ojemann, G. A., & Lettich, E. (1992). Neu- (2011). Evidence for a general face salience signal in
ronal activity related to faces and matching in human right human amygdala. NeuroImage, 14, 3111–3116.
nondominant temporal cortex. Brain, 115, 1–13. Sato, W., Kochiyama, T., Uono, S., Matsuda, K., Usui, K.,
Olson, I. R., & Marshuetz, C. (2005). Facial attractiveness Inoue, Y., & Toichi, M. (2012). Temporal profile of amyg-
is appraised in a glance. Emotion, 5, 498–502. dala gamma oscillations in response to faces. J Cogn Neurosci,
Pasley, B. N., Mayes, L. C., & Schultz, R. T. (2004). Subcor- 24, 1420–1433.
tical discrimination of unperceived objects during binocu- Saygin, Z. M., Osher, D. E., Augustinack, J., Fischl, B., &
lar rivalry. Neuron, 42, 163–172. Gabrieli, J. D. (2011). Connectivity-based segmentation of
Perrett, D. I., Rolls, E. T., & Caan, W. (1982). Visual human amygdala nuclei using probabilistic tractography.
neurons responsive to faces in the monkey temporal cortex. NeuroImage, 56, 1353–1361.
Exp Brain Res, 47, 329–342. Seeck, M., Schomer, D., Mainwaring, N., Ives, J.,
Perrett, D. I., Smith, P. A. J., Potter, D. D., Mistlin, A. J., Dubuisson, D., Blume, H., … Mesulam, M. M. (1995).
Head, A. S., Milner, A. D., & Jeeves, M. A. (1984). Neurons Selectively distributed processing of visual object
mende-siedlecki and todorov: the role of the amygdala in face processing 775
67 The Cultural Neuroscience of
Human Perception
NALINI AMBADY AND JONATHAN B. FREEMAN
ABSTRACT Culture and the brain were once thought of as Bharucha, 2009; Chiao & Ambady, 2007; Han &
mutually exclusive views on behavioral variation—an idea that Northoff, 2008; Han et al., 2013; Kitayama & Park,
is changing with the emerging field of cultural neuroscience. 2010). Accumulating evidence for neuroplasticity and
In this chapter, we discuss recent research examining the
interplay of cultural and genetic factors on the neural bases
the coextension of culture and the brain has made clear
of human perception. We conclude that cultural experience that cultural and neural processes are not inherently
readily impacts basic mechanisms underlying perception, distinct objects of inquiry; instead, they may richly inter-
ranging from lower-level nonsocial domains (e.g., attentional act. A complete and accurate understanding of behav-
deployment and object perception) to higher-order social ioral variation requires the comprehensive study of
domains (e.g., emotion recognition and theory of mind).
these tandem processes.
More broadly, we discuss the promise and pitfalls of a cultural
neuroscience approach to psychological processes and Arguably, the human brain evolved to permit elabo-
explain how this multilevel approach can contribute to both rate social behavior, particularly so that such behavior
cognitive neuroscience as well as to social and cultural may be adapted to the social structures and patterns in
psychology. which individuals find themselves. Cultural neurosci-
ence is centered around the assumption of a bidirec-
tional relationship between culture and the brain:
the brain adapts to cultural processes, and cultural
The cultural neuroscience of human perception
processes adapt to neural constraints (Ambady &
Humans are biological systems embedded in larger Bharucha, 2009). As such, this burgeoning field aims to
social systems. We see, think, and act in the context of provide a fuller understanding of psychological phe-
others. We exist in cultural environments in which spe- nomena at multiple levels of analysis.
cific meanings, practices, and institutions organize our
perceptual, cognitive, and behavioral tendencies. In Neuroplasticity
spite of this fact, culture and the brain were once
thought of as mutually exclusive views on variation in Progress in the emerging field of cultural neuroscience
behavior and were subjects of study divided by relatively has been bolstered by accumulating evidence for
strict divisions across the natural and social sciences. neuroplasticity. The human brain has long been known
Although it is clear now that culture and social experi- to be intrinsically malleable, with environmental and
ence may readily shape brain function, this was not experiential factors determining both its function and
always the case. Even with respect to more general, structure. Occipital regions centrally involved in vision,
nonsocial experience, the broader field of cognitive for example, can be recruited to process sounds in
neuroscience early on was surprised by such influences blind individuals (Gougoux et al., 2009), and primary
on the brain. In the early 1990s, Posner (1993, p. 674) auditory cortex can be co-opted to process visual stimuli
remarked in Science: “If the neural systems used for a in deaf individuals (Finney et al., 2001). Thus, in the
given task can change with 15 minutes of practice … face of impairments, the brain is able to flexibly reor-
how can we any longer separate organic structures from ganize itself. Even beyond recovering from impair-
their experience in the organism’s history?” ments, recent evidence has documented the astounding
The answer, we now know, is simply that we cannot. flexibility of neural mechanisms to high-level social and
Recent research examining the neural mechanisms cultural experience.
underlying the effects of cultural experience on the Consider a study examining the influences of jug-
brain and, conversely, examining how the brain gling experience on neural activity. All participants
gives rise to cultural experience has led to the emer- lacked juggling experience; half were asked to teach
gence of a new field: cultural neuroscience (Ambady & themselves to be able to juggle for at least one minute
Figure 67.1 Cultural specificity to (A) dominant and subordinate figural outlines in (B) caudate and dorsal medial prefrontal
cortex (mPFC) response. (Adapted from Freeman et al., 2009.) (See color plate 56.)
That being said, there are a number of logistical on a broad range of perceptual phenomena at multi-
challenges in conducting research in the area of cul- ple levels of analysis, from lower-level attention to
tural neuroscience. Attention must be paid to ensure higher-order social processes. Advances in concepts
proper control is implemented, given differing and methodology in cognitive neuroscience provide a
scanner sites. Further, almost all the recent discoveries solid foundation for examining the mutual interplay
in cognitive neuroscience have come from richer of cultural, neural, and genetic forces throughout the
industrialized nations. For example, to our knowledge, life span.
there have been no studies in cultural neuroscience in Historically, psychology has vacillated in its focus; at
Africa, South America, or the Middle East. Within the some times, popular approaches focus on influences of
field of psychology, 95% of psychological samples learning, experience, and culture, and at other times
come from countries with only 12% of the world’s they focus on innateness and fixity. Like most dichoto-
population (Arnett, 2008), typically Western industrial- mies, rarely does either side provide a full and complete
ized nations (Henrich et al., 2010). Within the field of characterization of the phenomenon. Theoretical and
human neuroimaging alone, 90% of peer-reviewed empirical work identifying the various factors underly-
neuroimaging studies come from Western countries ing what is learned and what is innate and how they
(Chiao & Ambady, 2007). Nevertheless, the cultural interact will be critical, and the cultural neuroscience
neuroscience framework represents an unprecedented approach would seem highly promising in this regard.
opportunity to link human diversity across multiple The cultural neuroscience approach provides the excit-
levels of analysis, from genes and brain to mind and ing opportunity to examine the mutual interplay of
behavior. As outlined throughout this chapter, it is culture and biology across multiple levels of analysis,
revealing novel insights into the influences of culture from genes and brain to mind and behavior, and
69 MASSIMINI 801
71 BLUMENFELD 821
74 BLANKE 865
75 HAGGARD 875
Introduction
GIULIO TONONI
790 consciousness
68 Neural Correlates of Consciousness:
Perception and Volition
GABRIEL KREIMAN
ABSTRACT Consciousness is the result of interactions among theorems. An elegant theoretical framework to define
neuronal networks in our brains. Although our mechanistic when, how, and why a given circuit of neurons can give
understanding of consciousness remains tentative, the last two rise to consciousness whereas an apparently similar
decades have seen intense and increasing efforts aimed
toward elucidating the neural circuits and spatiotemporal
circuit or ensemble firing pattern does not has been
dynamics underlying certain aspects of conscious experience. proposed by Tononi. The theory proposes that “the
Experimental paradigms such as those involving bistable per- level of consciousness of a physical system is related
ception (wherein a constant stimulus can give rise to different to the repertoire of causal states (information) available
percepts), combined with neurophysiological recordings and to the system as a whole (integration)” (2004, p. 253,
computational theories, have provided hints at neuronal
emphasis in original). This theoretical framework can
signals that correlate with subjective perception. It is also
revealing to determine which firing patterns do not correlate help distinguish why some physical systems may experi-
with consciousness or which ones correlate with preconscious ence consciousness while others do not, which species
sensations. Both nonconscious and preconscious signals have can experience consciousness, why different neuronal
been described in perceptual studies as well as in studies of circuits show distinct contributions to consciousness,
volitional decisions. Discriminating neuronal activity linked to
and even why different activity patterns in the same
internal perceptual changes in the absence of external
changes provides empirical constraints and initial glimpses neuronal circuit may show different correlations with
into how signaling cascades in the brain can give rise to conscious experience.
consciousness. To formulate a principled strategy to begin to inves-
tigate the problem of which aspects of brain function
correlate with specific contents of conscious sensations
The phenomenological feeling of consciousness is central and which do not, Crick and Koch (1990) defined the
to our moment-to-moment experiences. What we see, neural correlates of consciousness (NCC). The NCC
hear, feel, or reminisce forms the content of conscious represents a “minimal set of neuronal events and mech-
sensations. Most scientists would agree that conscious- anisms sufficient for a specific conscious percept”
ness is ultimately encoded and orchestrated by the activ- (Koch, 2005). Over the last two decades, growing enthu-
ity of neurons in the brain, but the elucidation of where, siasm and blossoming efforts have capitalized on tools
when, how, and why neural ensemble dynamics lead to to interrogate brain activity at the circuit level in order
consciousness remains a deep and fundamental mystery. to take initial steps toward investigating the NCC.
Our brains are physical entities. In many ways, the There are multiple fascinating aspects of conscious
chemical components of brains are similar to those experience that need to be explained. We focus the
encountered in plants and trees and distinct from those discussion here on two of them, sensory perception and
in chairs and tables. An important distinguishing volition. With some degree of approximation, we can
feature of the organization of the chemical components think of these as representative of brain input and
in our brains is the existence of interconnected output, respectively. We aim to provide an overview of
neurons—yet interconnected neural circuits are also the advances, difficulties, experiments, and theories
present in Caenorhabditis elegans and related worms. that have shaped the discussions around the relation-
Whether the neuronal circuits in worms lead to con- ship between neural activity and conscious perception
scious sensations somewhat akin to the ones experi- in a few specific instances, which we cover more thor-
enced by humans is not clear. Out of the bewildering oughly at the expense of other themes in the field. In
complexity (Tononi, 2004; Tononi & Edelman, 1998) doing so, we are not doing justice to a large body of
that results from vast numbers of interconnected heroic efforts. We refer the reader to several reviews
neurons arises the feelings of love and pain and the (see Baars, 1989; Blanke, 2012; Blumenfeld, 2011;
capacity to plan our future and prove new mathematical Cotterill, 2001; Crick, Koch, Kreiman, & Fried, 2004;
792 consciousness
the output of those neurons (Koch, 2005). Information Experimental Paradigms As we ascend through the
about local contrast changes is explicitly encoded in the visual hierarchy, neuronal responses become more
retina. Information about the presence or absence of a sophisticated, and the neurons’ firing preferences grad-
face is not explicitly encoded in the retina (even though ually begin to acquire some of the properties that we
the information can be decoded from retinal activity associate with our subjective visual percepts (Connor,
through a cascade of multiple operations). Several Brincat, & Pasupathy, 2007; Logothetis & Sheinberg,
aspects of the properties of retinal photoreceptors 1996; Tanaka, 1996). The arguments used above to rule
argue against their explicit representation of conscious out a role for the retina or LGN in conscious perception
information: (1) retinal photoreceptors can follow lose their weight, and there is a need to use more
rapid spatiotemporal changes that we are not aware of sophisticated experimental paradigms in an attempt to
(e.g., a monitor’s refresh rate); (2) there are no photo- dissociate perception and sensory inputs. Common to
receptors at the blind spot, but we do not see a hole in many of these empirical approaches is to consider two
the corresponding location in an image; (3) there is no conditions in which the external inputs are identical,
feedback to the retina from other parts of the brain; where other internal variables are as similar as possible
(4) visual percepts can be elicited through imagery or and the percepts are different (Kim & Blake, 2005).
dreaming in the absence of retinal activation; (5) the Consider the famous Necker cube (figure 68.1), which
type of visual information that we are typically con- can be perceived in two different configurations. Let us
scious of cannot be decoded in a single step from the assume that fixation, attention, arousal, and other vari-
retina; (6) retinal photoreceptors show major activity ables are identical in two trials in which the viewer
changes in response to multiple small eye movements perceives the two different possible configurations
that we constantly make and that we are completely (figure 68.1B, C). By assumption (identical fixation),
unaware of. Similar comments can be made about the retinal activity should be similar in both trials (except
output cells in the retina (retinal ganglion cells) as well that neurons are capricious and may not show the exact
as their target cells in the thalamus (within the lateral same spiking pattern in two seemingly identical trials,
geniculate nucleus, or LGN). Several, but not all, of not even in the retina; see, e.g., Van Steveninck, Lewen,
these arguments are also pertinent in the case of Strong, Koberle, & Bialek, 1997). Yet, for the percept
primary visual cortex (V1; Crick & Koch, 1995; Leopold, to be distinct in the two conditions, something must be
2012). different in the brain.
Figure 68.1 Example of bistable percepts. (A) The Necker predominantly red lines (B2). (C) Binocular rivalry. When
cube can be interpreted in two possible configurations (A1 presenting different stimuli to each eye, subjects report a
and A2). Upon viewing the Necker cube, subjects can volun- percept that alternates between the two inputs (C1, C2) in a
tarily switch from one configuration to the other (bottom). seemingly random fashion. During binocular rivalry, it is dif-
(B) Monocular rivalry. Upon fixating on the grid of horizontal ficult to voluntarily switch between the two percepts. (See
red lines and vertical blue lines, subjects report a percept color plate 58.)
that alternates between predominantly blue lines (B1) and
794 consciousness
Sheinberg, 1996; Wilke, Leopold, & Logothetis, 2002; & Kastner, 2005). These discrepancies between blood
see, however, Blake, 1989; Blake & Logothetis, 2002). flow measurements and neurophysiological signals have
In areas V4 and MT, 38% and 43% of the neurons also been observed in other experiments (Logothetis,
showed perceptual modulation (Leopold & Logothetis, 2002; Posner & Gilbert, 1999; Sirotin & Das, 2009;
1996; Logothetis & Schall, 1989). Intriguingly, in area Watanabe et al., 2011). An elegant study by Maier and
MT, half of the neurons increased their firing rate when colleagues examined blood oxygen level–dependent
their preferred stimulus was perceived, while the other (BOLD) functional MRI signals, local field potentials
half showed the reverse pattern; that is, they increased (LFPs), and spiking responses in primary visual cortex
their activity when their preferred stimulus was sup- during binocular rivalry (Maier et al., 2008). During
pressed. The correlation between subjective perception physical removal of a stimulus, BOLD signals, LFPs, and
and the activity of some, but not all, MT neurons was spiking responses agreed with each other. However,
also apparent in a structure-from-motion task during during perceptual suppression, there were small
conditions in which the input was ambiguous (Bradley changes in BOLD signals and LFP responses that were
et al., 1998). The bewildering variety of neurons that not observed at the spiking level. These observations
show different degrees of correlation with perception highlight the notion that blood flow and spiking signals
is perhaps a reflection of the intricacy of connectivity measure different aspects of brain function, and the
patterns in neural circuits. The advent of techniques nature of the relationship between these signals may
based on optogenetics may enable the possibility of prove to be important to gain further mechanistic
interrogating (and manipulating) different layers and insights into the circuitry involved in conscious percep-
different types of neurons. Analysis of circuitry at the tion (Leopold, 2012). A potential explanation of these
level of “brain areas” may prove insufficient to uncover results is that blood flow signals in the LGN and in
the NCC. The neurons in V1, V4, and MT that show primary visual cortex reflect feedback modulation from
stronger correlation with subjective perception may higher visual areas after perceptual rivalry has been
have distinct properties (in terms of their location, resolved, consistent with the notion that V1 does not
inputs, and outputs, and how they interact with other have a direct role in visual awareness (Crick & Koch,
neurons) from their neighboring neurons whose firing 1995; Leopold, 2012).
is oblivious or anticorrelated with the percepts. Strong modulation in blood flow signals in early
In the highest echelons of the ventral visual stream, visual areas in the absence of concomitant strong
90% of the neurons in the inferior temporal cortex and modulation at the spike level has also been observed
the superior temporal sulcus showed a correlation in studies of attentional modulation (Kastner &
between their firing rate and the subjective report of Ungerleider, 2000; Reynolds & Chelazzi, 2004). It has
the neuron’s preferred stimulus (Sheinberg & Logothe- been argued that modulatory changes observed during
tis, 1997). In a variant of binocular rivalry–denominated binocular rivalry experiments could reflect attentional
flash suppression, a stimulus is flashed monocularly fol- fluctuations (Macknik & Martinez-Conde, 2009). The
lowed by presentation of another stimulus to the other extent to which attention and awareness can be dissoci-
eye (Wolfe, 1984). The flashed stimulus dominates per- ated has been a matter of significant debate (Dehaene
ception, even though the initial stimulus remains & Changeux, 2011; Koch & Tsuchiya, 2012; Posner,
present. In parallel to the binocular rivalry results, 1994). Under most everyday circumstances, attention
neurons in the macaque inferior temporal cortex and and awareness go hand in hand, yet psychophysical
superior temporal sulcus show a strong correlation with experiments suggest that it is possible to attend to a
the percept (Sheinberg & Logothetis, 1997). Similarly, stimulus even in the absence of awareness (e.g., Koch
most neurons in the human medial temporal lobe also & Tsuchiya, 2012; van Boxtel, Tsuchiya, & Koch, 2010).
show a correlation with subjective perception during Furthermore, a functional imaging study showed that
flash suppression (Kreiman, Fried, & Koch, 2002). blood flow signals in V1 are modulated by attention but
not by changes in awareness (Watanabe et al., 2011).
Attention, Imaging, and Stimulus Configuration These results are consistent with recent neurophysio-
Dependencies In contrast to neurophysiological logical recordings that lend further support to distinct
recordings in macaque V1, several human functional signals giving rise to attentional modulation and aware-
neuroimaging studies have suggested that activity in ness (Maier, 2012).
primary visual cortex also correlates with subjective per- Other factors beyond attention can also influence the
ception (Haynes, Deichmann, & Rees, 2005; Polonsky, relationship between neuronal activity and awareness.
Blake, Braun, & Heeger, 2000; Tong & Engel, 2001; see The discussion in the previous section described differ-
also similar claims in the LGN: Wunderlich, Schneider, ences in the correlations between subjective perception
796 consciousness
Libet asked subjects to tap their index finger at will and approach them with a new arsenal of tools that is
also report the time of their intention to act based on making rapid strides in elucidating other aspects of
an analog clock present on the screen during the exper- cognition, including multielectrode arrays, computa-
iment (Libet, 1985; Libet, Gleason, Wright, & Pearl, tional modeling, microstimulation, optogenetics, and
1983). These experiments revealed that the averaged so on.
scalp electroencephalographic signals preceded the Advances and controversies in trying to correlate
conscious intention of the urge to move by several hun- neural signals and conscious perception have led to a
dreds of milliseconds. The interpretation of these theoretical framework that provides quantitative defini-
experiments has been the subject of much debate in tions of how neuronal interactions could lead to con-
the field (e.g., see Libet, 1985, 2002, and discussions in sciousness (Tononi, 2004). Additionally, initial but
the same issue). significant steps have been made toward better defining
In a recent study, Fried and colleagues took advan- questions about conscious processing (e.g., dissociating
tage of a rare clinical opportunity to record from >1,000 pure attentional effects from conscious perception;
neurons in the human frontal and temporal lobes when Koch and Tsuchiya, 2012), toward sharpening experi-
subjects performed willed action and reported the time mental tools (e.g., noticing that blood flow signals may
of volition onset, as in the Libet experiment. Consistent not reveal underlying spiking; Maier, 2008), and toward
with earlier studies using noninvasive methods, they a richness of experimental paradigms and approaches
found evidence at the single neuron level in humans (Dehaene & Changeux, 2011).
for an anatomically localized early frontal cortex signal There is currently significant excitement in cognitive
that preceded conscious will. Over a time period of neuroscience with the advent of tools that enable the
∼1,500 msec prior to the awareness of will, an increasing manipulation of circuits at unprecedented resolution
number of neurons in two specific brain regions, the (e.g., Han et al., 2009). These tools open the doors to
supplementary motor area and anterior cingulate exciting and promising opportunities to attempt to
cortex, were progressively recruited. The subjectively transiently inactivate and also directly stimulate local
reported onset of volition could be accurately predicted circuits, and thus bias subjective decisions in ways that
on a single trial basis based on neural activity in the have not been possible before. While it is anyone’s guess
supplementary motor area well before the subject’s whether the quest for the NCC will be resolved in the
awareness. Based on these findings, the authors pro- near future, there is no question that we should expect
posed a computational model and a biophysically plau- fascinating surprises and novel insights in the commu-
sible mechanism for the emergence of conscious will in nity’s efforts to elucidate how physical systems lead to
humans based on progressive recruitment of neuronal consciousness.
ensembles in frontal cortex until a threshold is crossed.
The model is consistent with the notion that the all-or-
none nature of consciousness is the result of gradual REFERENCES
accumulation reaching a threshold (Crick & Koch, Alais, D., & Blake, R. (2005). Binocular rivalry. Cambridge,
2003). Furthermore, in another study, Fried showed MA: MIT Press.
that electrical stimulation in the human supplementary Baars, B. (1989). A cognitive theory of consciousness. Cambridge,
motor area triggered an “urge” to perform motor UK: Cambridge University Press.
actions (Fried et al., 1991). Blake, R. (1989). A neural theory of binocular rivalry. Psychol
Rev, 96, 145–167.
Blake, R., & Logothetis, N. (2002). Visual competition. Nat
Outlook Rev Neurosci, 3, 13–21.
Blanke, O. (2012). Multisensory brain mechanisms of bodily
The fundamental problem of understanding how neu- self-consciousness. Nat Rev Neurosci, 13, 556–571.
ronal circuits give rise to conscious sensations has risen Blumberg, J., & Kreiman, G. (2010). How cortical neurons
help us see: Visual recognition in the human brain. J Clin
from nebulous beginnings and debates to become a
Invest, 120, 3054–3063.
major effort in cognitive neuroscience. Progress in the Blumenfeld, H. (2011). Epilepsy and the consciousness
field should be interpreted with cautious optimism. system: Transient vegetative state? Neurol Clin, 29,
Caution is important because this is undoubtedly a dif- 801–823.
ficult problem, and current theories are as diverse as Bonneh, Y., Cooperman, A., & Sagi, D. (2001). Motion-
they could be. And yet there is optimism and steady induced blindness in normal observers. Nature, 411,
798–801.
progress. The last two decades have seen the blossom- Bradley, D. C., Chang, G. C., & Andersen, R. A. (1998).
ing of a young generation of energetic and heroic inves- Encoding of 3D structure from motion by primate area MT
tigators who have dared to ask difficult questions and neurons. Nature, 392, 714–717.
798 consciousness
Macknik, S. L., & Livingstone, M. S. (1998). Neuronal cor- Romo, R., & Schultz, W. (1992). Role of primate basal
relates of visibility and invisibility in the primate visual ganglia and frontal cortex in the internal generation of
system. Nat Neurosci, 1, 144–149. movements. III. Neuronal activity in the supplementary
Macknik, S. L., & Martinez-Conde, S. (2009). The role motor area. Exp Brain Res, 91(3), 396–407.
of feedback in visual attention and awareness. In Schmolesky, M., Wang, Y., Hanes, D., Thompson, K.,
M. Gazzaniga, The cognitive neurosciences (Cambridge, MA: Leutgeb, S., Schall, J., & Leventhal, A. (1998). Signal
MIT Press, 2009), 1165–1179. timing across the macaque visual system. J Neurophysiol, 79,
Maier, A. (2012). The cortical microcircuitry of conscious 3272–3278.
perception and selective attention. Paper presented the Searle, J. (1998). How to study consciousness scientifically.
Annual Meeting of the Society for Neuroscience, New Philos Trans R Soc Lond B Biol Sci, 353, 1935–1942.
Orleans, LA. Searle, J. (2005). Consciousness: What we still don’t know
Maier, A., Logothetis, N., & Leopold, D. (2007). Context- [Review of the book The Quest for Consciousness by C. Koch].
dependent perceptual modulation of single neurons in The New York Review of Books. Available from http://www.
primate visual cortex. Proc Natl Acad Sci USA, 104, nybooks.com/articles/archives/2005/jan/13/consciousness-
5620–5625. what-we-still-dont-know/.
Maier, A., Wilke, M., Aura, C., Zhu, C., Ye, F. Q., & Leopold, Sergent, C., Baillet, S., & Dehaene, S. (2005). Timing
D. A. (2008). Divergence of fMRI and neural signals in V1 of the brain events underlying access to consciousness
during perceptual suppression in the awake monkey. Nat during the attentional blink. Nat Neurosci, 8, 1391–1400.
Neurosci, 11, 1193–1200. Sheinberg, D. L., & Logothetis, N. K. (1997). The role of
Maimon, G., & Assad, J. A. (2006a). Parietal area 5 and the temporal areas in perceptual organization. Proc Natl Acad
initiation of self-timed movements versus simple reactions. Sci USA, 94, 3408–3413.
J Neurosci, 26, 2487–2498. Siegel, R. M., & Andersen, R. A. (1988). Perception of three-
Maimon, G., & Assad, J. A. (2006b). A cognitive signal for the dimensional structure from motion in monkey and man.
proactive timing of action in macaque LIP. Nat Neurosci, 9, Nature, 331, 259–261.
948–955. Simons, D. J., & Rensink, R. A. (2005). Change blindness:
Metzinger, T. (2000). Neural correlates of consciousness: Empiri- Past, present, and future. Trends Cogn Sci, 9, 16–20.
cal and conceptual questions. Cambridge, MA: MIT Press. Singer, W. (1998). Consciousness and the structure of neu-
Mukamel, R., & Fried, I. (2012). Human intracranial record- ronal representations. Philos Trans R Soc Lond B Biol Sci, 353,
ings and cognitive neuroscience. Annu Rev Psychol, 63, 1829–1840.
511–537. Sirotin, Y. B., & Das, A. (2009). Anticipatory haemodynamic
Myerson, J., Miezin, F., & Allman, J. (1981). Binocular signals in sensory cortex not predicted by local neuronal
rivalry in macaque monkeys and humans: A comparative activity. Nature, 457, 475–479.
study in perception. Behav Anal Lett, 1, 149–159. Tanaka, K. (1996). Inferotemporal cortex and object vision.
Okano, K., & Tanji, J. (1987). Neuronal activities in the Annu Rev Neurosci, 19, 109–139.
primate motor fields of the agranular frontal cortex preced- Thorpe, S., Fize, D., & Marlot, C. (1996). Speed of
ing visually triggered and self-paced movement. Exp Brain processing in the human visual system. Nature, 381,
Res, 66, 155–166. 520–522.
O’Regan, J., Rensink, R., & Clark, J. (1999). Change- Tong, F., & Engel, S. (2001). Interocular rivalry revealed in
blindness as a result of mudsplashes. Nature, 398, 34. the human cortical blind-spot representation. Nature, 411,
Parker, A. J., & Newsome, W. T. (1998). Sense and the single 195–199.
neuron: Probing the physiology of perception. Annu Rev Tononi, G. (2004). An information integration theory of con-
Neurosci, 21, 227–277. sciousness. BMC Neurosci, 5, 42.
Polonsky, A., Blake, R., Braun, J., & Heeger, D. (2000). Tononi, G. (2005). Consciousness, information integration,
Neuronal activity in human primary visual cortex correlates and the brain. Prog Brain Res, 150, 109–126.
with perception during binocular rivalry. Nat Neurosci, 3, Tononi, G., & Edelman, G. (1998). Consciousness and
1153–1159. complexity. Science, 282(5395), 1846–1851.
Posner, M. I. (1994). Attention: The mechanisms of con- Tononi, G., & Koch, C. (2011). The neural correlates
sciousness. Proc Natl Acad Sci USA, 91, 7398–7403. of consciousness: An update. Ann NY Acad Sci, 1124,
Posner, M. I., & Gilbert, C. D. (1999). Attention and primary 239–261.
visual cortex. Proc Natl Acad Sci USA, 96, 2585–2587. Van Boxtel, J. J., Tsuchiya, N., & Koch, C. (2010). Oppos-
Raymond, J. E., Shapiro, K. L., & Arnell, K. M. (1992). ing effects of attention and consciousness on afterimages.
Temporary suppression of visual processing in an RSVP Proc Natl Acad Sci USA, 107, 8883–8888.
task: An attentional blink? J Exp Psychol Hum Percept Perform, Van Steveninck, R., Lewen, G. D., Strong, S. P.,
18, 849–860. Koberle, R., & Bialek, W. (1997). Reproducibility
Rees, G., Kreiman, G., & Koch, C. (2002). Neural and variability in neural spike trains. Science, 275,
correlates of consciousness in humans. Nat Rev Neurosci, 3, 1805–1808.
261–270. VanRullen, R., & Thorpe, S. (2002). Surfing a spike wave
Rees, G., Russell, C., Frith, C. D., & Driver, J. (1999). Inat- down the ventral stream. Vis Res, 42, 2593–2615.
tentional blindness versus inattentional amnesia for fixated Wang, Q., & Burkhalter, A. (2007). Area map of mouse
but ignored words. Science, 286, 2504–2507. visual cortex. J Comp Neurol, 502, 339–357.
Reynolds, J. H., & Chelazzi, L. (2004). Attentional Watanabe, M., Cheng, K., Murayama, Y., Ueno, K.,
modulation of visual processing. Annu Rev Neurosci, 27, Asamizuya, T., Tanaka, K., & Logothesis, N. (2011).
611–647. Attention but not awareness modulates the BOLD signal in
800 consciousness
69 Toward an Objective Index
of Consciousness
MARCELLO MASSIMINI
ABSTRACT We usually assess another individual’s level of con- to questions. If awakened, at times she may say she was
sciousness based on her/his ability to connect to the sur- experiencing nothing; at other times that she was
rounding environment and produce appropriate responses. dreaming, and recall a vivid experience. During anes-
However, we know that consciousness can be entirely gener-
ated within the brain, even in the absence of any interaction
thesia, some people may regain consciousness yet be
with the external world; this happens almost every night, unable to signal it. Similarly, some patients with brain
while we dream, and may occur during certain forms of anes- damage may be behaviorally unresponsive and thus
thesia as well as in brain-injured patients who emerge from judged clinically unconscious, yet they may be able to
coma and remain unresponsive. Yet, to this day, we still lack generate brain signals indicating they understood a
a way to assess the level of consciousness that is independent
question or a command. In general, the problem is
of processing sensory inputs and producing appropriate
motor outputs. The aim of this chapter is to suggest that that, while we assess level of consciousness based on an
establishing an objective index of the level of consciousness individual’s ability to access and respond to the external
that does not rely on a subject’s capacity to access or respond environment, these features are not necessary.
to the surrounding environment is a challenging undertak-
ing, but not an impossible one. One way to accomplish this
task involves three fundamental steps. First, start from theo-
retical principles that suggest which intrinsic properties are Dissociations between consciousness and
fundamental for a physical system to give rise to conscious responsiveness
experience. Second, devise a practical means to gauge these
properties in human brains. Third, test the candidate metric
in different controlled conditions—such as wakefulness, Sleep Behaviorally, the most striking consequence of
sleep, anesthesia, and brain injury—in which consciousness is falling asleep is a progressive disconnection from the
known to be present, diminished, or lost. To the extent that external environment. Reaction times to auditory tones
this procedure yields a reliable, graded measurement scale become longer prior to falling asleep, and responses
along the unconsciousness-consciousness spectrum, an objec- are absent coincident with the transition to nonrapid
tive frame of reference may be available to assess subjects who
are completely disconnected from the external environment.
eye movement (NREM) sleep (Ogilvie & Wilkinson,
1984). With the deepening of NREM sleep, responses
Everyone knows what consciousness is: it is what van- can be obtained only with progressively louder tones;
ishes when we fall into dreamless sleep or general anes- this feature is known as “high arousal threshold,” and
thesia and reappears when we wake up or when we it persists during REM sleep (Rechtschaffen, Hauri, &
dream—in other words, it is synonymous with experi- Zeitlin, 1966).
ence. Though we are very familiar, via a first-person Conscious experience during sleep can be assessed
perspective, with the transition from consciousness to by studying subjective reports obtained after awaken-
unconsciousness and back, we still lack a scientifically ings from different stages or at different times of night.
well-grounded method to assess the level of conscious- Generally, during NREM sleep, there are longer reports
ness of other individuals. How do we judge if somebody later in the night and short reports early in the night.
is conscious—experiencing things such as sights, Notably, during the first NREM episode, when slow
sounds, and maybe pains? Usually, if we observe pur- waves are prevalent in the electroencephalogram
poseful behavior and appropriate responses to sensory (EEG), a substantial number of awakenings yield no
stimuli or commands, we decide that the person is con- report whatsoever (McNamara et al., 2010). Thus,
scious. If in doubt, as when someone is resting with eyes NREM sleep early in the night is the only phase of adult
closed, we can ask: if she answers that she was thinking life during which healthy human subjects may deny that
or daydreaming, we infer she was conscious. they were experiencing anything at all. In this case,
But sometimes matters are less clear: someone fast responsiveness and consciousness are concurrently
asleep shows no purposeful activity and will not respond reduced.
802 consciousness
The empirical search for an objective index of crucial role has been proposed for high-frequency
consciousness: Advances and problems neural activity and the dynamic formation of neuronal
coalitions, including prefrontal areas (Crick & Koch,
The physiological, pharmacological, and pathological 2003); the occurrence of reentrant interactions between
examples outlined above demonstrate that conscious the front and the back of the brain (Edelman, 2001); a
experience can be present in subjects who are unable high level of cortical depolarization with a background
to access sensory events and/or unable to produce of high-frequency activity (Llinás, Ribary, Contreras, &
appropriate (behavioral or neural) responses. Ideally, Pedroarena, 1998); the involvement of a “global work-
since sensory and motor functions are not necessary, space” (Baars, 2005) encompassing frontoparietal areas
one should try to identify and measure directly in the (Dehaene & Changeux, 2011); the activation of higher-
brain the minimal set of neuronal mechanisms that are order association cortices (Laureys, 2004); the long-
necessary and sufficient for consciousness to emerge, range, high-frequency synchronization of brain activity
the so-called neural correlates of consciousness (NCC). (Singer, 2001); and related ideas.
Practically, these aspects of neuronal activity should While these proposals have heuristic value, the
always be detectable when consciousness is present and empirical evidence does not provide criteria for neces-
always absent when consciousness is lost. sity and sufficiency. For example, prefrontal areas are
Over the past two decades, great progress has been certainly important for evaluating and reflecting and
made in the search for the NCC. This approach, spear- deciding upon an experience, but why would they be
headed by Francis Crick and Christof Koch (Crick & necessary for the experience itself? Indeed, conscious-
Koch, 2003) is often implemented by comparing brain ness appears to survive large prefrontal lesions
responses to sensory stimuli that are or are not per- (Markowitsch & Kessler, 2000). Reentry can favor
ceived, as indicated by the subject’s verbal report. cooperative interactions between distant areas, but
Stimuli can be embedded in noise so they are at thresh- there is abundant opportunity for reentry in structures
old for visibility, they can be masked by strong stimuli that do not seem to participate directly in generating
presented briefly afterwards, or they can be made consciousness, such as the hippocampal formation
visible/invisible through manipulations such as binocu- (Crick & Koch, 1998). High-frequency firing may
lar rivalry (see Dehaene & Changeux, 2011, for review). signal strong activation, but it can occur in the absence
In this way, it has been possible to ask whether the vis- of consciousness, and vice versa. For example, strong,
ibility of a stimulus is correlated with changes in activity high-frequency activity, and even reactivity to sensory
in primary versus higher-order visual areas, whether stimuli, can occur during both deep sleep and anesthe-
visibility is accompanied by late EEG potentials (e.g., sia (Kakigi et al., 2003; Kroeger & Amzica, 2007;
P300), or by changes in power and coherence in various Steriade, 2000), and both humans and rats undergoing
frequency bands. While this work is essential in advanc- inhalation anesthesia show loss of consciousness (in
ing the scientific study of consciousness, it should be rats, loss of righting reflex) despite increased gamma
remembered that in these experimental paradigms sub- power in the EEG (Imas et al., 2005). Late event-
jects remain fully conscious throughout—it is only the related potentials, such as the mismatch negativity and
content of consciousness that changes. Thus, the level P300, reflect widespread brain activations that occur
of consciousness is not the variable under study. Cru- when subjects report detecting a stimulus, thus repre-
cially, these experimental approaches depend on a sub- senting a neural correlate of conscious access to
ject’s capacity to decide whether she saw or heard sensory events (Dehaene & Changeux, 2011). However,
something and her willingness to communicate her more often than not, these components are absent in
decision: in other words, in these paradigms it is diffi- conscious brain-injured patients (Fischer et al., 2010;
cult to dissociate consciousness from sensory processing Höller et al., 2011; King et al., 2013). Synchronization
and executive function. Yet, as described in the previous usually reflects the occurrence of distributed interac-
section, these functions may be impaired or absent tions, which is presumably important for conscious-
independently of consciousness, especially in pathologi- ness, but there are well-known situations, such as
cal conditions. generalized seizures, in which frontoparietal areas and
Undoubtedly, studying the neural correlates of per- high-order associative networks are highly active and
ception has added new insights, suggesting several pos- massively synchronized, yet consciousness is lost
sibilities concerning the neural mechanisms that may (Arthuis et al., 2009). Similarly, coherence and Granger
underlie conscious experience. Thus, it has been pro- causality can actually be increased when consciousness
posed that consciousness may require the ignition of is reduced in NREM sleep (Duckrow & Zaveri, 2005)
strong, widespread brain activations. For example, a and during loss of consciousness induced by general
804 consciousness
(integration) to produce differentiated responses Conversely, when subjects lose consciousness during
(information). According to this proposal, a signature NREM sleep, TMS pulses invariably produce a simple
of consciousness is that the thalamocortical system wave of activation that remains localized to the site of
should respond to perturbations with complex, rapidly stimulation, indicating a breakdown of communication
changing activity patterns (information) that affect a and a loss of integration within thalamocortical net-
distributed set of cortical areas (integration). On the works (Massimini et al., 2005). The disappearance of a
other hand, it can be predicted that during loss of con- long-range, differentiated pattern of cortical activation
sciousness, whether this is caused by sleep, anesthesia, is not simply due to a reduction of responsiveness of
or coma, the brain should react to perturbations with a hyperpolarized cortical neurons. In fact, increasing
response that is local (loss of integration) and/or ste- TMS intensity only results in a larger, simple positive-
reotypical (loss of information). negative wave, closely resembling a spontaneous sleep
Practically, these predictions can be tested in human slow wave (Massimini et al., 2007). In this case, the
brains by employing a combination of transcranial mag- response to TMS is stereotypical and spreads like an oil
netic stimulation (TMS) and high-density EEG, a tech- spot to vast regions of the cortex, revealing a loss of
nique that allows stimulating directly a subset of cortical differentiation. Similar local and/or stereotypical
neurons and measuring, with good spatial-temporal responses are invariably also found during general anes-
resolution, the effects produced by this perturbation on thesia (Ferrarelli et al., 2010) as well as in brain-injured
the rest of the thalamocortical system (Ilmoniemi et al., patients with an unambiguous clinical diagnosis of a
1997). As shown in figure 69.1, in healthy, awake sub- vegetative state (Rosanova et al., 2012). Crucially,
jects TMS triggers a complex EEG response involving wakefulness-like, complex responses always recover
different cortical areas at different times (Massimini during REM sleep (Massimini et al., 2010) in minimally
et al., 2005; Rosanova et al., 2009). conscious patients, who show signs of nonreflexive
Figure 69.1 Examples of transcranial magnetic stimulation temporally differentiated patterns of activation. During non-
(TMS)-evoked cortical activations in sleep (A), anesthesia rapid eye movement (NREM) sleep, anesthesia, and the veg-
(B), and brain-injured patients (C). The gray arrows indicate etative state, the thalamocortical system, despite being active
the cortical sites of TMS. The values in the parentheses indi- and reactive, loses its ability to engage in distributed, complex
cate the intensity of the electric field (volt/meter) induced by activity patterns; it either breaks down in casually indepen-
TMS on the cortical surface. The traces show the TMS-evoked dent modules (loss of integration) or, when TMS is delivered
currents recorded from eight cortical sources in both hemi- at high intensity, bursts in a large and stereotypical response
spheres (gray circles); the thick trace highlights the activation (loss of differentiation). During rapid eye movement (REM)
recorded from the cortical areas located under the stimulator. sleep and in the minimally conscious state, the TMS response
During wakefulness, TMS triggers a sustained response recovers its spatial spread and differentiation.
that engages distributed cortical sources in spatially and
806 consciousness
activity, and in locked-in syndrome (LIS) subjects, who consciousness can be known based on subjective reports
are totally paralyzed except for vertical eye movements upon awakening—PCI provides a reproducible and
through which they signal that they are aware (Rosanova reliable scale along the unconsciousness-consciousness
et al., 2012). spectrum. This scale can then be used as an indepen-
dent frame of reference to assess more challenging
Establishing a Measuring Scale Besides providing cases. For example, brain-injured patients (MCS, EMCS,
qualitative support to basic theoretical predictions, and LIS) who show minimal signs of consciousness
TMS and high-density EEG measurements open the attained values of brain complexity that were invariably
possibility of developing a quantitative index of the above the maximum value obtained in unconscious,
level of consciousness, a necessary step to construct a anesthetized, or sleeping, healthy subjects.
measuring scale. To this end, a novel empirical measure To the extent that this measurement scale is further
called the perturbational complexity index (PCI) was validated by empirical data, it may be then applicable
recently introduced (Casali et al., 2013). Calculating to disconnected/unresponsive individuals whose level
PCI involves two fundamental steps: (1) perturbing the of consciousness is unknown. Already in the context of
cortex with TMS to engage distributed interactions in the present results, finding a PCI value above the sleep-
the brain (integration) and (2) “zipping” (i.e., compress- anesthesia distribution in a patient who is otherwise
ing) the resulting electrocortical responses to measure completely disconnected from the external environ-
their algorithmic complexity (information). The under- ment would suggest that she/he is conscious to some
lying idea is that PCI should be low if causal interaction extent.
among cortical areas is reduced (loss of integration), Crucially, PCI is measured by evaluating the com-
because the matrix of activation engaged by TMS is pressibility of the deterministic brain response to TMS,
spatially restricted; PCI is also expected to be low if a perturbation that engages large portions of the thala-
many interacting areas react to the perturbation but mocortical system directly without requiring the sub-
they do so in a stereotypical way (loss of differentiation) jects to perform any sensory, motor, or cognitive task.
because, in this case, the resulting matrix is large but In this way, the brain’s capacity for consciousness can
redundant and can be effectively compressed. In fact, be assessed based on the complexity of cortical interac-
PCI should reach high values only if the initial perturba- tions, independent of the subject’s capacity to access
tion is transmitted to a large set of integrated areas that and react to external stimuli.
react in a differentiated way, giving rise to a spatiotem-
poral pattern of deterministic activation that cannot be Conclusion
easily reduced. In these terms, PCI provides a rough
estimation of the theoretical measure of Φ, which can Compelling evidence suggests that subjective experi-
be defined as the amount of irreducible information ence can be generated within a brain that is discon-
that a system generates above its parts (Tononi, 2008). nected from the external world on the input and output
In recent work, PCI was tested on a large data set of side. In these cases, behavioral and neurophysiological
TMS-evoked potentials recorded in healthy subjects measures of consciousness that rely on the brain’s
during wakefulness, dreaming, NREM sleep, and differ- capacity to access or respond to the external environ-
ent levels of sedation induced by various anesthetic ment are not applicable by definition. In the present
agents (midazolam, xenon, and propofol), as well as in chapter, we suggest that resorting to a theoretical
brain-injured patients who emerged from coma and approach, combined with systematic empirical testing
entered different clinical states (vegetative, MCS, EMCS, in controlled conditions, may help overcome such cir-
LIS). As shown in figure 69.2, PCI is reproducible within cularity. Theoretical principles suggest that in order to
and across subjects and depends in a graded fashion on obtain a reliable index of level of consciousness, mea-
the level of consciousness. Crucially, in healthy (awake, suring either the spread or the entropy of neuronal
sleeping, and anesthetized) subjects—whose level of activations is not enough; instead, one should assess the
during intermediate levels of propofol sedation and during of functional communication (emergence from the mini-
sleep stage 1. (D) PCI values are shown for 48 TMS sessions mally conscious state; EMCS) and attaining levels of healthy
collected from 20 brain-injured patients. PCI followed the awake subjects in locked-in syndrome (LIS). Patient results
level of consciousness (as clinically assessed with coma recov- are within the frame of reference obtained in awake, sleeping,
ery scale, or CRS-R) progressively increasing from vegetative and anesthetized control subjects (modified from Casali et al.,
(VS) through a minimally conscious state (MCS) to recovery 2013). (See color plate 59.)
808 consciousness
inter-individual variability of bispectral index and spectral Monti, M. M., Vanhaudenhuyse, A., Coleman, M. R.,
entropy at loss of consciousness during increasing concen- Boly, M., Pickard, J. D., Tshibanda, L., … Laureys, S.
trations of dexmedetomidine, propofol, and sevoflurane. (2010). Willful modulation of brain activity in disorders of
Br J Anaesth, 107(4), 573–580. consciousness. N Engl J Med, 362(7), 579–589.
King, J. R., Faugeras, F., Gramfort, A., Schurger, A., Nir, Y., & Tononi, G. (2010). Dreaming and the brain: From
El Karoui, I., Sitt, J. D., … Dehaene, S. (2013). Single- phenomenology to neurophysiology. Trends Cogn Sci, 14(2),
trial decoding of auditory novelty responses facilitates 88–100.
the detection of residual consciousness, NeuroImage, 83, Ogilvie, R. D., & Wilkinson, R. T. (1984). The detection
726–738. of sleep onset: Behavioral and physiological convergence.
King, J. R., Sitt, J. D., Faugeras, F., Rohaut, B. A., Psychophysiology, 21(5), 510–520.
El Karoui, I., Cohen, L., … Dehaene, S. (2013). Informa- Owen, A. M., Coleman, M. R., Boly, M., Davis, M. H.,
tion sharing in the brain indexes consciousness in noncom- Laureys, S., & Pickard, J. D. (2006). Detecting awareness
municative patients, Curr Biol, 23(19), 1914–1919. in the vegetative state. Science, 313(5792), 1402.
Koulack, D. (1969). Effects of somatosensory stimulation on Rechtschaffen, A., Hauri, P., & Zeitlin, M. (1966). Audi-
dream content. Arch Gen Psychiatry, 20(6), 718–725. tory awakening thresholds in REM and NREM sleep stages.
Kroeger, D., & Amzica, F. (2007). Hypersensitivity of the Percept Mot Skills, 22(3), 927–942.
anesthesia-induced comatose brain. J Neurosci, 27(39), Rosanova, M., Casali, A., Bellina, V., Resta, F.,
10597–10607. Mariotti, M., & Massimini, M. (2009). J Neurosci, 29(24),
Laureys, S. (2004). Functional neuroimaging in the vegeta- 7679–7685.
tive state. NeuroRehabilitation, 19(4), 335–341. Rosanova, M., Gosseries, O., Casarotto, S., Boly, M.,
Lempel, A., & Ziv, J. (1976). On the complexity of finite Casali, A. G., Bruno, M.-A., … Massimini, M. (2012).
sequences. IEEE Trans Inform Theory, 22, 75–81. Recovery of cortical effective connectivity and recovery
Llinás, R., Ribary, U., Contreras, D., & Pedroarena, C. of consciousness in vegetative patients. Brain, 135(Pt 4),
(1998). The neuronal basis for consciousness. Philos Trans 1308–1320.
R Soc Lond B Biol Sci, 353(1377), 1841–1849. Sanders, R. D., Tononi, G., Laureys, S., & Sleigh, J. W.
Markowitsch, H. J., & Kessler, J. (2000). Massive impair- (2012). Unresponsiveness ≠ unconsciousness. Anesthesiology,
ment in executive functions with partial preservation of 116(4), 946–959.
other cognitive functions: The case of a young patient with Sandin, R. H., Enlund, G., Samuelsson, P., & Lennmarken,
severe degeneration of the prefrontal cortex. Exp Brain Res, C. (2000). Awareness during anesthesia: A prospective case
133(1), 94–102. study. Lancet, 355(9205), 707–711.
Marinazzo, D., Gosseries, O., Boly, M., Ledoux, D., … Schnakers, C., Majerus, S., & Laureys, S. (2005). Bispectral
Laureys, S. (2014). Directed information transfer analysis of electroencephalogram signals during recovery
in scalp electroencephalographic recordings: Insights from coma: Preliminary findings. Neuropsychol Rehabil,
on disorders of consciousness. Clin EEG Neurosci, 45, 15(3–4), 381–388.
33–39. Seth, A. K., Barrett, A. B., & Barnett, L. (2011). Causal
Massimini, M., Boly, M., Casali, A., Rosanova, M., & density and integrated information as measures of con-
Tononi, G. (2009). A perturbational approach for evaluat- scious level. Philos Transact A Math Phys Eng Sci, 369,
ing the brain’s capacity for consciousness. Prog Brain Res, 3748.
177, 201–214. Seth, A. K., Dienes, Z., Cleeremans, A., Overgaard, M., &
Massimini, M., Ferrarelli, F., Esser, S. K., Riedner, B. A., Pessoa, L. (2008). Measuring consciousness: Relating
Huber, R., Murphy, M., … Tononi, G. (2007). Triggering behavioural and neurophysiological approaches. Trends
sleep slow waves by transcranial magnetic stimulation. Proc Cogn Sci, 12, 314.
Natl Acad Sci USA, 104(20), 8496–8501. Singer, W. (2001). Consciousness and the binding problem.
Massimini, M., Ferrarelli, F., Huber, R., Esser, S. K., Ann NY Acad Sci, 929, 123–146.
Singh, H., & Tononi, G. (2005). Breakdown of cortical Steriade, M. (2000). Corticothalamic resonance, states of
effective connectivity during sleep. Science, 309(5744), vigilance and mentation. Neuroscience, 101(2), 243–276.
2228–2232. Supp, G. G., Siegel, M., Hipp, J. F., & Engel, A. K. (2011).
Massimini, M., Ferrarelli, F., Murphy, M., Huber, R., Cortical hypersynchrony predicts breakdown of sensory
Riedner, B., Casarotto, S., & Tononi, G. (2010). Cortical processing during loss of consciousness. Curr Biol, 21(23),
reactivity and effective connectivity during REM sleep in 1988–1993.
humans. Cogn Neurosci, 1(3), 176–183. Tononi, G. (2004). An information integration theory of
McNamara, P., Johnson, P., McLaren, D., Harris, E., consciousness. BMC Neurosci, 5, 42.
Beauharnais, C., & Auerbach, S. (2010). REM and NREM Tononi, G. (2008). Consciousness as integrated information:
sleep mentation. Int Rev Neurobiol, 92, 69–86. A provisional manifesto. Biol Bull, 215(3), 216–242.
Mhuircheartaigh, R. N., Rosenorn-Lanng, D., Wise, R., Tononi, G., & Edelman, G. M. (1998). Consciousness and
Jbabdi, S., Rogers, R., & Tracey, I. (2010). Cortical complexity. Science, 282(5395), 1846–1851.
and subcortical connectivity changes during decreasing Tononi, G., Sporns, O., & Edelman, G. M. (1994). A measure
levels of consciousness in humans: A functional magnetic for brain complexity: Relating functional segregation and
resonance imaging study using propofol. J Neurosci, 30(27), integration in the nervous system. Proc Natl Acad Sci USA,
9095–9102. 91, 5033.
ABSTRACT The past fifteen years have provided an unprece- in a linear manner. For instance, every night when
dented collection of discoveries that bear upon our scientific falling asleep, we experience a decrease of the level of
understanding of consciousness in the human brain following wakefulness up to the point where we lose awareness of
severe brain damage. Highlighted among these discoveries
are unique demonstrations that patients with little or no
our environment. Clinical conditions resulting from
behavioral evidence of conscious awareness may retain critical severe brain injury, where the relationship between
cognitive capacities. These first scientific demonstrations wakefulness and awareness is violated, challenge our
support the possibility that some severely brain-injured understanding of how consciousness works in healthy
patients in long-standing conditions of limited behavioral situations.
responsiveness may nonetheless retain latent capacities for
awareness. Such capacities include the human functions of
language and higher-level cognition that, either spontane- Consciousness alterations after severe brain injury
ously or by thought-directed interventions, may reemerge
even at long time intervals or can remain unrecognized. Func- Brain Death Classically, brain death is caused by a
tional neuroimaging, such as positron emission tomography
and functional magnetic resonance imaging, as well as elec- massive brain lesion, such as trauma, intracranial hem-
troencephalography and evoked potential studies, have orrhage, or anoxia. The diagnostic guidelines for brain
offered the possibility to objectively approach covert cognitive death are (Quality Standards Subcommittee of the
processes in patients who are otherwise incapable of intelli- American Academy of Neurology, 1995):
gible or sustained behavioral expression. Such studies have
used experimental protocols to assess brain function during 1. demonstration of coma
resting-state conditions and after external stimulation. These 2. evidence for the cause of coma
technologies have further permitted the detection of nonver- 3. absence of confounding factors (hypothermia,
bal command-following and even established muscle- drugs, electrolyte, endocrine disturbances)
independent means of communication with some behaviorally
unresponsive patients. Such advances are expected to shed
4. absence of brainstem reflexes
light on the gray zones between the clinical entities of con- 5. absent motor responses
sciousness and help resolve medical and ethical controversies 6. positive apnea testing
around the management of such challenging situations. 7. a repeat evaluation in six hours (but the time
period is considered arbitrary)
8. confirmatory laboratory tests (only when specific
Defining consciousness from a clinical perspective components of the clinical testing cannot be reliably
evaluated)
Consciousness is a multifaceted term for which there is
no universal definition (Zeman, 2001). Clinical practice No recovery from brain death has ever been reported
dealing with patients with disorders of consciousness over the last 50 years in a patient fulfilling the above-
teaches that we can define consciousness by reducing it mentioned clinical criteria (Laureys, 2005).
to two components: wakefulness and awareness (Posner,
Saper, Schiff, & Plum, 2007). Clinically, the level of Coma Coma can result from bihemispheric diffuse
wakefulness can be gauged based on eye opening, cortical or white matter damage or bilateral brainstem
which may be absent, stimulus-induced, or spontaneous lesions, affecting the subcortical reticular arousing
and sustained. Awareness is more difficult to define and systems. Coma is a time-limited condition leading to
more challenging to assess behaviorally. At the clinical death, recovery of consciousness, or transition to a
level, we can only infer awareness by asking patients to vegetative state (Laureys, 2007). Many factors such as
follow simple commands or by observing nonreflex etiology, the patient’s general medical condition, age,
behaviors such as visual pursuit. The relationship clinical signs, and complimentary examinations influ-
between wakefulness and awareness can be described ence the management and prognosis of coma. In terms
demertzi and laureys: consciousness alterations after severe brain injury 811
of clinical signs, after three days of observation, a nega- in 2002 by the Aspen Workgroup to differentiate it from
tive outcome is heralded by absence of pupillary VS/UWS. According to the defining criteria (Giacino
or corneal reflexes, stereotyped or absent motor et al., 2002), patients in MCS manifest at least one of
response to noxious stimulation, absent bilateral corti- the following:
cal responses of somatosensory evoked potentials, and, 1. purposeful behavior (including movements or
for anoxic coma, biochemical markers (i.e., high levels affective behavior) contingent to relevant environment
of serum neuron-specific enolase) (Wijdicks, Hijdra, stimuli and not due to reflexive activity, such as visual
Young, Bassetti, & Wiebe, 2006). pursuit or sustained fixation occurring in direct
response to moving or salient stimuli, smiling or crying
Vegetative State, or Unresponsive Wakefulness in response to verbal or visual emotional but not neutral
Syndrome The vegetative state (VS) is usually caused stimuli, reaching for objects demonstrating a relation-
by diffuse lesions to gray and white matter. It can be a ship between object location and direction of reach,
transition to further recovery, or it may be permanent. touching or holding objects in a manner that accom-
This is the case for VS that lasts more than one year modates the size and shape of the object, and vocaliza-
after traumatic injury or three months after nontrau- tions or gestures occurring in direct response to the
matic injury. At present, there are no validated para- linguistic content of questions
clinical prognostic markers for individual patients 2. following simple commands
except that the chances of recovery depend on a 3. gestural or verbal yes/no response, regardless of
patient’s age, etiology, and time spent in this condition accuracy
(Multi-Society Task Force on PVS, 1994b). 4. intelligible verbalization
The VS is usually described as a “state of arousal
Like the VS/UWS, MCS may be chronic and sometimes
without awareness.” The criteria for the diagnosis of VS
permanent. Emergence from MCS is defined by the
are (Multi-Society Task Force on PVS, 1994a):
ability to exhibit functional interactive communication
1. no evidence of awareness of self or environment or functional use of objects.
and an inability to interact with others
2. no evidence of sustained, reproducible, purpose-
Locked-In Syndrome Locked-in syndrome (LIS) can
ful, or voluntary behavioral responses to visual, audi-
result from a bilateral ventral pontine lesion (Posner,
tory, tactile, or noxious stimuli
Saper, Schiff, & Plum, 2007b), but mesencephalic
3. no evidence of language comprehension or
lesions have also been reported (for a review, see
expression
Laureys et al., 2005). In LIS there is no dissociation
4. intermittent wakefulness manifested by the pres-
between arousal and awareness, but it is mentioned
ence of sleep-wake cycles
here since it can be misdiagnosed as a disorder of
5. sufficiently preserved hypothalamic and brain-
consciousness. According to the American Congress of
stem autonomic functions to permit survival with
Rehabilitation Medicine criteria (1995), LIS patients
medical and nursing care
demonstrate:
6. bowel and bladder incontinence
7. variably preserved cranial-nerve and spinal reflexes 1. sustained eye opening (bilateral ptosis should be
ruled out as a complicating factor)
More recently, it has been recognized that some in the
2. quadriplegia or quadriparesis
health care community, media, and lay public feel
3. aphonia or hypophonia
uncomfortable using the unintentionally denigrating
4. a primary mode of communication via vertical or
“vegetable-like” connotation, seemingly intrinsic to the
lateral eye movements or blinking of the upper eyelid
term VS. Hence, the European Task Force on Disorders
to signal yes/no responses.
of Consciousness proposed the alternative name “unre-
5. preserved cognitive abilities
sponsive wakefulness syndrome” (UWS), a more neutral
and descriptive term pertaining to patients showing a Based on motor capacities, LIS can be divided into
number of clinical signs of unresponsiveness (i.e., three categories (Bauer, Gerstenbrand, & Rumpl,
lacking response to commands or oriented voluntary 1979):
movements) in the presence of wakefulness (Laureys a. classic LIS, which is characterized by quadriplegia
et al., 2010). and anarthria with eye-coded communication
b. incomplete LIS, which is characterized by rem-
Minimally Conscious State The minimally conscious nants of voluntary responsiveness other than eye
state (MCS) was defined as a disorder of consciousness movements
812 consciousness
c. total LIS, which is characterized by complete immo- assess brain function during resting-state conditions
bility, including all eye movements, combined with pre- and after external sensory stimulation.
served consciousness
Once a LIS patient becomes medically stable and is Neuroimaging In resting conditions, when we do not
given appropriate medical care, life expectancy is esti- perform any task and receive no external stimulation,
mated to be up to several decades (Laureys et al., 2005). our brains are engaged in some kind of typical cognitive
Even if the chances of good motor recovery are very activity when the mind is unconstrained (e.g., Mason
limited, existing eye-controlled computer-based com- et al., 2007; Raichle & Snyder, 2007). We have recently
munication technologies currently allow these patients proposed to reduce the phenomenological complexity
to control their environment (Chatelle et al., 2012; Stoll of such cognitive process into two components: external
et al., 2013). awareness, namely, everything we perceive through our
senses, and internal awareness, or stimulus-independent
thoughts (Demertzi, Soddu, & Laureys, 2013). Internal
Detecting consciousness in severely brain-injured and external awareness have been shown to negatively
patients correlate both behaviorally and at the brain level (Van-
haudenhuyse et al., 2011). More particularly, when
Behavior The existing behavioral scales mainly focus healthy subjects were asked to independently self-rate
on deducing awareness of the environment. For their external and internal awareness as these were
example, the widely used Glasgow Coma Scale scores before an auditory prompt (and hence considered to
eye, verbal, and motor responses to external stimuli reflect a resting condition), they reported an anticor-
(Teasdale & Jennett, 1974). The Full Outline of Unre- related pattern between these two states. Interestingly,
sponsiveness (FOUR) scale recognizes the difficulty to the alternation between the external and internal
assess verbal responses, especially in cases of intubated milieu not only was found to characterize overt behav-
patients or patients with tracheotomy, and requires ioral reports but also had a cerebral correlate: behav-
patients to show nonverbal conscious behaviors such as ioral reports of internal awareness were linked to fMRI
eye blinking or hand signing to command (Wijdicks activity of midline anterior cingulate/mesiofrontal
et al., 2005). In that case, the FOUR can detect patients areas as well as posterior cingulate/precuneal cortices
with LIS but cannot always differentiate VS/UWS from classically coined as the default mode network (DMN).
MCS patients. The Revised Coma Recovery Scale Inversely, subjective ratings for external awareness
(CRS-R) is the most sensitive scale to differentiate MCS correlated with the activity of lateral fronto-parieto-
from unresponsive patients because it assesses auditory, temporal regions often linked to attentional processing
visual, motor, oromotor, and communication abilities (figure 70.1). These data suggest a coupling between
next to arousal assessment, covering the diagnostic cri- brain and behavior during the resting state and under-
teria for MCS (Giacino, Kalmar, & Whyte, 2004). Among score that certain brain areas seem to play an important
the existing scales, the CRS-R has been recommended role in sustaining consciousness (Heine et al., 2012).
as the most appropriate tool to evaluate patients with Indeed, in brain death, no DMN functional connectiv-
disorders of consciousness (Seel et al., 2010). ity could be identified (Boly et al., 2009; Soddu et al.,
To date, bedside evaluation is the gold standard for 2011). But alongside the spectrum of consciousness
diagnosing patients with consciousness impairments. impairment, ranging from healthy controls and patients
Nevertheless, incorrect diagnosis is not rare (Schnakers with LIS toward MCS, VS/UWS, and coma, functional
et al., 2009). In order to minimize misdiagnosis rate, connectivity is reduced in DMN (Vanhaudenhuyse
neuroimaging and electrophysiology procedures have et al., 2010) and other congitive-related networks
begun to assume an adjunctive role in the diagnostic (Demertzi et al., 2014). Such data suggest that fMRI
assessment of patients with disorders of consciousness. resting-state acquisitions could be used as an assisting
means to gain insight in patients’ diagnostic picture.
Assisting Technologies Functional neuroimaging, Using positron emission tomography, when patients
such as positron emission tomography and functional in VS/UWS were compared to healthy subjects in a
magnetic resonance imaging (fMRI) as well as electro- resting state, they were characterized by reduced levels
encephalography (EEG) and evoked potential studies, of global metabolism; nevertheless, recovery from VS/
have offered the possibility to objectively approach UWS did not coincide with resumption of global meta-
covert cognitive processes in patients who are otherwise bolic activity (Laureys, Owen, & Schiff, 2004). Rather,
incapable of intelligible or sustained behavioral expres- patients in VS/UWS showed impaired metabolism in a
sion. Such studies have used experimental protocols to widespread network encompassing midline and lateral
demertzi and laureys: consciousness alterations after severe brain injury 813
brain function in noncommunicating clinical condi-
tions. Preserved pain perception in MCS patients is
suggested by findings of cerebral correlates of pain
processing in a network similar to that in healthy con-
trols (Boly et al., 2008). The activation pattern observed
in patients in MCS was also much more widespread
than in unresponsive patients, suggesting a difference
in capacity for pain perception. The type of adminis-
tered stimuli also seems to make a difference to the
observed neural responses and therefore further assist
in the inference of awareness. Stimuli with emotional
valence, for example, such as infant cries and the
patient’s own name, induced a much more widespread
activation in patients in MCS than did meaningless
noise (Laureys, Perrin, et al., 2004). The activation
pattern was comparable with that previously obtained
in healthy controls. Patients also showed higher fMRI
activity in the anterior cingulate cortex after listening
to their own name as compared to listening to a famil-
iar name, and this activity correlated with the behavior-
Figure 70.1 Subjective ratings about awareness are coupled ally assessed level of consciousness of the patient (Qin
to the activity of the resting brain. Behavioral reports of et al., 2010). Such results imply that self-referential
increased internal awareness (blue areas) correlate with con- stimuli, like one’s own name, are attention-grabbing
nectivity in cerebral regions of the so-called default mode and therefore can be used in the assessment of resid-
network, encompassing anterior and posterior cingulate cor- ual brain function of these patients (Demertzi, Van-
tices. Inversely, when subjects report being more externally
oriented toward their environment, areas of a frontoparietal haudenhuyse et al., 2013).
attentional network (red areas) are recruited (adapted from
Vanhaudenhuyse et al., 2011). These data suggest that resting- Electrophysiology Various EEG paradigms have
state fMRI acquisitions could be an alternative paradigm to made an effort to differentiate between the clinical enti-
study residual brain function in human populations where ties of disorders of consciousness. Fifteen-minute EEG
communication is hindered (e.g., due to motor paralysis, lan-
guage comprehension, or lack of motivation), such as in
resting-state acquisitions showed that patients in VS/
patients with disorders of consciousness. (See color plate 60.) UWS had significantly higher correlated oscillations
than patients in MCS in the delta-frequency band
(Lehembre et al., 2012). Such delta-frequency activity
associative cortices compared to healthy controls is represented as a high-amplitude brain wave with an
(Laureys et al., 1999). Importantly, functional connec- oscillation between 0 and 4 Hz usually associated with
tions of these areas with the thalami were restored after the deepest stages of sleep, also known as slow-wave
recovery from VS/UWS (Laureys et al., 2000). It was sleep. As previously shown, power in the delta band also
recently shown that, compared to healthy controls, increases with severity of disorders of consciousness
patients in VS/UWS exhibit metabolic dysfunction in (Leon-Carrion, Martin-Rodriguez, Damas-Lopez, Barroso
both external and internal awareness networks as well y Martin, & Dominguez-Morales, 2008). Similarly, the
as in the thalami. In contrast, patients in MCS com- bispectral index, a measure of the depth of anesthesia,
pared to healthy controls showed dysfunction mostly in was shown to discriminate between unresponsive and
internal awareness network and thalami, which could patients in MCS (Schnakers, Majerus, & Laureys, 2005).
reflect an altered self-awareness in these patients that The bispectral index was also positively correlated with
is difficult to quantify at the bedside (Thibaut et al., behavioral scores of awareness at the time of testing and
2012). associated with outcome results at one year post-trauma.
Brain responses to external stimuli provide valuable Additionally, an EEG entropy score of 52 (value ranging
information not only about the preserved functional from 0 to 91, with higher scores indicating higher con-
(and to some degree anatomical) connectivity among sciousness level) was shown to be able to differentiate
distinct brain regions but also about the nature of acutely unconscious from patients in MCS with 89%
detected responses. Pain is among the most robust sensitivity and 90% specificity (Gosseries et al., 2011).
stimulations allowing for the investigation of residual Other efforts have also been made to use EEG signal
814 consciousness
patterns as a prognostic tool for these patients. For
example, it has been observed that patients in VS/UWS
who made a behavioral recovery, at three-month follow-
up they showed higher occipital source power in the
alpha band of resting EEG when compared to those
who did not (Babiloni et al., 2009). Normally, high
power of prestimulus cortical alpha rhythms (about
8–12 Hz) underlies conscious perception in healthy
subjects. As such, cortical sources of resting alpha
rhythms might predict recovery in VS/UWS patients.
EEG studies measuring effective connectivity also
seem to be able to differentiate between unresponsive
patients and those in MCS. Effective connectivity is a
measure of the causal relationship between brain areas.
One study using a mismatch negativity paradigm and
applying dynamic causal modeling found that the only
significant difference in functional connectivity between
VS/UWS and MCS was an impairment of backward Figure 70.2 Muscle-independent fMRI communication with
connectivity from frontal to temporal cortices (Boly a patient clinically diagnosed as “vegetative”/unresponsive.
Following a certain communication protocol, where the
et al., 2011). In addition, measurement of EEG effective
instruction to imagine playing tennis was used as a coded “yes”
connectivity after the application of transcranial mag- response (encompassing supplementary motor area; upper
netic stimulation (TMS) revealed that unresponsive panel in the illustration) and the command to imagine navi-
patients showed a simple, local response after the TMS gating in one’s house corresponded to a “no” answer (para-
pulses. In contrast, patients in MCS showed more hippocampal area; lower panel in the illustration), this patient
was able to willfully modulate the brain activity and provide
complex activations after the TMS pulses, which
responses to a series of autobiographical questions (adapted
involved distant cortical areas ipsilateral and contralat- from Monti et al., 2010).
eral to the site of stimulation (Rosanova et al., 2012).
Importantly, an EEG-TMS objective index has recently
been proposed allowing for accurate patient classifica- around their house, and one patient was able to use the
tion at the single-subject level (Casali et al., 2013). modulation of brain activity to answer reliably “yes” or
“no” to simple questions, even though no signs of com-
Functional Neuroimaging and Electrophysiology munication had ever been found through bedside
as Tools for Command-Following and Communica- examination (Monti et al., 2010). The automated user-
tion A willful modulation of brain activity to a certain independent analysis of the acquired fMRI data classi-
command can be an alternative way to evidence aware- fied the brain’s responses as a “yes” or “no” answer to a
ness in the absence of motor output. Using fMRI, it was series of simple questions (figure 70.2).
demonstrated that a posttraumatic patient behaviorally Similarly, active paradigms have been attempted with
diagnosed as in VS/UWS could repeatedly and for a cheaper and portable EEG-based technologies. In the
prolonged time follow commands when assessed using case of a young comatose woman who failed to show
a novel fMRI paradigm of mental imagery (Owen et al., any motor signs of conscious awareness, only EEG-
2006). When asked to imagine playing tennis and evoked potential based on command-following allowed
walking through her house, the patient activated the the diagnosis of total LIS at the intensive care unit
supplementary motor area and parahippocampal areas, (Schnakers et al., 2008). The task was to count a target
respectively. These specific activation patterns were not name or her own name in a list of other names. A previ-
different from those previously observed in a cohort of ous study using this task demonstrated that while most
healthy volunteers (Boly et al., 2007). Such a command- patients in MCS exhibited increased amplitude of the
following paradigm has been further developed to be P300 when instructed to count, no task-related P300
implemented as a communication system. In such a changes were observed in unresponsive patients. A
protocol, differentiation between brain responses was number of studies have demonstrated that EEG power
used as a proxy for behavioral responses. Using the spectral analysis can be also used as a flexible bedside
previously described “tennis paradigm,” it was demon- tool to demonstrate awareness in brain-injured patients
strated that of 54 patients, 5 were able to follow the who are otherwise unable to communicate (Chatelle
commands to imagine playing tennis and navigating et al., 2012; Cruse et al., 2011). What remains to be
demertzi and laureys: consciousness alterations after severe brain injury 815
shown is whether such technologies can be used as health care workers, those who are more likely to inter-
evidence of the expressed will of a competent patient act with patients?
(Gantner, Bodart, Laureys, & Demertzi, 2012). For When clinicians were asked to express their opinions
example, how can a negative response of an “unrespon- on possible pain perception in VS/UWS, a significant
sive” patient to the question of whether he or she wants number of medical doctors ascribed pain perception in
to continue to live be considered a reliable response to VS/UWS (56%) despite formal guidelines suggesting
be respected? Similarly, should pain treatment in a the opposite (e.g., The Multi-Society Task Force on
patient in MCS change once he or she has communi- PVS, 1994b). For MCS, there was no discrepancy in
cated that he or she is suffering? Should proving con- opinions, and the majority (97%) of respondents
sciousness in these patients be considered a piece of thought that MCS patients feel pain (Demertzi, Racine
evidence to be celebrated, or can it work against et al., 2013; Demertzi et al., 2009), in line with neuro-
patients’ and families’ best interests (Jox, Bernat, imaging data strongly suggesting preserved pain per-
Laureys, & Racine, 2012)? These aforementioned ques- ception in MCS (Boly et al., 2008). The issue of pain
tions require answers that future establishment of management in unresponsive patients becomes more
ethical and legal provisions can provide. challenging when withdrawal from life-supporting treat-
ments, such as artificial nutrition and hydration, has
been agreed upon. In a wide survey around Europe, we
Medico-ethical implications of dealing with showed that health care workers’ opinions on end of
patients with disorders of consciousness life in disorders of consciousness differed depending
on the diagnosis (i.e., respondents supported treatment
Ever since disorders of consciousness appeared in the withdrawal more often for patients in VS/UWS than in
clinical setting, clinicians, scholars, theologians, and MCS), professional background (i.e., when physicians
ethicists have wondered what it is like to be in a state imagined being in MCS, they preferred more often to
of profoundly disturbed consciousness (e.g., Thomp- be kept alive compared to paramedical professionals),
son, 1969). Such controversies mainly stem from how region of origin (i.e., Northern Europeans agreed with
different people regard indefinite survival in disorders treatment withdrawal more often compared to Central
of consciousness (Demertzi, Laureys, & Bruno, 2011; and Southern European respondents), and religious
Jennett, 2002). Despite the general view that quality of beliefs (i.e., religious respondents agreed less with treat-
life is diminished in disease as a result of limited capaci- ment limitation in both VS/UWS and MCS compared
ties to functionally engage in everyday living, one needs to nonreligious respondents; Demertzi, Ledoux, et al.,
to consider that these attitudes are formulated from a 2011). These data show that personal opinions about
third-person perspective. Consequently, only rough esti- ethical issues in disorders of consciousness differ, and
mations about what it is like to be in such a situation hence different clinical practice can be expected. For
can be made with the risk of imminent bias. Such bias example, in the European survey, the majority of par-
could be attributed to the fact that patients’ quality of ticipants approved of stopping treatment in VS/UWS
life evaluations are made from the perspective of healthy (66%), much more than in MCS (28%). In this case,
individuals who tend to underestimate patients’ subjec- patients in VS/UWS may run the risk of being left
tive well-being (Demertzi, Gosseries, Ledoux, Laureys, without administration of opioids or other analgesic
& Bruno, 2013a; Nizzi et al., 2012). Indeed, it was drugs during their dying process (Fins, 2006; Laureys,
recently showed that patients in LIS expressed a posi- 2005) on the grounds that they are unable to experi-
tive subjective quality of life, contrary to what could be ence suffering due to hunger or thirst.
expected in this condition (Bruno et al., 2011). In this From such studies on clinicians’ attitudes and atti-
self-reporting survey, it was shown that the majority of tudes of patients’ families (e.g., Kuehlmeyer, Borasio, &
patients in chronic LIS, despite mentioning severe Jox, 2012), it becomes evident that medical and ethical
restrictions in community reintegration, professed controversies continue to exist for patients with disor-
good subjective well-being. Self-reported happiness ders of consciousness. In order to resolve them, at least
status was associated with longer duration in this condi- to a certain degree, we need to improve our current
tion, the ability to produce speech via assisting tech- understanding of how these patients function. The use
nologies, and lower rates of anxiety. These findings of objective biomarkers may help us to better deter-
suggest that healthy persons who are not in direct mine the differences in underlying pathophysiology
contact with this patient population can have distorted characterizing the clinical entities of consciousness.
pictures about what life is like in these severely con- Consequently, clinicians should learn about patients’
strained situations. But what about the opinions of values and preferences and focus attention on changes
816 consciousness
in patient status, keeping the patients’ best interests in Demertzi, A., Gómez, F., Crone, J. S., Vanhaudenhuyse, A.,
mind (Jox et al., 2012). Tshibanda, L., & Noirhomme, Q. (2014). Multiple fMRI
system-level baseline connectivity is disrupted in patients
ACKNOWLEDGMENTS This work was supported by the Belgian
with consciousness alterations. Cortex, 52, 35–46.
National Funds for Scientific Research (FNRS), the European Demertzi, A., Gosseries, O., Ledoux, D., Laureys, S., &
Commission, the James McDonnell Foundation, the Euro- Bruno, M.-A. (2013). Quality of life and end-of-life deci-
pean Space Agency, Mind Science Foundation, the French- sions after brain injury. In N. Warren & L. Manderson
Speaking Community Concerted Research Action, the Public (Eds.), Reframing disability and quality of life: A global perspec-
Utility Foundation, Université Européenne du Travail, Fon- tive (pp. 95–110). Dordrecht: Springer.
dazione Europea di Ricerca Biomedica, and the University Demertzi, A., Laureys, S., & Bruno, M.-A. (2011). The ethics
and University Hospital of Liège. in disorders of consciousness. In J. L. Vincent (Ed.), Annual
update in intensive care and emergency medicine (pp. 675–682).
Berlin: Springer-Verlag.
Demertzi, A., Ledoux, D., Bruno, M.-A., Vanhaudenhuyse,
REFERENCES
A., Gosseries, O., Soddu, A., … Laureys, S. (2011). Atti-
American Congress of Rehabilitation Medicine. (1995). tudes towards end-of-life issues in disorders of conscious-
Recommendations for use of uniform nomenclature perti- ness: A European survey. J Neurol, 258(6), 1058–1065.
nent to patients with severe alterations of consciousness. Demertzi, A., Racine, E., Bruno, M. A., Ledoux, D.,
Arch Phys Med Rehabil, 76(2), 205–209. Gosseries, O., Vanhaudenhuyse, A., … Laureys, S.
Babiloni, C., Sara, M., Vecchio, F., Pistoia, F., (2013). Pain perception in disorders of consciousness: Neu-
Sebastiano, F., Onorati, P., … Rossini, P. M. (2009). Cor- roscience, clinical care, and ethics in dialogue. Neuroethics,
tical sources of resting-state alpha rhythms are abnormal in 6(1), 37–50.
persistent vegetative state patients. Clin Neurophysiol, 120(4), Demertzi, A., Schnakers, C., Ledoux, D., Chatelle, C.,
719–729. Bruno, M.-A., Vanhaudenhuyse, A., … Laureys, S. (2009).
Bauer, G., Gerstenbrand, F., & Rumpl, E. (1979). Varieties Different beliefs about pain perception in the vegetative
of the locked-in syndrome. J Neurol, 221(2), 77–91. and minimally conscious states: A European survey of
Boly, M., Coleman, M. R., Davis, M. H., Hampshire, A., medical and paramedical professionals. Prog Brain Res, 177,
Bor, D., Moonen, G., … Owen, A. M. (2007). When 329–338.
thoughts become action: An fMRI paradigm to study voli- Demertzi, A., Soddu, A., & Laureys, S. (2013). Conscious-
tional brain activity in non-communicative brain injured ness supporting networks. Curr Opin Neurobiol, 23(2),
patients. NeuroImage, 36(3), 979–992. 239–244.
Boly, M., Faymonville, M.-E., Schnakers, C., Peigneux, P., Demertzi, A., Vanhaudenhuyse, A., Bredart, S., Heine, L.,
Lambermont, B., Phillips, C., … Laureys, S. (2008). Per- di Perri, C., & Laureys, S. (2013). Looking for the self
ception of pain in the minimally conscious state with PET in pathological unconsciousness. Front Hum Neurosci, 7,
activation: An observational study. Lancet Neurol, 7(11), 538.
1013–1020. Fins, J. J. (2006). Affirming the right to care, preserving the
Boly, M., Garrido, M. I., Gosseries, O., Bruno, M.-A., right to die: Disorders of consciousness and neuroethics
Boveroux, P., Schnakers, C., … Friston, K. (2011). Pre- after Schiavo. Palliat Support Care, 4(2), 169–178.
served feedforward but impaired top-down processes in the Gantner, I. S., Bodart, O., Laureys, S., & Demertzi, A.
vegetative state. Science, 332(6031), 858–862. (2012). Our rapidly changing understanding of acute and
Boly, M., Tshibanda, L., Vanhaudenhuyse, A., Noirhomme, chronic disorders of consciousness: Challenges for neurolo-
Q., Schnakers, C., Ledoux, D., … Laureys, S. (2009). gists. Fut Neurol, 8(1), 43–54.
Functional connectivity in the default network during Giacino, J. T., Ashwal, S., Childs, N., Cranford, R.,
resting state is preserved in a vegetative but not in a brain Jennett, B., Katz, D. I., … Zasler, N. D. (2002). The
dead patient. Hum Brain Mapp, 30, 2393–2400. minimally conscious state: Definition and diagnostic crite-
Bruno, M.-A., Bernheim, J., Ledoux, D., Pellas, F., ria. Neurology, 58(3), 349–353.
Demertzi, A., & Laureys, S. (2011). A survey on self- Giacino, J. T., Kalmar, K., & Whyte, J. (2004). The JFK
assessed well-being in a cohort of chronic locked-in syn- coma recovery scale-revised: Measurement characteristics
drome patients: Happy majority, miserable minority. BMJ and diagnostic utility. Arch Phys Med Rehab, 85(12),
Open, 1(1), e000039. 2020–2029.
Casali, A. G., Gosseries, O., Rosanova, M., Boly, M., Gosseries, O., Schnakers, C., Ledoux, D., Vanhaudenhuyse,
Sarasso, S., Casali, K. R., … Massimini, M. (2013). A A., Bruno, M.-A., Demertzi, A., … Laureys, S. (2011).
theoretically based index of consciousness independent of Automated EEG entropy measurements in coma, vegetative
sensory processing and behavior. Sci Transl Med, 5(198), state/unresponsive wakefulness syndrome and minimally
198ra105. conscious state. Funct Neurol, 36(1), 25–30.
Chatelle, C., Chennu, S., Noirhomme, Q., Cruse, D., Heine, L., Soddu, A., Gomez, F., Vanhaudenhuyse, A.,
Owen, A. M., & Laureys, S. (2012). Brain-computer inter- Tshibanda, L., Thonnard, M., … Demertzi, A. (2012).
facing in disorders of consciousness. Brain Inj, 26(12), Resting state networks and consciousness: Alterations of
1510–1522. multiple resting state network connectivity in physiological,
Cruse, D., Chennu, S., Chatelle, C., Bekinschtein, T. A., pharmacological and pathological consciousness states.
Fernandez-Espejo, D., Pickard, J. D., … Owen, A. M. Front Psychol, 3, 1–12.
(2011). Bedside detection of awareness in the vegetative Jennett, B. (2002). Attitudes to the permanent vegetative
state: A cohort study. Lancet, 378(9809), 2088–2094. state. In B. Jennett (Ed.), The vegetative state: Medical facts,
demertzi and laureys: consciousness alterations after severe brain injury 817
ethical and legal dilemmas (pp. 97–125). Cambridge, UK: Nizzi, M. C., Demertzi, A., Gosseries, O., Bruno, M.-A.,
Cambridge University Press. Jouen, F., & Laureys, S. (2012). From armchair to wheel-
Jox, R. J., Bernat, J. L., Laureys, S., & Racine, E. (2012). chair: How patients with a locked-in syndrome integrate
Disorders of consciousness: Responding to requests for bodily changes in experienced identity. Conscious Cogn,
novel diagnostic and therapeutic interventions. Lancet 21(1), 431–437.
Neurol, 11(8), 732–738. Owen, A. M., Coleman, M. R., Boly, M., Davis, M. H.,
Kuehlmeyer, K., Borasio, G. D., & Jox, R. J. (2012). How Laureys, S., & Pickard, J. D. (2006). Detecting awareness
family caregivers’ medical and moral assumptions influence in the vegetative state. Science, 313(5792), 1402.
decision making for patients in the vegetative state: A quali- Posner, J., Saper, C., Schiff, N. D., & Plum, F. (Eds). (2007).
tative interview study. J Med Ethics, 38(6), 332–337. Plum and Posner’s diagnosis of stupor and coma. New York, NY:
Laureys, S. (2005). Science and society: Death, unconscious- Oxford University Press.
ness and the brain. Nat Rev Neurosci, 6(11), 899–909. Qin, P., Di, H., Liu, Y., Yu, S., Gong, Q., Duncan, N., …
Laureys, S. (2007). Eyes open, brain shut. Sci Am, 296(5), Northoff, G. (2010). Anterior cingulate activity and the
84–89. self in disorders of consciousness. Hum Brain Mapp, 31(12),
Laureys, S., Celesia, G., Cohadon, F., Lavrijsen, J., 1993–2002.
Leon-Carrrion, J., Sannita, W. G., … European Task Quality Standards Subcommittee of the American Academy
Force on Disorders of Consciousness. (2010). Unresponsive of Neurology. (1995). Practice parameters for determining
wakefulness syndrome: A new name for the vegetative state brain death in adults (summary statement). Neurology,
or apallic syndrome. BMC Med, 8(1), 68. 45(5), 1012–1014.
Laureys, S., Faymonville, M.-E., Luxen, A., Lamy, M., Raichle, M. E., & Snyder, A. Z. (2007). A default mode of
Franck, G., & Maquet, P. (2000). Restoration of thalamo- brain function: A brief history of an evolving idea. NeuroIm-
cortical connectivity after recovery from persistent vegeta- age, 37(4), 1083–1090; discussion 1097–1099.
tive state. Lancet, 355(9217), 1790–1791. Rosanova, M., Gosseries, O., Casarotto, S., Boly, M.,
Laureys, S., Goldman, S., Phillips, C., van Bogaert, P., Casali, A. G., Bruno, M.-A., … Massimini, M. (2012).
Aerts, J., Luxen, A., … Maquet, P. (1999). Impaired effec- Recovery of cortical effective connectivity and recovery of
tive cortical connectivity in vegetative state: Preliminary consciousness in vegetative patients. Brain, 135(Pt 4),
investigation using PET. NeuroImage, 9(4), 377–382. 1308–1320.
Laureys, S., Owen, A. M., & Schiff, N. D. (2004). Brain Schnakers, C., Majerus, S., & Laureys, S. (2005). Bispectral
function in coma, vegetative state, and related disorders. analysis of electroencephalogram signals during recovery
Lancet Neurol, 3(9), 537–546. from coma. Neuropsychol Rehabil, 15(3–4), 381–388.
Laureys, S., Pellas, F., van Eeckhout, P., Ghorbel, S., Schnakers, C., Perrin, F., Schabus, M., Majerus, S., Ledoux,
Schnakers, C., Perrin, F., … Goldman, S. (2005). The D., Damas, P., … Laureys, S. (2008). Voluntary brain pro-
locked-in syndrome : What is it like to be conscious but cessing in disorders of consciousness. Neurology, 71(20),
paralyzed and voiceless? Prog Brain Res, 150, 495–511. 1614–1620.
Laureys, S., Perrin, F., Faymonville, M.-E., Schnakers, C., Schnakers, C., Vanhaudenhuyse, A., Giacino, J. T.,
Boly, M., Bartsch, V., … Maquet, P. (2004). Cerebral Ventura, M., Boly, M., Majerus, S., … Laureys, S. (2009).
processing in the minimally conscious state. Neurology, Diagnostic accuracy of the vegetative and minimally con-
63(5), 916–918. scious state: Clinical consensus versus standardized neu-
Lehembre, R., Bruno, M.-A., Vanhaudenhuyse, A., robehavioral assessment. BMC Neurol, 9, 35.
Chatelle, C., Cologan, V., Leclercq, Y., … Noirhomme, Q. Seel, R. T., Sherer, M., Whyte, J., Katz, D. I., Giacino, J. T.,
(2012). Resting-state EEG study of comatose patients: A Rosenbaum, A. M., … Zasler, N. (2010). Assessment scales
connectivity and frequency analysis to find differences for disorders of consciousness: Evidence-based recommen-
between vegetative and minimally conscious states. Funct dations for clinical practice and research. Arch Phys Med
Neurol, 27(1), 41–47. Rehabil, 91(12), 1795–1813.
Leon-Carrion, J., Martin-Rodriguez, J. F., Damas-Lopez, J., Soddu, A., Vanhaudenhuyse, A., Demertzi, A., Bruno, M.-A.,
Barroso y Martin, J. M., & Dominguez-Morales, M. R. Tshibanda, L., Di, H., … Noirhomme, Q. (2011). Resting
(2008). Brain function in the minimally conscious state: A state activity in patients with disorders of consciousness.
quantitative neurophysiological study. Clin Neurophysiol, Funct Neurol, 26(1), 37–43.
119(7), 1506–1514. Stoll, J., Chatelle, C., Carter, O., Koch, C., Laureys, S.,
Mason, M. F., Norton, M. I., van Horn, J. D., Wegner, D. & Einhauser, W. (2013). Pupil responses allow communi-
M., Grafton, S. T., & Macrae, C. N. (2007). Wandering cation in locked-in syndrome patients. Curr Biol, 23(15),
minds: The default network and stimulus-independent R647–648.
thought. Science, 315(5810), 393–395. Teasdale, G., & Jennett, B. (1974). Assessment of coma and
Monti, M. M., Vanhaudenhuyse, A., Coleman, M. R., impaired consciousness. A practical scale. Lancet, 2(7872),
Boly, M., Pickard, J. D., Tshibanda, L., … Laureys, S. 81–84.
(2010). Willful modulation of brain activity in disorders of Thibaut, A., Bruno, M.-A., Chatelle, C., Gosseries, O.,
consciousness. N Engl J Med, 362(7), 579–589. Vanhaudenhuyse, A., Demertzi, A., … Laureys, S. (2012).
Multi-Society Task Force on PVS. (1994a). Medical aspects of Metabolic activity in external and internal awareness net-
the persistent vegetative state (1). N Engl J Med, 330(21), works in severely brain-damaged patients. J Rehabil Med,
1499–1508. 44(5), 487–494.
Multi-Society Task Force on PVS. (1994b). Medical aspects of Thompson, G. T. (1969). An appeal to doctors. Lancet, 2, 1353.
the persistent vegetative state (2). N Engl J Med, 330(22), Vanhaudenhuyse, A., Demertzi, A., Schabus, M.,
1572–1579. Noirhomme, Q., Bredart, S., Boly, M., … Laureys, S.
818 consciousness
(2011). Two distinct neuronal networks mediate the aware- a new coma scale: The FOUR score. Ann Neurol, 58(4),
ness of environment and of self. J Cogn Neurosci, 23(3), 585–593.
570–578. Wijdicks, E. F. M., Hijdra, A., Young, G. B., Bassetti, C. L.,
Vanhaudenhuyse, A., Noirhomme, Q., Tshibanda, L. J., & Wiebe, S. (2006). Practice parameter: Prediction of
Bruno, M.-A., Boveroux, P., Schnakers, C., … Boly, M. outcome in comatose survivors after cardiopulmonary
(2010). Default network connectivity reflects the level of resuscitation (an evidence-based review): Report of the
consciousness in non-communicative brain-damaged Quality Standards Subcommittee of the American Academy
patients. Brain, 133(Pt. 1), 161–171. of Neurology. Neurology, 67(2), 203–210.
Wijdicks, E. F. M., Bamlet, W. R., Maramattom, B. V., Zeman, A. (2001). Consciousness. Brain, 124(7), 1263–1289.
Manno, E. M., & McClelland, R. L. (2005). Validation of
demertzi and laureys: consciousness alterations after severe brain injury 819
71 Consciousness and Seizures
HAL BLUMENFELD
ABSTRACT Why do seizures temporarily interrupt conscious- and temporal lobe complex partial seizures. Although
ness? To answer this question, recent studies have combined these seizures differ from each other markedly in their
behavioral, electrophysiological, and neuroimaging measure-
physiology and behavioral features, they converge on a
ments to explore brain networks crucial for both normal
consciousness and its disruption by seizures. The most common set of neuroanatomical structures when con-
common seizure types leading to impaired consciousness are sciousness is impaired (table 71.1). These include the
absence seizures, generalized tonic-clonic seizures, and tem- frontoparietal association cortex and subcortical arousal
poral lobe complex partial seizures. Although these seizure networks, often affected in other disorders of conscious-
types differ in many ways, they share a final common set of ness as well (Laureys & Schiff, 2009; Laureys & Tononi,
anatomical structures leading to impaired consciousness.
These structures, which can be referred to as the “conscious- 2008). These key structures, which can be referred to
ness system,” comprise regions well known to play a role in as the “consciousness system” (Blumenfeld, 2009, 2010),
controlling the overall level of conscious arousal, including are affected through different mechanisms in different
the upper brainstem activating systems, thalamus, basal fore- epileptic seizures, which may reflect important aspects
brain, and higher-order frontoparietal association cortex. of normal brain network function.
Interestingly, different seizures affect these same structures
in different ways. Absence seizures lead to abnormal
This chapter will first introduce the consciousness
increased activity in the thalamus, with a mixture of increased system, its main components, and how it is involved
and decreased activity in the frontoparietal cortex. General- in disorders of consciousness, including epilepsy. The
ized tonic-clonic seizures are accompanied by increases in discussion will next focus on recent advances in behav-
activity in many brain regions, but following seizures while ioral, electrophysiological, and neuroimaging tech-
consciousness is still impaired there is decreased activity in
niques that have shed new light on the mechanisms of
frontoparietal cortex associated with persistently increased
activity in the cerebellum. Temporal lobe complex partial impaired consciousness in epileptic seizures (table
seizures exhibit an interesting pattern of increased activity in 71.1). Hopefully, with further work, we will soon be able
the temporal lobe and thalamus, along with decreased activ- to prevent this devastating interruption in the lives of
ity in frontoparietal cortex resembling slow-wave sleep. people with epilepsy and provide them the full benefits
Human studies and animal models support a model in which
of normal consciousness.
temporal lobe seizures inhibit the subcortical arousal systems
provoking a transition to sleep-like cortical activity. Better
understanding of impaired consciousness in seizures may
shed light on both normal consciousness mechanisms and The consciousness system
on potential new treatment approaches for people living
with epilepsy. Although philosophical discussions of consciousness
can be enlightening (Baars, Ramsoy, & Laureys, 2003;
Transient impaired consciousness in epileptic seizures Chalmers, 1996; Dennett, 1991; Nagel, 1974; Searle,
provides a window into mechanisms of normal con- 1997), neurologists tend to think of consciousness and
sciousness. Brain circuits operating in optimal fashion its disruption as a problem in functional neuroanatomi-
generate consciousness, but are disrupted by specific cal localization. In studying coma and related disorders,
anatomical and physiological alterations during epilep- Plum and Posner introduced a classic distinction
tic seizures. Understanding these mechanisms has enor- between brain systems that control the level of conscious-
mous importance for patients with epilepsy, since ness and those that generate the content of consciousness
transient loss of consciousness has a negative impact on (Plum & Posner, 1972, 1982). The content of conscious-
safety, productivity, emotional health, and quality of life ness includes all of the hierarchically organized sensory
(Sperling, 2004; Vickrey et al., 2000). New insights and motor systems, memory, and emotions and drives.
gained from understanding impaired consciousness in Individual brain networks with specialized functions
epilepsy may also provide novel treatment avenues for each contribute to the content of consciousness. The
other disorders of consciousness. level of consciousness depends on other specialized
Several types of seizures can cause impaired con- brain systems that control whether we are alert, atten-
sciousness, including absence, generalized tonic-clonic, tive, and aware (mnemonic: AAA; Blumenfeld, 2010).
Our level of consciousness affects each of the many lateral temporal-parietal association cortex on the
contents of consciousness. lateral surface (figure 71.1B). Portions of the insula
The specialized brain networks that regulate the level (not shown) also likely participate. Subcortical compo-
of consciousness have been studied extensively over the nents include the basal forebrain, hypothalamus, thala-
past century. Early work on human brain disorders mus, and upper brainstem activating systems (figure
(Penfield, 1950; Von Economo, 1930) and experimen- 71.1A), as well as portions of the basal ganglia, cerebel-
tal animal models (Bremer, 1955; Moruzzi & Magoun, lum, and amygdala (not shown). The anatomy of the
1949) has been complemented by more recent studies consciousness system was discussed in greater detail in a
demonstrating the importance of both cortical and sub- recent review (Blumenfeld, 2012). Of note, the cortical
cortical structures in controlling level of consciousness components of the consciousness system include both
(Steriade & McCarley, 2010). In analogy to sensory, the recently described default mode network, which is
motor, and other cortical-subcortical brain systems, the important for internally directed processing (Raichle
specialized structures regulating level of consciousness et al., 2001), as well as other cortical regions important
can be called the “consciousness system” (Blumenfeld, for externally oriented attention (Asplund, Todd,
2009, 2010; figure 71.1). Snyder, & Marois, 2010; Buschman & Miller, 2007;
The consciousness system includes cortical compo- Dosenbach et al., 2007; Vanhaudenhuyse et al., 2011).
nents such as the medial frontal, anterior cingulate, Impaired consciousness in epilepsy and other disor-
posterior cingulate, and medial parietal (precuneus, ders occurs when the systems providing either the
retrosplenial) cortex on the medial surface (figure content or level of consciousness are disturbed. Selec-
71.1A), as well as the lateral frontal, orbital frontal, and tive loss of individual systems serving the content of
822 consciousness
Figure 71.1 The consciousness system. Anatomical struc- components (shown in red) include the basal forebrain,
tures known to regulate the level of consciousness. (A) Medial hypothalamus, thalamus, and upper brainstem activating
view. (B) Lateral view. Cortical components of the conscious- systems. Note that other circuits, such as the basal ganglia and
ness system (shown in blue) include the medial and lateral cerebellum, may also participate in attention and other
frontoparietal association cortex, anterior and posterior cin- aspects of consciousness. (Reproduced with permission from
gulate, precuneus, and retrosplenial cortex. Subcortical Blumenfeld, 2010.)
consciousness, such as a lesion in the visual cortex or been called simple partial seizures, while those with
in the language areas, causes specific deficits in that impaired consciousness are called complex partial sei-
aspect of consciousness but is not usually considered a zures. Generalized seizures can also have variable mani-
disorder of consciousness per se. On the other hand, if festations depending on the intensity, duration, and
many or nearly all contents of consciousness are physiological pattern of the epileptic discharges.
impaired together, this represents a disorder of con- The most common generalized seizure types are gener-
sciousness. Most commonly, this occurs when there is alized tonic-clonic seizures and absence seizures,
dysfunction in the consciousness system (figure 71.1). both of which usually cause marked impairment of
Disorders of the consciousness system cause an impaired consciousness. Because absence, generalized tonic-
level of consciousness, which leads to widespread dys- clonic, and complex partial seizures are the major
function in most contents of consciousness. Examples seizure types associated with impaired consciousness,
include coma, vegetative state, minimally conscious the remainder of our discussion will focus on these
state (Laureys, Owen, & Schiff, 2004; Laureys & Schiff, three types of seizures (table 71.1). Interestingly, despite
2009; Laureys & Tononi, 2008), and certain types of the differences between these three seizure types, all
epileptic seizures discussed in the next section. converge on a final common set of anatomical regions,
leading to impaired consciousness (table 71.1).
Seizures associated with impaired consciousness Absence (petit mal) seizures have been recognized in
the medical literature since the 1700s (Temkin, 1971)
Seizures are classified into those that are partial and consist of brief episodes of staring and unrespon-
(focal)—involving more localized areas of the brain— siveness. Most commonly seen in childhood absence
and those that are generalized—affecting widespread epilepsy, absence seizures can also occur in adolescents
regions bilaterally (Berg et al., 2010; International and adults. The electroencephalogram (EEG) shows a
League Against Epilepsy, 1981, 1989). Not all seizures characteristic widespread bilateral 3–4 Hz spike-wave
cause impaired consciousness. Focal seizures may cause discharge, which begins and ends abruptly along with
localized symptoms, such as hand twitching or an the absence behavioral event. Although classified as a
odd sensation in the epigastric area, without alterations form of generalized epilepsy, recent work suggests that
in overall alertness, attention, or awareness. Partial sei- absence seizures involve selective bilateral cortical and
zures without impaired consciousness have traditionally subcortical networks while sparing others (Blumenfeld,
824 consciousness
consciousness, such as global encephalopathy, which that spike-wave discharges lasting less than three seconds
classically impair higher-order functions more severely do not cause deficits, and although this may be true for
than simple tasks (Mesulam, 2000). clinically obvious absence seizures, with careful behav-
Another factor that influences behavioral impair- ioral testing transient deficits can be detected for even
ment during absence seizures is the precise timing of brief episodes lasting less than one second (Berman
tasks relative to seizure onset and end. There is often et al., 2010; Browne, Penry, Porter, & Dreifuss, 1974).
some recovery toward the end of seizures, and depend- In addition to deficits during seizures, it has been shown
ing on the task there may also be some initial sparing that patients with childhood absence epilepsy often
just after seizure onset, resulting in a U-shaped time have significant attention deficits in the interictal period
course of deficits (figure 71.2A). It has been claimed even when no spike-wave discharges are occurring
(Killory et al., 2011; Levav et al., 2002; Mirsky & van
Buren, 1965; Vega et al., 2010).
Generalized tonic-clonic seizures usually last for
about two minutes, with profound impairment of con-
sciousness during seizures and for a variable time in the
post-ictal period. Onset may be focal or bilateral, and
behavior progresses through a series of stages, includ-
ing bilateral clonic, tonic, vibratory, and clonic activity,
followed by post-ictal lethargy (Blumenfeld et al., 2009;
Jobst, 2001; Theodore et al., 1994; Varghese et al.,
2009). During and following generalized tonic-clonic
seizures, patients are deeply unresponsive to even basic
tasks such as ball grasp, visual tracking, or blink to visual
threat (McPherson et al., 2012). Amnesia commonly
occurs for events around the time of seizures. Interest-
ingly, in rare cases patients remain conscious during
generalized tonic-clonic seizures and can reliably
describe their experiences afterwards (Bell, Walczak,
Shin, & Radtke, 1997; Botez, Serbanescu, & Stoica,
1966; Weinberger & Lusins, 1973). The mechanisms for
this spared function should be investigated further, but
it has been speculated that such seizures may involve
bilateral frontal regions while sparing other areas neces-
sary for consciousness.
826 consciousness
strongly than others. For example, secondarily general- thalamocortical synchrony, which may contribute to
ized tonic-clonic seizures show some EEG asymmetry these physiological changes (Arthuis et al., 2009; Bar-
even during the generalized phase (Kriss, Halliday, Hal- tolomei, 2012; Guye et al., 2006; Rosenberg et al., 2006).
liday, & Pratt, 1978; McNally & Blumenfeld, 2004), and Although less is known about impaired consciousness
intracranial EEG has demonstrated that generalized in complex partial seizures originating outside the tem-
tonic-clonic seizures can spare some brain regions poral lobe, a recent study showed similar findings using
(Schindler, Leung, Lehnertz, & Elger, 2007). intracranial EEG in parietal lobe epilepsy (Lambert
Temporal lobe complex partial seizures exhibit focal et al., 2012).
5–7 Hz theta frequency discharges over the temporal
lobe on scalp EEG, often accompanied by more wide- Neuroimaging
spread slower delta- and theta-frequency activity. Intra-
cranial EEG shows periodic spikes or low-voltage fast Functional neuroimaging has provided important new
activity initially in one medial temporal lobe, followed insights into the mechanisms of impaired consciousness
by theta-frequency polyspike-and-wave activity extend- in epilepsy. Earlier work based on lower-resolution
ing to the lateral temporal cortex unilaterally or bilater- methods has given way to functional MRI (fMRI) and
ally (figure 71.3A, B, C). Loss of consciousness is more other techniques. Single-photon emission computed
common with bilateral involvement (figure 71.3E, G) tomography (SPECT) does not have the resolution of
but can also occur with unilateral temporal lobe seizure fMRI, but allows imaging of cerebral blood flow despite
activity (Englot et al., 2010; Gloor, Olivier, & Ives, 1980; patient movement during seizures. SPECT is based on
Lux et al., 2002). Seizures with left-sided onset are more injection of a radiotracer during seizures, providing a
commonly associated with impaired consciousness; “snapshot” of cerebral blood flow at the time of injec-
however, this effect may be caused at least in part by a tion. The tracer is rapidly taken up by the brain and
bias favoring verbal testing methods; impaired con- remains relatively stable, allowing neuroimaging to be
sciousness can certainly be seen in temporal lobe sei- done 1–2 hours later when the patient is no longer
zures with right-sided onset as well (Englot et al., 2010). moving (Devous, Leroy, & Homan, 1990; Kim, Zubal, &
We recently observed an interesting phenomenon in Blumenfeld, 2009; McNally et al., 2005). Caution must
which sleep-like slow-wave activity occurs in the fronto- be exercised in interpreting functional neuroimaging
parietal association cortex during temporal lobe sei- studies, since they are only indirectly related to neural
zures with impaired consciousness (Blumenfeld, Rivera, activity through neurometabolic and neurovascular
et al., 2004). This slow-wave activity had been observed coupling. Direct measurements of neuronal activity in
previously but interpreted as seizure propagation (Lieb, animal models have revealed unexpected relationships
Dasheiff, & Engel, 1991). However, the frontoparietal and shed important light on the interpretation of neu-
slow-wave activity during temporal lobe seizures more roimaging signals in epilepsy and under normal condi-
closely resembles the EEG of slow-wave sleep, coma, or tions (Englot et al., 2008; Hyder et al., 2010; Mishra
encephalopathy than seizures. Fundamental studies in et al., 2011; Schridde et al., 2008; Suh, Ma, Zhao, Sharif,
an animal model demonstrated very similar phenom- & Schwartz, 2006).
ena and confirmed that this ictal neocortical slow-wave Neuroimaging in patients with absence epilepsy has
activity has reduced neuronal firing, blood flow, and recently used simultaneous EEG-fMRI to demonstrate
metabolism, similar to deep anesthesia or sleep, whereas changes in all parts of the consciousness system during
seizure activity shows increases in all of these parame- seizures. Spike-wave discharges in these patients
ters (Englot et al., 2008, 2009). We concluded that focal produce fMRI increases in the thalamus as well as the
seizures in the temporal lobe can induce a unique state primary visual, sensorimotor, and auditory cortex bilat-
of depressed cortical function, which may be crucial for erally, while predominantly fMRI decreases are seen in
impaired consciousness in this disorder. In support of default-mode cortical regions such as the medial fron-
this, we found that bilateral delta-frequency slow-wave toparietal cortex and lateral parietal cortex (Archer,
activity on intracranial EEG recordings from the fron- Abbott, Waites, & Jackson, 2003; Berman et al., 2010;
toparietal cortex is strongly associated with impaired Carney et al., 2010; Gotman et al., 2005; Moeller, LeVan,
consciousness in temporal lobe epilepsy (figure 71.3A– et al., 2010; Moeller, Siebner, Wolff, Muhle, Granert,
F). Temporal lobe seizures without neocortical slow- et al., 2008; Salek-Haddadi et al., 2003). The lateral
wave activity (figure 71.3G) are not associated with frontal cortex shows variable changes (Carney,
impaired consciousness. Masterton, Flanagan, Berkovic, & Jackson, 2012). It was
Depth electrode recordings from patients with tem- recently recognized that the standard “canonical”
poral lobe epilepsy have revealed abnormally enhanced hemodynamic response function does not adequately
components of the consciousness system (figure 71.5A). brainstem, thalamus, hypothalamus, and basal fore-
Focal temporal lobe seizures produce abnormal poly- brain (figure 71.5D). Removal of subcortical arousal
spike discharges in the medial temporal lobe (figure leads to widespread cortical slow-wave activity (figure
71.5B). Seizures propagate along known anatomical 71.5D) resembling coma or deep sleep and produces
pathways to subcortical structures, particularly to impaired consciousness.
GABAergic inhibitory neurons in the lateral septal In addition to the human data, neuroimaging and
nuclei, anterior hypothalamic ventrolateral preoptic other measurements in animal models have provided
area, thalamic reticular nucleus, habenula, substantia valuable insights into the mechanisms of impaired con-
nigra pars reticulata, ventral pallidum, and cerebellar sciousness in complex partial temporal lobe seizures.
cortex (figure 71.5C). These regions may powerfully Rat hippocampal seizures show behavioral arrest and
inhibit subcortical arousal systems in the upper neocortical slow-wave activity similar to humans (Englot
830 consciousness
Figure 71.5 Network inhibition hypothesis for impaired of consciousness. (C) Spread of seizure activity from the tem-
consciousness in temporal lobe complex partial seizures. poral lobe to midline subcortical structures. Propagation
(A) Under normal conditions, the upper brainstem-dience- often occurs to the contralateral mesial temporal as well (not
phalic activating systems interact with the cerebral cortex to shown). (D) Inhibition of subcortical activating systems leads
maintain normal consciousness. (B) Focal seizure involving to depressed activity in bilateral frontoparietal association
the mesial temporal lobe. If the seizure remains confined, cortex and to loss of consciousness. (Modified with permis-
then a simple partial seizure will occur without impairment sion from Blumenfeld & Taylor, 2003.)
et al., 2008). Multiunit recordings revealed up and (Motelow et al., 2012). The combination of human
down states of neuronal firing, closely resembling coma, data and work from the animal model suggests that
deep sleep, or encephalopathy (Englot et al., 2008; therapeutic neurostimulation of subcortical arousal
Haider, Duque, Hasenstaub, & McCormick, 2006; Ste- systems (Schiff et al., 2007) may be a reasonable
riade, Contreras, Curro Dossi, & Nunez, 1993). Rat future treatment option, with a goal of improving
fMRI and electrophysiology confirmed that ictal neo- consciousness in medically and surgically refractory
cortical slow activity is a unique state of depressed corti- complex partial seizures.
cal function, associated with a mean decrease in neuronal
firing, cerebral blood flow, cerebral blood volume, and Summary and future directions
cerebral metabolic rate of oxygen consumption (Englot
et al., 2008). In contrast, hippocampal or neocortical We have seen that absence, generalized tonic-clonic,
seizure activity elicits increases in all of these variables. and complex partial seizures all converge on the same
Further experiments demonstrated that cutting the set of anatomical structures when consciousness is
fornix prevented subcortical spread of seizures, neocor- impaired. These seizures differ in behavior, physiology,
tical slow activity, and behavioral arrest (Englot et al., and neuroimaging, yet all converge on the conscious-
2009). In addition, stimulation of inhibitory subcortical ness system (figure 71.1) through different mechanisms
structures such as the lateral septum replicated slow (table 71.1). Seizures can disrupt the consciousness
activity and behavioral arrest (Englot et al., 2009) and system either through direct propagation or through
single-unit recordings revealed a shutdown of neurons indirect network effects, leading to impaired level of
involved in subcortical arousal, such as cholinergic consciousness. Mechanisms involving changes in infor-
neurons in the penunculopontine tegmental nucleus mation integration or the global workspace have also
832 consciousness
retained consciousness. J Neurol Neurosurg Psychiatry, 63, Browne, T. R., Penry, J. K., Porter, R. J., & Dreifuss, F. E.
792–795. (1974). Responsiveness before, during and after spike-wave
Berg, A. T., Berkovic, S. F., Brodie, M. J., Buchhalter, J., paroxysms. Neurology, 24, 659–665.
Cross, J. H., van Emde Boas, W., … Scheffer, I. E. (2010). Buschman, T. J., & Miller, E. K. (2007). Top-down versus
Revised terminology and concepts for organization of sei- bottom-up control of attention in the prefrontal and poste-
zures and epilepsies: Report of the ILAE Commission on rior parietal cortices. Science, 315, 1860–1862.
Classification and Terminology, 2005–2009. Epilepsia, 51, Carney, P. W., Masterton, R. A., Flanagan, D., Berkovic,
676–685. S. F., & Jackson, G. D. (2012). The frontal lobe in absence
Berger, H. (1929). Ueber das elektrenkephalogramm des epilepsy: EEG-fMRI findings. Neurology, 78, 1157–1165.
menschen. Arch Psychiatr Nervenkr, 87, 527. Carney, P. W., Masterton, R. A., Harvey, A. S., Scheffer,
Berman, R., Negishi, M., Vestal, M., Spann, M., Chung, M., I. E., Berkovic, S. F., & Jackson, G. D. (2010). The core
Bai, X., … Blumenfeld, H. (2010). Simultaneous EEG, network in absence epilepsy. Differences in cortical and
fMRI, and behavioral testing in typical childhood absence thalamic BOLD response. Neurology, 75, 904–911.
seizures. Epilepsia, 51(10), 2011–2022. Cavanna, A. E., Mula, M., Servo, S., Strigaro, G., Tota, G.,
Blum, D. E., Eskola, J., Bortz, J. J., & Fisher, R. S. (1996). Barbagli, D., … Monaco, F. (2008). Measuring the
Patient awareness of seizures. Neurology, 47, 260–264. level and content of consciousness during epileptic sei-
Blumenfeld, H. (2005a). Cellular and network mechanisms zures: The Ictal Consciousness Inventory. Epilepsy Behav, 13,
of spike-wave seizures. Epilepsia, 46(Suppl. 9), 21–33. 184–188.
Blumenfeld, H. (2005b). Consciousness and epilepsy: Why Cavanna, A. E., Rickards, H., & Ali, F. (2011). What makes
are patients with absence seizures absent? Prog Brain Res, a simple partial seizure complex? Epilepsy Behav, 22,
150, 271–286. 651–658.
Blumenfeld, H. (2009). Epilepsy and consciousness. In Chalmers, D. J. (1996). The conscious mind: In search of a fun-
S. Laureys & G. Tononi (Eds.), The neurology of consciousness: damental theory. Oxford, UK: Oxford University Press.
Cognitive neuroscience and neuropathology (pp. 15–30). New Charidimou, A., & Selai, C. (2011). The effect of alterations
York, NY: Academic. in consciousness on quality of life (QoL) in epilepsy:
Blumenfeld, H. (2010). Neuroanatomy through clinical cases Searching for evidence. Behav Neurol, 24, 83–93.
(2nd ed.). Sunderland, MA: Sinauer. Chassagnon, S., Namer, I. J., Armspach, J. P., Nehlig, A.,
Blumenfeld, H. (2011). Epilepsy and the consciousness Kahane, P., Kehrli, P., … Hirsch, E. (2009). SPM analysis
system: Transient vegetative state? Neurol Clin, 29, of ictal-interictal SPECT in mesial temporal lobe epilepsy:
801–823. Relationships between ictal semiology and perfusion
Blumenfeld, H. (2012). Impaired consciousness in epilepsy. changes. Epilepsy Res, 85, 252–260.
Lancet Neurol, 11, 814–826. Chipaux, M., Vercueil, L., Kaminska, A., Mahon, S., &
Blumenfeld, H., McNally, K. A., Ostroff, R. B., & Zubal, Charpier, S. (2013). Persistence of cortical sensory pro-
I. G. (2003). Targeted prefrontal cortical activation with cessing during absence seizures in human and an animal
bifrontal ECT. Psychiatry Res, 123, 165–170. model: Evidence from EEG and intracellular recordings.
Blumenfeld, H., McNally, K. A., Vanderhill, S. D., Paige, PLoS ONE, 8(3), e58180.
A. L., Chung, R., Davis, K., … Spencer, S. S. (2004). Posi- Cunningham, C., Chen, W. C., Shorten, A., McClurkin, M.,
tive and negative network correlations in temporal lobe Choezom, T., Schmidt, C.P., … Blumenfeld, H. (2014).
epilepsy. Cereb Cortex, 14, 892–902. Impaired consciousness in partial seizures is bimodally dis-
Blumenfeld, H., Rivera, M., McNally, K. A., Davis, K., tributed. Neurology, 82(19), 1736–1744.
Spencer, D. D., & Spencer, S. S. (2004). Ictal neocortical Davidoff, R. A., & Johnson, L. C. (1964). Paroxysmal EEG
slowing in temporal lobe epilepsy. Neurology, 63, activity and cognitive-motor performance. Electroen Clin
1015–1021. Neurophysiol, 16, 343–354.
Blumenfeld, H., & Taylor, J. (2003). Why do seizures cause De Boer, H. M., Mula, M., & Sander, J. W. (2008). The
loss of consciousness? Neuroscientist, 9, 301–310. global burden and stigma of epilepsy. Epilepsy Behav, 12,
Blumenfeld, H., Varghese, G., Purcaro, M. J., Motelow, 540–546.
J. E., Enev, M., McNally, K. A., … Paige, A. L. (2009). Dennett, D. (1991). Consciousness explained. New York, NY:
Cortical and subcortical networks in human secondarily Little, Brown.
generalized tonic-clonic seizures. Brain, 132, 999–1012. Devous, M. D., Sr., Leroy, R. F., & Homan, R. W. (1990).
Blumenfeld, H., Westerveld, M., Ostroff, R. B., Vander- Single photon emission computed tomography in epilepsy.
hill, S. D., Freeman, J., Necochea, A., … Zubal, I. G. Semin Nucl Med, 20, 325–341.
(2003). Selective frontal, parietal and temporal networks in Dosenbach, N. U., Fair, D. A., Miezin, F. M., Cohen, A. L.,
generalized seizures. NeuroImage, 19, 1556–1566. Wenger, K. K., Dosenbach, R. A., … Petersen, S. E.
Botez, M. I., Serbanescu, T., & Stoica, I. (1966). The (2007). Distinct brain networks for adaptive and stable task
problem of focal epileptic seizures on both parts of the control in humans. Proc Natl Acad Sci USA, 104,
body without loss of consciousness. Psychiatr Neurol Neuro- 11073–11078.
chir, 69, 431–437. Ebersole, J. S., & Pedley, T. A. (2003). Current practice of clini-
Boudin, G., Barbizet, J., & Masson, S. (1958). Etude de la cal electroencephalography (3rd ed.). Philadelphia, PA: Lip-
dissolution de la conscience dans 3 cas de petit mal avec pincott Williams & Wilkins.
crises prolongées. Rev Neurol, 99, 483–487. Enev, M., McNally, K. A., Varghese, G., Zubal, I. G.,
Bremer, F. (1955). Interrelationships between cortex and Ostroff, R. B., & Blumenfeld, H. (2007). Imaging onset
subcortical structures; introductory remarks. Electroen Clin and propagation of ECT-induced seizures. Epilepsia, 48,
Neurophysiol, 1(Suppl. 4), 145–148. 238–244.
834 consciousness
Laureys, S., & Schiff, N. D. (Eds.). (2009). Disorders of con- disagree: Corticothalamic and striatal activity patterns in
sciousness. Annals of the New York Academy of Sciences, the WAG/Rij rat. J Neurosci, 31, 15053–15064.
Vol. 1157. New York, NY: Wiley-Blackwell. Moeller, F., LeVan, P., Muhle, H., Stephani, U., Dubeau,
Laureys, S., & Tononi, G. (2008). The neurology of conscious- F., Siniatchkin, M., & Gotman, J. (2010). Absence sei-
ness: Cognitive neuroscience and neuropathology. Waltham, MA: zures: Individual patterns revealed by EEG-fMRI. Epilepsia,
Academic Press. 51, 2000–2010.
Lee, B. I., Markand, O. N., Wellman, H. N., Siddiqui, A. R., Moeller, F., Muhle, H., Wiegand, G., Wolff, S.,
Mock, B., Krepshaw, J., & Kung, H. (1987). HIPDM single Stephani, U., & Siniatchkin, M. (2010). EEG-fMRI
photon emission computed tomography brain imaging in study of generalized spike and wave discharges without
partial onset secondarily generalized tonic-clonic seizures. transitory cognitive impairment. Epilepsy Behav, 18(3),
Epilepsia, 28, 305–311. 313–316.
Lee, K. H., Meador, K. J., Park, Y. D., King, D. W., Murro, Moeller, F., Siebner, H. R., Wolff, S., Muhle, H., Boor, R.,
A. M., Pillai, J. J., & Kaminski, R. J. (2002). Pathophysiol- Granert, O., … Siniatchkin, M. (2008). Changes in activ-
ogy of altered consciousness during seizures: Subtraction ity of striato-thalamo-cortical network precede generalized
SPECT study. Neurology, 59, 841–846. spike wave discharges. NeuroImage, 39(4), 1839–1849.
Levav, M., Mirsky, A. F., Herault, J., Xiong, L., Amir, N., & Moeller, F., Siebner, H. R., Wolff, S., Muhle, H., Granert,
Andermann, E. (2002). Familial association of neuropsy- O., Jansen, O., … Siniatchkin, M. (2008). Simultaneous
chological traits in patients with generalized and partial EEG-fMRI in drug-naive children with newly diagnosed
seizure disorders. J Clin Exp Neuropsychol, 24, 311–326. absence epilepsy. Epilepsia, 49(9), 1510–1519.
Li, Q., Luo, C., Yang, T., Yao, Z., He, L., Liu, L., … Zhou, D. Monti, M. M., Vanhaudenhuyse, A., Coleman, M. R., Boly,
(2009). EEG-fMRI study on the interictal and ictal general- M., Pickard, J. D., Tshibanda, L., … Laureys, S. (2010).
ized spike-wave discharges in patients with childhood Willful modulation of brain activity in disorders of con-
absence epilepsy. Epilepsy Res, 87, 160–168. sciousness. N Engl J Med, 362, 579–589.
Lieb, J. P., Dasheiff, R. B., & Engel, J., Jr. (1991). Role of Moruzzi, G., & Magoun, H. W. (1949). Brain stem reticular
the frontal lobes in the propagation of mesial temporal formation and activation of the EEG. Electroen Clin Neuro-
lobe seizures. Epilepsia, 32, 822–837. physiol, 1, 455–473.
Lux, S., Kurthen, M., Helmstaedter, C., Hartje, W., Motelow, J. E., Gummadavelli, A., Zayyad, Z., Mishra,
Reuber, M., & Elger, C. E. (2002). The localizing value of A. M., Sachdev, R. N. S., Sanganahalli, B. G., … Blumen-
ictal consciousness and its constituent functions: A video- feld, H. (2012). Brainstem cholinergic and thalamic dys-
EEG study in patients with focal epilepsy. Brain, 125, function during limbic seizures: Possible mechanism for
2691–2698. cortical slow oscillations and impaired consciousness. Soc
Mayanagi, Y., Watanabe, E., & Kaneko, Y. (1996). Mesial Neurosci Abstracts, 487.25.
temporal lobe epilepsy: Clinical features and seizure mech- Nagel, T. (1974). What is it like to be a bat? Philos Rev, 82,
anism. Epilepsia, 37(Suppl. 3), 57–60. 435–456.
McNally, K. A., & Blumenfeld, H. (2004). Focal network Nei, M., & Bagla, R. (2007). Seizure-related injury and death.
involvement in generalized seizures: New insights from Curr Neurol Neurosci Rep, 7, 335–341.
electroconvulsive therapy. Epilepsy Behav, 5, 3–12. Nersesyan, H., Herman, P., Erdogan, E., Hyder, F., &
McNally, K. A., Paige, A. L., Varghese, G., Zhang, H., Blumenfeld, H. (2004). Relative changes in cerebral blood
Novotny, E. J., Spencer, S. S., … Blumenfeld, H. (2005). flow and neuronal activity in local microdomains during
Localizing value of ictal-interictal SPECT analyzed by SPM generalized seizures. J Cereb Blood Flow Metab, 24,
(ISAS). Epilepsia, 46, 1450–1464. 1057–1068.
McPherson, A., Rojas, L., Bauerschmidt, A., Ezeani, C. C., Norden, A. D., & Blumenfeld, H. (2002). The role of sub-
Yang, L., Motelow, J. E., … Blumenfeld, H. (2012). cortical structures in human epilepsy. Epilepsy Behav, 3,
Testing for minimal consciousness in complex partial and 219–231.
generalized tonic-clonic seizures. Epilepsia, 53(10), Owen, A. M., Coleman, M. R., Boly, M., Davis, M. H.,
e180–183. Laureys, S., & Pickard, J. D. (2006). Detecting awareness
Meeren, H. K., Pijn, J. P., van Luijtelaar, E. L., Coenen, in the vegetative state. Science, 313, 1402.
A. M., & Lopes da Silva, F. H. (2002). Cortical focus drives Pavone, A., & Niedermeyer, E. (2000). Absence seizures and
widespread corticothalamic networks during spontaneous the frontal lobe. Clin Electroencephalogr, 31, 153–156.
absence seizures in rats. J Neurosci, 22, 1480–1495. Penfield, W. (1950). Epileptic automatism and the centren-
Meeren, H., van Luijtelaar, G., Lopes da Silva, F., & cephalic integrating system. Res Publ Assoc Res Nerv Ment Dis,
Coenen, A. (2005). Evolving concepts on the pathophysiol- 30, 513–528.
ogy of absence seizures: The cortical focus theory. Arch Picard, F., & Craig, A. D. (2009). Ecstatic epileptic seizures:
Neurol, 62, 371–376. A potential window on the neural basis for human self-
Mesulam, M. M. (2000). Principles of behavioral and cognitive awareness. Epilepsy Behav, 16, 539–546.
neurology. New York, NY: Oxford University Press. Plum, F., & Posner, J. B. (1972). The diagnosis of stupor and
Mirsky, A. F., & van Buren, J. M. (1965). On the nature of coma. Contemp Neurol Ser, 10, 1–286.
the “absence” in centrencephalic epilepsy: A study of some Plum, F., & Posner, J. B. (1982). The diagnosis of stupor and
behavioral, electroencephalographic, and autonomic coma. Philadelphia, PA: Davis.
factors. Electroen Clin Neurophysiol, 18, 334–348. Raichle, M. E., MacLeod, A. M., Snyder, A. Z., Powers,
Mishra, A. M., Ellens, D. J., Schridde, U., Motelow, J. E., W. J., Gusnard, D. A., & Shulman, G. L. (2001). A default
Purcaro, M. J., DeSalvo, M. N., … Blumenfeld, H. mode of brain function. Proc Natl Acad Sci USA, 98,
(2011). Where fMRI and electrophysiology agree to 676–682.
836 consciousness
(2010). A prospective study of loss of consciousness in epi- sciousness in epilepsy investigated by a prospective
lepsy using virtual reality driving simulation and other video responsiveness in epilepsy scale (RES). Epilepsia, 53,
games. Epilepsy Behav, 18, 238–246. 437–447.
Yang, L., Shklyar, I., Lee, H. W., Ezeani, C. C., Anaya, J., Yu, L., & Blumenfeld, H. (2009). Theories of impaired con-
Balakirsky, S., … Blumenfeld, H. (2012). Impaired con- sciousness in epilepsy. Ann NY Acad Sci, 1157, 48–60.
ABSTRACT Over the last 20 years, our understanding of the (Laureys, 2005). Though the content of consciousness
neuronal basis of perceptual consciousness and selective includes notions related to the self, we focus here on the
attention has greatly progressed. This advancement was facili- vastly better-studied aspects of visual consciousness (for
tated by research using visual illusions and task designs that
keep sensory input constant yet vary internal factors such as other modalities, see several articles featured in Tsuchiya
top-down attention and subjective visibility. To isolate the neu- & van Boxtel, 2013). Furthermore, we focus on top-
ronal mechanisms of consciousness and attention, however, it down selective attention, which is an endogenous, goal-
has become increasingly clear that keeping the sensory input oriented, and volitionally controlled component of
constant is not enough. Unless manipulated independently, attention, as opposed to bottom-up selective attention,
consciousness and attention usually covary. Recent studies
that independently vary both consciousness and attention
which is an exogenous, automatic, stimulus-driven com-
have found that the behavioral and neuronal effects of con- ponent of attention (Itti & Koch, 2001).
sciousness and attention can be dissociated, implying that Top-down attention and consciousness are both
their neuronal mechanisms may be largely independent. Yet, mental processes that can be manipulated without
even if independent neuronal mechanisms underlie con- changes in sensory inputs. As such, they are amenable
sciousness and attention, there remains a conceptual dispute
to contrastive analysis (Baars, 1997) without stimulus
over the exact relationship between these processes. It is now
generally accepted that subjects can selectively attend to attri- confound. Contrastive methods have been extensively
butes of events or objects without becoming aware of them. used in the search for the neuronal correlates of con-
Whether the converse is also true is much more contentious. sciousness (NCC). The NCC is defined as the minimal
That is, is attentional amplification of the neural representa- set of neuronal mechanisms jointly sufficient for any
tion of an event or object always necessary to experience it
one conscious percept (Koch, 2004). To dissociate the
(i.e., is consciousness without attention possible?). We argue
that attentional amplification is necessary to experience an NCC from neural activity that merely correlates with the
object only when it needs to be “selected” among other objects physical characteristics of the stimulus, an array of visual
that compete with it in space and time. In a situation without illusions has been extensively used (Kim & Blake, 2005).
any competition (e.g., an isolated object or a uniform texture), Researchers have investigated behavioral and neuro-
selective attention may not play any significant role. Accord- nal effects of top-down attention under identical sensory
ingly, we argue that the neuronal mechanisms that give rise to
consciousness need to be carefully disentangled from the inputs while instructing subjects to direct their atten-
neuronal mechanisms that resolve competition. Using an tion in one way or the other. This contrasts with studies
isolated-object paradigm, future studies can test the possibility of bottom-up attention, where some aspects of the
of consciousness with no top-down attentional amplification stimulus array (e.g., an abrupt cue) are manipulated,
in mice or monkeys by inactivating synaptic inputs from fron- making it likely that some of the observed attentional
toparietal attentional areas back to visual areas using the
rapidly advancing technology of optogenetics.
effects are directly caused by external factors.
Ultimately, the function of selective attention is to
reduce the onslaught of sensory information streaming
Conceptual issues of consciousness and attention in from sensory arrays to a trickle that can be processed
in real time by the organism. At the neuronal level
Definition of Consciousness and Attention Both (Kastner & Ungerleider, 2000), both top-down and
“consciousness” and “attention” are words used in bottom-up attention act in at least three ways: (1) to
everyday conversation with different meanings. As in increase the baseline neuronal activity (but also see
previous papers (Koch & Tsuchiya, 2007; van Boxtel, Otazu, Tai, Yang, & Zador, 2009); (2) to amplify the
Tsuchiya, & Koch, 2010a), we restrict our discussion to neuronal response to the selected location, feature,
“consciousness” in the sense of “contents of conscious- and/or object; this may occur by reducing noise cor-
ness,” to be distinguished from the meaning inherent relations among neighboring neurons (Cohen & Maun-
in “levels of consciousness,” as in sleep or arousal sell, 2009; Mitchell, Sundberg, & Reynolds, 2009); and
tsuchiya and koch: on the relationship between consciousness and attention 839
(3) to suppress the neuronal response to locations, fea- consciousness,” which is distinguished from “access
tures, objects, or events that are not selected. In the consciousness” (Block, 1996, 2007). Phenomenal con-
discussion on the relationship between consciousness sciousness, defined as “what it is like to have any one
and attention, the second aspect of attention, atten- specific experience,” is closely related to both broad- and
tional amplification, is most relevant. The third aspect narrow-sense qualia. A representation is access-conscious
is also critical when attention resolves competition if it is posed for direct control of reasoning, reporting,
among visual objects, as we emphasize in this chapter. and action (Block, 1996). Cognitively accessed contents
There are two core questions with regard to the rela- of consciousness are stored for working memory and
tionship between consciousness and attention: Can top- flexibly guide present and future behaviors. Such report-
down attention select and amplify the neuronal activity able conscious contents are sometimes called “aware-
representing an object, even when that object is not ness” and are sometimes distinguished from phenomenal
consciously perceived? Is attentional amplification nec- consciousness (Chalmers, 1995). To avoid confusion,
essary to perceive an object consciously? Before review- we will not use the term “awareness” in this chapter.
ing recent studies on these issues, we elaborate what we Here, we will discuss the neuronal mechanisms that
mean by the contents of consciousness. are directly responsible for narrow-sense qualia rather
than the neuronal mechanisms that enable cognitive
Qualia in a Broad and Narrow Sense The funda- access. Cognitive access mechanisms may involve selec-
mental question at the heart of the mind-body problem tive attention, especially when sensory inputs involve
is how electrochemical activity in the brain generates competition among objects. Thus, phenomenal con-
subjective conscious experience. Consciousness is sciousness without cognitive access is closely related
always experienced, at any given moment, as a unified to qualia without top-down attention (Block, 2007;
whole. This single moment of conscious experience is Campana & Tallon-Baudry, 2013; Koch & Tsuchiya,
sometimes called “a quale in a broad sense,” as defined 2008; Lamme, 2010).
in Balduzzi and Tononi (2009; a quale is the singular
form for the Latin word qualia). In a narrow sense, a Possible relationships between consciousness
quale refers to a particular aspect within a quale in the and attention
broad sense. Examples of the narrow-sense qualia are
the redness of a Rothko painting or the dull pain of a Consciousness Is Not Necessary for Atten-
toothache. A quale in the narrow sense can be consid- tion What is the relationship between consciousness,
ered an elemental unit of conscious experience and as in narrow- or broad-sense qualia, and attention?
corresponds to the “one conscious percept” that the While some philosophers argue that attention without
NCC is minimally sufficient for. Note that narrow-sense consciousness is not possible, largely on a theoretical
qualia are the targets of most psychophysical studies, basis (de Brigard & Prinz, 2010; Marchetti, 2012; Mole,
since broad-sense qualia are more difficult to study. In 2008), accumulating evidence strongly suggests that
a typical psychophysical experiment, the redness of attention can amplify neuronal activity that remains,
one object is compared with that of another object. however, inaccessible to consciousness. By last count, at
Although two objects presented at different positions least 37 papers report such an effect (for reviews, see
or time points produce different broad-sense qualia, Cohen, Cavanagh, Chun, & Nakayama, 2012; Dehaene,
they can be comparable in the narrow sense (Kanai & Changeax, Naccache, Sackur, & Sargent, 2006; Koch &
Tsuchiya, 2012). Tsuchiya, 2007; Tallon-Baudry, 2012; van Boxtel et al.,
Does top-down attention alter qualia in either the 2010a). In particular, bottom-up spatial attention
broad or the narrow sense? In this chapter, we argue can be attracted by invisible stimuli (Hsieh, Colas, &
that the role of top-down attention depends on the Kanwisher, 2011; also see Zhaoping, 2008, 2012), top-
nature of the sensory input. If the input is cluttered with down spatial and feature-based attention can be directed
many competing objects, top-down attention resolves to invisible stimuli (Faivre & Kouider, 2011; Kaunitz,
the competition among the stimuli for access to neuro- Fracasso, & Melcher, 2011), invisible objects can elicit
nal resources and alters qualia (Liu, Abrams, & Carrasco, object-based attention (Norman, Heywood, & Kentridge,
2009; see Carrasco, 2011, for a review). It is unlikely, 2013; but also see Tapia, Breitmeyer, Jacob, & Broyles,
however, for top-down attention to have any major 2012), and temporal attention improves priming for
effects on qualia if an object is presented in isolation. invisible objects (Naccache, Blandin, & Dehaene, 2002).
Given this mounting evidence, we conclude that visual
Phenomenal Consciousness and Cognitive Access consciousness is not necessary for selective visual atten-
A closely related concept to qualia is “phenomenal tion to operate. Put differently, subjects can attend to a
840 consciousness
feature or to an entire object without consciously seeing the attentional effects increase as the discrimination
that feature or object. task involves more intense competition among stimuli
or fine discrimination of an object that requires high
Is Attention Necessary for Consciousness? While spatial resolution (e.g., Landolt-square and vernier res-
attention without consciousness is largely agreed upon olution, Yeshurun & Carrasco, 1999). In other words,
among visual psychologists, the issue of “consciousness these studies suggest that certain kinds of conscious
without attention” has been more controversial (e.g., discriminations can be performed with little or perhaps
Cohen et al., 2011, 2012; Tsuchiya, Block, & Koch, 2012; no top-down attentional amplification, while others
Tallon-Baudry, 2012; Tsuchiya & van Boxtel, 2013). cannot. Unfortunately, however, these results do not
A popular conception of attention is to view it as a prove “consciousness without attention,” because the
“gateway” to consciousness. In this view, a subject can dual-task paradigm involves reporting when top-down
consciously experience some aspect of the sensory attention is drained away; it cannot be ruled out that
input only if he or she attends to it. It postulates that some minimal attentional resources have not been allo-
some amount of attention is always needed to amplify cated to these stimuli.
neuronal activity in order for any aspects of the input Why do certain types of discrimination substantially
to be consciously experienced at all. benefit from top-down attention, while others do not?
This view has been challenged by evidence that sug- Some studies suggest that natural stimuli are pro-
gests some classes of percepts and behaviors (e.g., iconic cessed so efficiently that they can proceed with little
memory, partial reportability, gist, and animal and or no top-down amplification (Fei-Fei, VanRullen,
gender detection in dual tasks) can give rise to con- Koch, & Perona, 2005; VanRullen, Reddy, & Koch,
sciousness with little, or perhaps no, top-down atten- 2004). A feedforward computational model of bottom-
tional amplification (Block, 2011; Hardcastle, 1997; up object recognition supports this idea, showing that
Iwasaki, 1993; Koch & Tsuchiya, 2007; Lamme, 2010). natural scenes can be categorized with little or no top-
Supportive evidence comes, for example, from a down attention (Serre, Oliva, & Poggio, 2007). Recent
series of dual-task experiments (Braun & Julesz, 1998), studies on the nature of peripheral vision suggest that
where subjects perform a demanding task at the fixa- summary statistics accessible at the periphery may be
tion point while simultaneously carrying out another sufficient for certain types of discrimination, possibly
task at the periphery. The dual-task paradigm has accounting for why these seemingly complex stimuli
revealed that subjects carry out certain peripheral dis- can be discriminated at the periphery without much
crimination tasks, such as form (triangular vs. circular; attentional amplification (Freeman & Simoncelli, 2011;
Braun, 1994) and conjunction of orientation and color Rosenholtz, Huang, & Ehinger, 2012; figure 72.1).
(Braun & Julesz, 1998), at the same level of perfor- Recently, Cohen and colleagues (2012) argued for
mance when these tasks are performed alone, (that is, the lack of any direct evidence supporting “conscious-
as single tasks) and when they are performed together ness without attention.” As to the dual-task results, they
with the demanding central task (the performance level point out that even discrimination of natural scenes
is adjusted to be ∼75% correct for each task when per- suffers when top-down attention is sufficiently with-
formed alone). drawn via more demanding tasks (Cohen, Alvarez, &
The features that can be discriminated with little or Nakayama, 2011; Walker, Stafford, & Davis, 2008).
no top-down attention include not only a number of The scope of the attentional gateway hypothesis,
low- and mid-level features (Braun, 1994; Lee, Koch, & however, seems limited to situations where intense com-
Braun, 1997) but also some high-level object categories petition among objects is present. Because attention
such as faces (Reddy, Reddy, & Koch, 2006; Reddy, selectively enhances neural responses for a target
Wilken, & Koch, 2004) and animals (Li, VanRullen, among possible distractors, it is not surprising that any
Koch, & Perona, 2002). Surprisingly, other classes of visual discrimination becomes more dependent on top-
seemingly simpler discrimination (e.g., color-bisected down attention when the target is exposed to suffi-
disks, rotated letter “L” vs. “T”) cannot be performed ciently strong competition. The conclusion obtained in
together with the attention-demanding task (VanRul- highly competitive situations is unlikely to generalize to
len, Reddy, & Koch, 2004). When detection or discrimi- situations where only a lone object is presented in isola-
nation thresholds in visual tasks are measured under tion or where a uniform texture is perceived. In these
single- or dual-task conditions, the effects of attention cases, the object or texture is experienced, while top-
are nearly absent for simple detection or coarse dis- down attention has no possibility to select a target
crimination (Lee, Itti, Koch, & Braun, 1999; Morrone, among competing visual stimuli (for there are none).
Denti, & Spinelli, 2002; Tsuchiya & Braun, 2007), while We will return to this point later in this chapter.
tsuchiya and koch: on the relationship between consciousness and attention 841
A B
Original Metamer
Figure 72.1 Why can we perceive natural scenes at the center, both look subjectively the same; that is, they produce
periphery with little or no top-down attention? (A, B) Percep- similar qualia in the broad sense. (C) Impoverished represen-
tual metamers (Freeman & Simoncelli, 2011), and (C) mon- tation at the periphery can still be sufficient to discriminate
grels (Rosenholtz et al., 2012). The nature of peripheral natural scenes. Thus, degraded representation at the periph-
vision has been studied using psychophysics and computa- ery is rich enough to generate vivid but coarse conscious
tional modeling. (A, B) One such computational model of phenomenology of a gist of the scene without top-down atten-
the ventral visual stream responds identical to (B) as it does tional amplification (Campana & Tallon-Baudry, 2013).
to the original photograph (A). Indeed, when fixated at the
Expectation, Attention, and Consciousness due to a failure of other mental processes, such as
Proponents of “the gateway theory” (i.e., that attention expectation (Braun, 2001; also see Cartwright-Finch &
is necessary for consciousness) emphasize visual phe- Lavie, 2007; Mack, 2001) and immediate memory
nomena in which a salient stimulus is not noticed unless (Landman, Spekreijse, & Lamme, 2003; Sligte, Scholte,
top-down attention is directed to the stimulus (Cohen & Lamme, 2008; Wolfe, 1999). Change/inattentional
et al., 2012; O’Regan & Noe, 2001). blindness occur especially when the change is not
In change blindness (Simons & Rensink, 2005), a large expected and is less familiar (Werner & Thies, 2000).
change between two otherwise identical images tends While expectation and attention are both top-down
to be missed until attention is directed to the changed mental processes, they are likely to be supported by
location. In inattentional blindness (Mack & Rock, 1998), distinct neuronal mechanisms, serving different bio-
a stimulus that appears at an unexpected location, away logical functions (Summerfield & Egner, 2009). A
from attention, goes unnoticed. In attentional blink recent functional MRI (fMRI) study (Kok et al., 2012),
(Chun & Potter, 1995; Raymond, Shapiro, & Arnell, which independently manipulated attention and
1992), two targets are embedded in a rapid temporal expectation, found that fMRI signals in V1 are
sequence of distractors. When the first target is detected, enhanced or reduced by attention only when the
the second target tends to be missed if it occurs ∼200– target is expected (figure 72.2), exposing the interde-
500 msec after the first target. pendency between attention and expectation. In the
There have been suggestions that these phenomena larger perspective, it suggests that the question of the
may occur not because of the failure of attention, but relation between consciousness and attention will need
842 consciousness
A Hypothesis 1: B Hypothesis 2: C Real signal
Attention and prediction Attention and observed in V1
have opposing effects prediction interact
Predicted stimulus
Parameter estimate
*
BOLD amplitude
Unpredicted stimulus 3
*
2
*
0
Unattended Attended Unattended Attended Unattended Attended
Figure 72.2 Dissociating expectation from attention using amplifies responses to a predicted stimulus compared to an
functional brain imaging (Kok et al., 2012). (A) Hypothesis 1 unpredicted stimulus, while it has little effect on an unpre-
states that attention amplifies neuronal activation while dicted stimulus. (C) Functional MRI response in V1 is consis-
expectation (or prediction) leads to reduction of the response tent with hypothesis 2, dissociating the effects of attention
in a predictive-coding framework (Friston, 2005; Rao & and expectation. (Modified from Kok et al., 2012.) Future
Ballard, 1999). (B) Hypothesis 2 states that the effects of studies need to investigate the relationship between attention
expectation (or prediction) reverse depending on whether a and consciousness, while controlling for expectation of the
stimulus is attended (Friston, 2009; Rao, 2005). Attention stimuli.
to be reformulated in broader terms to include expecta- standard psychophysical paradigm in humans (Koch &
tion. The next step in the investigation of the neural Tsuchiya, 2007; van Boxtel, Tsuchiya, & Koch, 2010b;
correlates of consciousness is to determine how atten- but see also Block, 2013). In the future, however, it may
tion, expectation, and consciousness relate to each become possible to test consciousness with no attention
other. using animal models, which we will discuss later in this
The attentional blink paradigm is claimed to be a chapter.
sensitive assay for the necessity of attention for con- Given the practical difficulty of establishing con-
sciousness (Cohen et al., 2012). However, it is a rather sciousness with no attention, we turn to recent studies
complicated phenomenon, combining forms of back- that try to test if consciousness and attention are sup-
ward and forward masking, and relies on working ported by different neuronal mechanisms. By indepen-
memory (Akyurek, Hommel, & Jolicoeur, 2007; Colzato, dently manipulating consciousness and attention, these
Spape, Pannebakker, & Hommel, 2007; for a review, see studies measure the behavioral and neuronal conse-
Dux & Marois, 2009). It is unclear whether the atten- quences of such manipulations. The results are mostly
tional blink paradigm purely manipulates top-down consistent with the proposition that consciousness and
attention without other confounds. attention are supported by distinctive neuronal mecha-
nisms (Tallon-Baudry, 2012).
An Optimal Psychophysical Test for Conscious-
ness Without Attention? What is the ideal test that Independent manipulations of consciousness
could determine whether or not attentional amplifica- and attention
tion is necessary for consciousness? As argued above,
change/inattentional blindness as well as attentional Recently, researchers have started manipulating con-
blink may induce perceptual blindness because of a sciousness and attention independently using the same
failure of interaction between top-down attention and stimuli within a single task structure (Kanai, Tsuchiya,
other processes, such as expectation and working & Verstraten, 2006; van Boxtel et al., 2010b; van
memory. On the other hand, the dual-task paradigm den Bussche, Hughes, Humbeeck, & Reynvoet, 2010;
has fewer complications, being a purer manipulation of Watanabe et al., 2011; Wyart & Tallon-Baudry, 2008;
top-down attention. However, as long as subjects report figure 72.3A). These studies have typically manipulated
on a poorly attended (rather than a completely unat- the conscious visibility of a target stimulus via backward
tended) stimulus, some amount of attention might be masking or continuous flash suppression (Tsuchiya &
directed to the stimulus, making the dual-task paradigm Koch, 2005), treating visibility as a proxy for conscious
unsuited to attain a “no attention” condition. awareness; that is, if the stimulus is not seen and not
Indeed, achieving zero top-down attention without reported on, it is not consciously perceived. Then, they
any confound may be nearly impossible within a simultaneously manipulated bottom-up attention by
tsuchiya and koch: on the relationship between consciousness and attention 843
A Manipulating consciousness B
Visible Invisible
grating grating 1
Visible
letters
0.6
0.4
Attend to 0.2
grating M C
0
Attended Unattended
Figure 72.3 (A) 2-by-2 factorial design for independent of a target grating inside the continuous flash suppression,
manipulation of top-down attention and conscious visibility arranged in a ring configuration, presented in the periphery.
of the stimulus (Watanabe et al., 2011). Subjects are asked to (B) Functional MRI responses in V1 are strongly modulated
carry out one of two attention tasks while viewing either a by top-down attention but not by conscious visibility of the
visible or an invisible target stimulus. At the same time, depen- grating. Modified based on figures 2 and S2 (7 subjects in
dent variables, such as hemodynamic responses in the brain, total) in Watanabe et al. (2011). The data was provided by the
are measured. Here, we illustrate roughly what participants original author. The mean BOLD signal over 7–18 sec from
perceived in each condition (not the physical stimulus) in the the block onset is normalized to the attended and visible
study by Watanabe and colleagues. Subjects either had to condition. The error bar represents a 95% confidence
report the presence of a target letter at fixation or the visibility interval.
cueing, or they modulated the spread of top-down (Haynes, Deichmann, & Rees, 2005; Polonsky, Blake,
attention using a dual-task design. Braun, & Heeger, 2000; Wunderlich, Schneider, &
When manipulated independently, consciousness Kastner, 2005) that used binocular rivalry and located
and attention affect the duration of afterimages in the neuronal correlates of consciousness to V1 and even
opposing ways (van Boxtel et al., 2010b). A number the lateral geniculate nucleus of the thalamus (also see
of previous experiments showed, paradoxically, that Maier et al., 2008, on the difference between single
attending to visual stimuli reduces the duration of the neuron activity and hemodynamic response in monkey
induced afterimages (Baijal & Srinivasan, 2009; Lak, V1). This is a prime example of the necessity of separat-
2008; Lou, 2001; Suzuki & Grabowecky, 2003; Wede & ing consciousness from attention (Koch & Tsuchiya,
Francis, 2007) and, independently, that perceptual 2011; Tse, Martinez-Conde, Schlegel, & Macknik, 2005);
invisibility reduces the duration of the afterimages unless explicitly manipulated independently, the neu-
(Gilroy & Blake, 2005; Tsuchiya & Koch, 2005). Because ronal correlates of consciousness can, and usually will,
the effects of consciousness depend on the details of covary with the neuronal correlates of attention.
the afterimage-inducing stimuli (e.g., spatial frequency; Taken together, the behavioral and neuronal effects
Brascamp, van Boxtel, Knapen, & Blake, 2010), it is of consciousness and attention do not always vary
critical to use the same stimuli to study the effects of together, especially when consciousness and attention
consciousness and attention. are independently manipulated, in contrast with what
Watanabe and colleagues (2011) likewise indepen- one might expect from the gateway theory. This is
dently manipulated consciousness and attention while easiest to explain assuming that consciousness and
recording fMRI signals from the primary visual cortex attention are independent biological processes.
(V1) in humans (figure 72.3A, B). They found that the
V1 hemodynamic response is strongly modulated by Untangling the neuronal correlates of
attention but not by the visibility of a grating. Similar consciousness from those of cognitive access
effects have been reported for neuronal activity
recorded via microelectrodes from monkey V1 (Maier, As we have seen in the previous section, the neuronal
Cox, Reavis, Adams, & Leopold, 2011). These two mechanisms that support consciousness and attention
experiments challenge many previous fMRI studies may well be distinct. Yet, at the level of behaviors and
844 consciousness
perception, the effects of consciousness and attention involve competition and thus do not engage attentional
are often closely related. In particular, when we attend mechanisms.
to a feature, object, or event, we usually become con-
scious of it. Furthermore, neuroimaging studies suggest The Neuronal Correlates of Access and Con-
that similar brain regions, such as frontoparietal areas, sciousness The research program for finding the
are commonly activated when probing for conscious- NCC assumes that for a given narrow-sense quale X,
ness or attention (Bor & Seth, 2012; Rees & Lavie, there exists a minimal set of neurons whose activity
2001). Without explicit and independent manipula- pattern is jointly sufficient to generate the quale X (or,
tion, however, neuronal activity in these areas might more generally, brain mechanisms sufficient to gener-
reflect the neuronal correlates of cognitive access rather ate the quale X; Crick, 1994; Metzinger, 2000; Tononi
than those of narrow-sense qualia (Frässle, Sommer, & Koch, 2008). Activating this NCC—by TMS, electrical
Jansen, Naber, & Einhäuser, 2014). Next, we consider stimulation, or optogenetics—would induce the associ-
potential reasons for this surface similarity and discuss ated quale while inactivating the NCC would eliminate
the role of the task designs that have been extensively the quale. It is also assumed that an object that
utilized for studying consciousness and attention. elicits a quale X is always experienced (as a part of a
broad-sense quale) when the NCC sustains a particular
The Role of Competition in the Study of the Neu- pattern of activity. This is assumed to be true whether
ronal Correlates of Attention and Consciousness or not there are competing stimuli for the object for
Studying top-down attention inherently necessitates a the quale X.
stage for competition among stimuli, where attention Now we consider those neuronal mechanisms that
selects one or a few target items and suppresses distract- enable the subject to cognitively access a quale X, a
ing stimuli, as is the case for the dual-task paradigm, process we call the neuronal correlates of access. It
change/inattentional blindness, and attentional blink. involves those mechanisms that hold the representation
In all these cases, a target competes against other dis- of the object in working memory and enable the subject
tractors that are presented within spatiotemporal to report on it using some response modality.
proximity. In this framework, the NCC that have been identified
Is competition also necessary to study the NCC? Do in most previous studies are an aggregate of the mecha-
we consciously experience an item only when it com- nisms underlying phenomenal consciousness, such as
petes against other objects? Obviously not; we can see the coalition of neurons that maximizes the integrated
perfectly well an isolated stimulus or perceive a uniform information in Tononi’s integrated information theory
texture when there are no other distracting stimuli to of consciousness (Oizumi, Albantakis, & Tononi, 2014;
be suppressed by attention. However, most NCC studies Tononi, 2012) or activity in the global neuronal
use experimental paradigms that introduce competi- workspace of Dehaene and Changeux (Dehaene &
tion among objects (Kim & Blake, 2005). Bistable Changeux, 2011), together with the additional mecha-
figures induce competition between two possible inter- nisms that mediate attention as well as access (Aru,
pretations. Binocular rivalry and continuous flash sup- Bachmann, Singer, & Melloni, 2012; de Graaf, Hsieh, &
pression render an object invisible via interocular Sack, 2012). Untangling all three will not be easy.
competition. While binocular rivalry is less controllable The proponents of the gateway theory, according to
by top-down attention than bistable figures (Meng & which every conscious percept must first pass through
Tong, 2004), a recent study suggests that binocular the crucible of attention, claim that a quale X without
rivalry requires visual attention (Zhang, Jamison, Engel, cognitive access cannot be scientifically studied, or that
He, & He, 2011). Crowding renders a target unrecog- it may not even exist (Cohen et al., 2012; Cohen &
nizable by surrounding objects in close proximity at the Dennett, 2011; Dehaene et al., 2006; O’Regan & Noe,
periphery (Faivre & Kouider, 2011; He, Cavanagh, & 2001). We argue, however, that cognitive access is only
Intriligator, 1996). Motion-induced blindness (Bonneh, necessary when the object for the quale X needs to be
Cooperman, & Sagi, 2001) results from competition selectively amplified to win the competition among
between a moving surface and an object. Therefore, other objects. The exact amount of attentional amplifi-
when these visual illusions are used as the main tools cation needed would depend on a category of the
for the search for the NCC, the putative NCC can object for the quale X (e.g., less for faces and natural
involve the neuronal mechanisms that resolve competi- scenes, more for rotated letters) as well as the intensity
tion, in addition to those that are directly responsible of competition.
for generating qualia. Therefore, it is desirable to have This raises the question of whether access is needed
experimental designs in the NCC literature that do not at all for phenomenal consciousness to occur (Block,
tsuchiya and koch: on the relationship between consciousness and attention 845
2007, 2011; Campana & Tallon-Baudry, 2013; Lamme, been activated at the time of perceptual switches while
2010). Conscious access and reports might enhance a viewing ambiguous stimuli and binocular rivalry
quale X to the extent that it can be held in working (Kleinschmidt, Buchel, Zeki, & Frackowiak, 1998;
memory and reported accurately (Kouider, de Gardelle, Lumer, Friston, & Rees, 1998; Lumer & Rees, 1999).
Sackur, & Dupoux, 2010; Lau & Rosenthal, 2011). In However, a recent replication of this effect by Knapen
fact, from a third-person perspective, it seems impossi- and colleagues (2011) discovered that the effects criti-
ble to assess a conscious experience without any report cally depend on the exact nature of the control stimu-
from the subject. However, so-called mind-reading tech- lus. When they created the control stimulus, whose
niques are being developed that allow perceptual dynamics during transitions is comparable to that
content to be read out from the hemodynamic response during genuine binocular rivalry (see figure 72.4), the
in many regions of the brain, including volunteers difficulty in deciding which stimulus was perceptually
watching movies (Nishimoto et al., 2011), patients with dominant became comparable between real rivalry and
grave neurological damage and in a minimally con- the control. As a result, the difference in frontoparietal
scious state (Monti et al., 2010; Owen et al., 2006), or activity between the rivalry and control conditions dis-
visual imagery experienced during sleep onset (Hori- appeared. This study implies that activity in frontopari-
kawa, Tamaki, Miyawaki, & Kamitani, 2013). Further- etal areas can be caused by, rather than cause, difficult
more, from a first-person perspective, on a day-to-day perceptual decisions that are required during percep-
basis we rarely report the contents of consciousness tual transitions.
outside the lab. That is, most of us don’t go through
our daily life and keep up a constant verbal report of On the Importance of a No-Report Paradigm While
what we momentarily experience. Keeping this argu- subjective reports are the gold standard in conscious-
ment in mind, next, we review recent studies, which ness research, the development of mind-reading tech-
suggest frontoparietal areas as the core of the NCC. niques allows researchers to infer the contents of
consciousness in the absence of explicit reports from
The Putative Neuronal Correlates of Conscious- subjects. Certain conscious visual content can be
ness in Frontoparietal Areas Based on the meta- decoded in healthy participants (Brown & Norcia, 1997;
analysis of the NCC studies, frontoparietal areas have Garcia, Srinivasan, & Serences, 2013; Haxby et al., 2001;
often been suggested as a common and central compo- Haynes & Rees, 2005; Horikawa et al., 2013; Kamitani
nent for conscious perception (Bor & Seth, 2012; Rees & Tong, 2005; Nishimoto et al., 2011) and in some
& Lavie, 2001). However, this claim needs to be reevalu- nonresponsive and noncommunicable patients (Monti
ated for several reasons. et al., 2010; Owen et al., 2006).
First, given a large number of distinct cytoarchitec- Consider binocular rivalry for moving gratings
tonically defined regions within frontoparietal cortex, (Frässle et al., 2014). Here, both optokinetic nystagmus
these claims are too unspecific. Even narrowing them (objective) or verbal (subjective) reports can be used to
to within dorsolateral prefrontal cortex and posterior infer the dominant direction of perceived motion.
parietal cortex, they are still rather coarse. Without When both objective and subjective reports are used,
identifying particular subareas within these areas for the usual occipital, parietal, and frontal regions are
each individual subject, it is unclear if the same regions identified as fMRI correlates of perceptual alternations.
are engaged in both consciousness- and attention- However, when subjects passively experience rivalry
related tasks for a given subject. without verbally reporting their percept and the domi-
Second, the studies that point to activities in fronto- nant direction of motion is inferred via the objective
parietal areas as the NCC typically utilize visual illusions optokinetic nystagmus, differential activity in frontal
that involve competition among objects, as we saw in areas disappeared and only activity in occipital and pari-
the last section. Studies that use illusions that minimize etal regions remained.
stimulus competition tend not to find frontoparietal The contents of consciousness can also be inferred
activation as the NCC (Tse et al., 2005). without overt reports in nonhuman primates, especially
Lastly, as discussed below, the act of reporting by using robust visual illusions. Wilke and colleagues pre-
button press on one’s own perception gives rise to activ- sented a salient red circle that could be masked using
ity in frontal regions that are not present under condi- generalized flash suppression (GFS; Wilke, Logothetis,
tions of unreported perception (Frässle et al., 2014). & Leopold, 2003; Wilke, Mueller, & Leopold, 2009;
Another recent fMRI study also points to the role of figure 72.5A) to monkeys who were trained to report
frontoparietal areas in solving competition between their percept with a lever. By titrating the stimulus
stimuli. In previous studies, frontoparietal cortex has parameters for GFS, they rendered the illusion
846 consciousness
Figure 72.4 Confounding activation in frontoparietal areas transition cannot be captured (e.g., instantaneous replay; top
can be introduced by a less faithful replay of what subjects right of panel A). With a third response option of accurately
really experience. (A) An exemplar time course of stimulus reporting the transitional states, a more faithful replay can be
and perception during genuine binocular rivalry. One eye is reproduced (e.g., duration-matched replay; bottom right of
stimulated by a vertical red and the other eye by a horizontal panel A). (B) Brain activity contrasted at the perceptual tran-
green grating (top). The subjective experience is highly sition between genuine rivalry and instantaneous replay
complex, involving various durations of intermediate per- (without faithfully reproduced transitions), indicating a wide-
cepts (rows 2 and 3). As a control condition, a perceptually spread right-lateralized frontoparietal activity, replicating pre-
matched replay movie can be presented to both eyes without vious studies. (C) No difference in the brain activity between
interocular conflict. However, if the replay is reproduced genuine rivalry and duration-matched replay. (Modified from
from binary responses, the variable nature of each perceptual Knapen et al., 2011.) (See color plate 62.)
ambiguous, so that monkeys sometimes saw a target and An alternative way to dissociate the neuronal corre-
sometimes did not. When they recorded single-unit lates of qualia from that of access is to ask subjects to
activity and local field potentials (LFP) from the dorsal report their percept with different response modalities
and ventral pulvinar, a visual thalamic nucleus, they (e.g., Imamoglu, Kahnt, Koch, & Haynes, 2012). The
found that, on average, both single-neuron activity and report-modality specific activity should be considered as
low-frequency power (9–30Hz) of the LFP were corre- a part of the neuronal basis of access but not of
lated with fluctuating perceptual reports on a trial-by- qualia.
trial basis (figures 72.5B, D). In sum, these experiments support the claim that
They then carefully tailored the stimulus parameters what we experience every waking minute of our lives—
to control the visibility of the circular target without consciousness for objects, thoughts, events, or memo-
requiring perceptual reports from the monkeys. In the ries—exists whether or not we report on them. Studies
absence of such report, the single neuron spikes in that employ a no-report paradigm can distinguish
the pulvinar continued to correlate with the visibility of access from qualia per se, while NCC studies exploiting
the target (figure 72.5C), while the low-frequency power visual phenomena that rely on competition among
of LFP did not (figure 72.5E). This is a striking dissocia- stimuli, objects, or eyes are likely to be contaminated
tion of multimodal neuronal responses within a single by access mechanisms (Macknik & Martinez-Conde,
brain area, with low-frequency power coding for a neu- 2009; Tse et al., 2005). Another alternative way to
ronal correlate of access while spikes encoded the neu- isolate the neuronal mechanisms of qualia is to focus
ronal correlates of qualia. The former disappeared on a single, isolated object without any competition.
without explicit reports, but the latter remained the
same regardless of the presence or absence of the Conscious Perception of an Isolated Object
reports (Wilke et al., 2009). Without Top-Down Attention In our opinion, the
tsuchiya and koch: on the relationship between consciousness and attention 847
A B Active (with report) C Passive (no report)
% change
15
spk/sec
60
Spiking Spiking
Invisible-visible
Invisible-visible
0 0
Visible
LE
–15 –60
–0.5 0 0.5 1 –0.5 0 0.5 1
D E
% change
% change
30
Removed (9-14Hz) (9-14Hz)
(15-30Hz) (15-30Hz)
Invisible-visible
Invisible-visible
0
Stimulus Percept
–30
–0.5 0 0.5 1 –0.5 0 0.5 1
Time since surround onset (s)
Figure 72.5 Neural correlates of access and qualia. subjective experience. Solid lines indicate the difference in
(A) Wilke and colleagues (2009) used a binocular masking firing rate between visible vs. (perceptually) invisible trials.
technique to manipulate the visibility of a disk. (“LE” = left Dotted lines indicate the difference in firing rate between
eye; “RE” = right eye.) The neuronal activity is compared with visible vs. physical removal trials. (D, E) The low-frequency
the control condition where the stimulus was physically (9–30 Hz) power of the local field potentials (LFP) in the
removed. (B, C) While recording spiking activity in the pulvi- pulvinar distinguished the visibility only when monkeys
nar, some neurons signaled the visibility of the stimulus inde- reported the visibility (D), but not when they did not report
pendent of the presence (B) or absence (C) of report, the visibility of the stimulus (E), making them neuronal cor-
suggesting that these spiking activities are a correlate of relates of access but not of qualia.
detection or coarse discrimination of a single object is top-down attentional amplification is likely to be some
unlikely to suffer even if top-down attentional mecha- parts of frontoparietal cortex (Bressler et al., 2008; Nou-
nisms were completely eliminated. However, any psy- doost, Chang, Steinmetz, & Moore, 2010). The extraor-
chophysical manipulation may not be complete and dinarily rapid development of ever more refined
cannot assure total removal of attentional resources. transgenic mice as well as viral techniques, both of which
How could one prove that no top-down attention is target specific, molecular, and projectional defined neu-
required to detect a target? ronal populations that can be labeled and turned on or
One way to do so is to transiently knock out the axons off with millisecond precision from anywhere from mil-
that project to early visual cortices from frontal and liseconds to hours, has given systems neuroscience an
parietal areas as well as higher-order visual cortices. In amazing ability to delicately, reversibly, and transiently
the limit, one would obtain a purely feedforward set of intervene and to observe the phenotype at the behav-
cortical regions (with local feedback, of course) but no ioral and the circuit levels (Huang & Zeng, 2013). In
modulatory input from higher cortical regions reaching particular, it is possible to either locally or globally turn
down into superficial layers in earlier cortical regions. off (and back on) feedback pathways originating in
Recent development of optogenetic techniques frontal or parietal regions and projecting back into one
promises an opportunity to test such an idea of “con- of the 10 or more visual regions in the common labora-
sciousness with no top-down attention” in animals, tory mouse (Cruikshank, Urabe, Nurmikko, & Connors,
especially in mice (Deisseroth, 2011; Yizhar, Fenno, 2010; Oh et al., 2014; Wang, Gao, & Burkhalter, 2011).
Davidson, Mogri, & Deisseroth,, 2011) and to a lesser The mouse can be trained in a visual foraging task, and
extent in monkeys (Diester et al., 2011; Gerits et al., its top-down attention can be selectively manipulated
2012; Han et al., 2009). The neuronal source of using optogenetic and pharmacogenetic tools.
848 consciousness
If top-down attention were to be transiently inacti- addressed with elegant experimentation, possibly utiliz-
vated by these interventionist optogenetic techniques, ing the development of optogenetics in transgenic
we predict that the affected animal would not be able animals. Consciousness research has successfully turned
to perform a visual task if the target is presented many philosophical questions into empirical questions,
together with distractors, whereas it would be able to and it will continue to do so in the future.
perform the task if the target is presented in isolation.
Without inactivation, the same animal would perform ACKNOWLEDGMENTS NT is supported by the JST PRESTO fel-
lowship (Japan), the ARC Future Fellowship (Australia), and
the task as well as before the inactivation, with or
the ARC Discovery Project (Australia), and CK is supported
without any distractors. by the G. Harold & Leila Y. Mathers Charitable Foundation
To make sure these tasks are performed consciously, (US). The authors thank Jakob Hohwy, Jeroen van Boxtel,
the mice or monkeys can be trained to report their Lisandro Kaunitz, Fabiano Baroni, and Ned Block for com-
confidence via post-decision wagering (Kepecs, Uchida, ments on an earlier version of the manuscript.
Zariwala, & Mainen, 2008; Kiani & Shadlen, 2009;
Persaud, McLeod, & Cowey, 2007). As the confidence
or metacognitive judgment is likely to be mediated by REFERENCES
medial orbitofrontal areas (Fleming, Weil, Nagy, Dolan, Akyurek, E. G., Hommel, B., & Jolicoeur, P. (2007). Direct
& Rees, 2010; but also see Komura, Nikkuni, Hirashima, evidence for a role of working memory in the attentional
Uetake, & Miyamoto, 2013), knocking out top-down blink. Mem Cognit, 35(4), 621–627.
Aru, J., Bachmann, T., Singer, W., & Melloni, L. (2012).
attention in dorsal prefrontal or parietal areas might
Distilling the neural correlates of consciousness. Neurosci
not affect confidence judgments. Biobehav Rev, 36(2), 737–746.
Baars, B. J. (1997). Some essential differences between con-
Conclusion sciousness and attention, perception, and working memory.
Conscious Cogn, 6(2–3), 363–371.
Baijal, S., & Srinivasan, N. (2009). Types of attention matter
Studies of visual attention and consciousness have
for awareness: A study with color afterimages. Conscious
greatly benefited from task designs that keep input Cogn, 18(4), 1039–1048.
stimuli constant while the contents of consciousness or Balduzzi, D., & Tononi, G. (2009). Qualia: The geometry of
the focus of top-down attention are shifted. This strat- integrated information. PLoS Comput Biol, 5(8), e1000462.
egy has been particularly successful, and continues to Block, N. (1996). How can we find the neural correlate of
be popular in the search for the NCC (Koch, 2004; consciousness? Trends Neurosci, 19(11), 456–459.
Block, N. (2007). Consciousness, accessibility, and the mesh
Logothetis, 1998). Now we know that such an approach between psychology and neuroscience. Behav Brain Sci,
is, by itself, not enough. Competing stimuli in visual 30(5–6), 481–499; discussion 499–548.
illusions invite involvement of attentional mechanisms, Block, N. (2011). Perceptual consciousness overflows cogni-
a part of the neuronal correlates of access, which is tive access. Trends Cogn Sci, 15(12), 567–575.
distinct from the neuronal correlates of qualia. In this Block, N. (2013). The grain of vision and the grain of atten-
tion. Thought, 1, 170–184.
chapter, we reviewed several ways to disentangle access Bonneh, Y. S., Cooperman, A., & Sagi, D. (2001). Motion-
and attention from consciousness, including (1) inde- induced blindness in normal observers. Nature, 411(6839),
pendent manipulation of consciousness and attention, 798–801.
(2) removing reports or modifying report modalities, Bor, D., & Seth, A. K. (2012). Consciousness and the pre-
and (3) using an isolated stimulus to diminish the role frontal parietal network: Insights from attention, working
memory, and chunking. Front Psychol, 3, 63.
of selection.
Brascamp, J. W., van Boxtel, J. J., Knapen, T., & Blake, R.
The refinement of experimental procedures is (2010). A dissociation of attention and awareness in phase-
coupled to the refinement of the conceptual framework sensitive but not phase-insensitive visual channels. J Cogn
for consciousness, attention, and access (see also Aru Neurosci, 22(10), 2326–2344.
et al., 2012; De Graaf, 2012; Hohwy, 2009). The con- Braun, J. (1994). Visual search among items of different
salience: Removal of visual attention mimics a lesion in
cepts of consciousness and attention will be further
extrastriate area V4. J Neurosci, 14(2), 554–567.
refined, as has been the case for the cognitive neurosci- Braun, J. (2001). It’s great but not necessarily about atten-
ence of memory. The concept of memory has tion. Psyche, 7(6).
faced many revisions, subdivided into iconic memory Braun, J., & Julesz, B. (1998). Withdrawing attention at little
(Sperling, 1960), fragile memory (Sligte et al., 2008), or no cost: Detection and discrimination tasks. Percept Psy-
working memory, and declarative and non-declarative chophys, 60(1), 1–23.
Bressler, S. L., Tang, W., Sylvester, C. M., Shulman, G. L.,
long-term memory (Milner, Squire, & Kandel, 1998). & Corbetta, M. (2008). Top-down control of human visual
Even such a seemingly purely metaphysical question cortex by frontal and parietal cortex in anticipatory visual
as “does consciousness exist with no attention” can be spatial attention. J Neurosci, 28(40), 10056–10061.
tsuchiya and koch: on the relationship between consciousness and attention 849
Brown, R. J., & Norcia, A. M. (1997). A method for investi- attention? Exploring attentional requirements for natural
gating binocular rivalry in real-time with the steady-state and synthetic stimuli. Vis Cognit, 12(6), 893–924.
VEP. Vision Res, 37(17), 2401–2408. Fleming, S. M., Weil, R. S., Nagy, Z., Dolan, R. J., &
Campana, F., & Tallon-Baudry, C. (2013). Anchoring visual Rees, G. (2010). Relating introspective accuracy to indi-
subjective experience in a neural model: The coarse vivid- vidual differences in brain structure. Science, 329(5998),
ness hypothesis. Neuropsychologia, 51(6), 1050–1060. 1541–1543.
Carrasco, M. (2011). Visual attention: The past 25 years. Vis Frässle, S., Sommer, J., Jansen, A., Naber, M., & Einhäuser,
Res, 51, 1484–1525. W. (2014). Binocular rivalry: Frontal activity relates to intro-
Cartwright-Finch, U., & Lavie, N. (2007). The role of per- spection and action but not to perception. J Neurosci, 34,
ceptual load in inattentional blindness. Cognition, 102(3), 1738–1747.
321–340. Freeman, J., & Simoncelli, E. P. (2011). Metamers of the
Chalmers, D. J. (1995). Facing up to the problem of con- ventral stream. Nat Neurosci, 14(9), 1195–1201.
sciousness. J Consciousness Stud, 2(3), 200–219. Friston, K. (2005). A theory of cortical responses. Philos Trans
Chun, M. M., & Potter, M. C. (1995). A two-stage model for R Soc Lond B Biol Sci, 360(1456), 815–836.
multiple target detection in rapid serial visual presentation. Friston, K. (2009). The free-energy principle: A rough guide
J Exp Psychol Hum Percept Perform, 21(1), 109–127. to the brain? Trends Cogn Sci, 13(7), 293–301.
Cohen, M. A., Alvarez, G. A., & Nakayama, K. (2011). Garcia, J. O., Srinivasan, R., & Serences, J. T. (2013). Near-
Natural-scene perception requires attention. Psychol Sci, real-time feature-selective modulations in human cortex.
22(9), 1165–1172. Curr Biol, 23(6), 515–522.
Cohen, M. A., Cavanagh, P., Chun, M. M., & Nakayama, K. Gerits, A., Farivar, R., Rosen, B. R., Wald, L. L., Boyden,
(2012). The attentional requirements of consciousness. E. S., & Vanduffel, W. (2012). Optogenetically induced
Trends Cogn Sci, 16(8), 411–417. behavioral and functional network changes in primates.
Cohen, M. A., & Dennett, D. C. (2011). Consciousness Curr Biol, 22(18), 1722–1726.
cannot be separated from function. Trends Cogn Sci, 15(8), Gilroy, L. A., & Blake, R. (2005). The interaction between
358–364. binocular rivalry and negative afterimages. Curr Biol, 15(19),
Cohen, M. R. & Maunsell, J. H. (2009). Nat Neurosci, 12, 1740–1744.
1594–1600. Han, X., Qian, X., Bernstein, J. G., Zhou, H. H., Franzesi,
Colzato, L. S., Spape, M., Pannebakker, M. M., & Hommel, G. T., Stern, P., … Boyden, E. S. (2009). Millisecond-
B. (2007). Working memory and the attentional blink: timescale optical control of neural dynamics in the nonhu-
Blink size is predicted by individual differences in opera- man primate brain. Neuron, 62(2), 191–198.
tion span. Psychon Bull Rev, 14(6), 1051–1057. Hardcastle, V. G. (1997). Attention versus consciousness:
Crick, F. (1994). Astonishing hypothesis: The scientific search for A distinction with a difference. Cogn Stud, 4, 56–66.
the soul. New York, NY: Scribner. Haxby, J. V., Gobbini, M. I., Furey, M. L., Ishai, A., Schouten,
Cruikshank, S. J., Urabe, H., Nurmikko, A. V., & Connors, J. L., & Pietrini, P. (2001). Distributed and overlapping
B. W. (2010). Pathway-specific feedforward circuits between representations of faces and objects in ventral temporal
thalamus and neocortex revealed by selective optical stimu- cortex. Science, 293(5539), 2425–2430.
lation of axons. Neuron, 65, 230–245. Haynes, J. D., Deichmann, R., & Rees, G. (2005). Eye-specific
De Brigard, F., & Prinz, J. (2010). Attention and conscious- effects of binocular rivalry in the human lateral geniculate
ness. Wiley Interdiscip Rev Cog Sci, 1, 51–59. nucleus. Nature, 438(7067), 496–499.
De Graaf, T. A., Hsieh, P. J., & Sack, A. T. (2012). The “cor- Haynes, J. D., & Rees, G. (2005). Predicting the orientation
relates” in neural correlates of consciousness. Neurosci Biobe- of invisible stimuli from activity in human primary visual
hav Rev, 36(1), 191–197. cortex. Nat Neurosci, 8(5), 686–691.
Dehaene, S., & Changeux, J. P. (2011). Experimental and He, S., Cavanagh, P., & Intriligator, J. (1996). Attentional
theoretical approaches to conscious processing. Neuron, resolution and the locus of visual awareness. Nature,
70(2), 200–227. 383(6598), 334–337.
Dehaene, S., Changeux, J. P., Naccache, L., Sackur, J., & Horikawa, T., Tamaki, M., Miyawaki, Y., & Kamitani, Y.
Sergent, C. (2006). Conscious, preconscious, and sublimi- (2013). Neural decoding of visual imagery during sleep.
nal processing: A testable taxonomy. Trends Cogn Sci, 10(5), Science, 340(6132), 639–642.
204–211. Hohwy, J. (2009). The neural correlates of consciousness:
Deisseroth, K. (2011). Optogenetics. Nat Methods, 8(1), New experimental approaches needed? Consciousness Cogn,
26–29. 18, 428–438.
Diester, I., Kaufman, M. T., Mogri, M., Pashaie, R., Goo, Hsieh, P. J., Colas, J. T., & Kanwisher, N. (2011). Pop-out
W., Yizhar, O., … Shenoy, K. V. (2011). An optogenetic without awareness: Unseen feature singletons capture
toolbox designed for primates. Nat Neurosci, 14(3), attention only when top-down attention is available. Psychol
387–397. Sci, 22(9), 1220–1226.
Dux, P. E., & Marois, R. (2009). The attentional blink: A Huang, Z. J., & Zeng, H. (2013). Genetic approaches to
review of data and theory. Atten Percept Psychophys, 71(8), neural circuits in the mouse. Annu Rev Neurosci, 36,
1683–1700. 183–215.
Faivre, N., & Kouider, S. (2011). Multi-feature objects Imamoglu, F., Kahnt, T., Koch, C., & Haynes, J. D. (2012).
elicit nonconscious priming despite crowding. J Vis, Changes in functional connectivity support conscious
11(3), 2. object recognition. NeuroImage, 63(4), 1909–1917.
Fei-Fei, L., VanRullen, R., Koch, C., & Perona, P. (2005). Itti, L., & Koch, C. (2001). Computational modelling of
Why does natural scene categorization require little visual attention. Nat Rev Neurosci, 2(3), 194–203.
850 consciousness
Iwasaki, S. (1993). Spatial attention and two modes of visual Laureys, S. (2005). The neural correlate of (un)awareness:
consciousness. Cognition, 49(3), 211–233. Lessons from the vegetative state. Trends Cogn Sci, 9(12),
Kamitani, Y., & Tong, F. (2005). Decoding the visual and 556–559.
subjective contents of the human brain. Nat Neurosci, 8(5), Lee, D. K., Itti, L., Koch, C., & Braun, J. (1999). Attention
679–685. activates winner-take-all competition among visual filters.
Kanai, R., & Tsuchiya, N. (2012). Qualia. Curr Biol, 22(10), Nat Neurosci, 2(4), 375–381.
R392–396. Lee, D. K., Koch, C., & Braun, J. (1997). Spatial vision thresh-
Kanai, R., Tsuchiya, N., & Verstraten, F. A. (2006). The olds in the near absence of attention. Vision Res, 37(17),
scope and limits of top-down attention in unconscious 2409–2418.
visual processing. Curr Biol, 16(23), 2332–2336. Li, F. F., VanRullen, R., Koch, C., & Perona, P. (2002). Rapid
Kastner, S., & Ungerleider, L. G. (2000). Mechanisms of natural scene categorization in the near absence of atten-
visual attention in the human cortex. Annu Rev Neurosci, 23, tion. Proc Natl Acad Sci USA, 99(14), 9596–9601.
315–341. Liu, T., Abrams, J., & Carrasco, M. (2009). Voluntary atten-
Kaunitz, L., Fracasso, A., & Melcher, D. (2011). Unseen tion enhances contrast appearance. Psychol Sci, 20(3),
complex motion is modulated by attention and generates 354–362.
a visible aftereffect. J Vis, 11(13), 10. Logothetis, N. K. (1998). Single units and conscious
Kepecs, A., Uchida, N., Zariwala, H. A., & Mainen, Z. F. vision. Philos Trans R Soc Lond B Biol Sci, 353(1377),
(2008). Neural correlates, computation and behavioural 1801–1818.
impact of decision confidence. Nature, 455(7210), Lou, L. (2001). Effects of voluntary attention on structured
227–231. afterimages. Perception, 30(12), 1439–1448.
Kiani, R., & Shadlen, M. N. (2009). Representation of con- Lumer, E. D., Friston, K. J., & Rees, G. (1998). Neural cor-
fidence associated with a decision by neurons in the parietal relates of perceptual rivalry in the human brain. Science,
cortex. Science, 324(5928), 759–764. 280(5371), 1930–1934.
Kim, C. Y., & Blake, R. (2005). Psychophysical magic: Render- Lumer, E. D., & Rees, G. (1999). Covariation of activity
ing the visible “invisible.” Trends Cogn Sci, 9(8), 381–388. in visual and prefrontal cortex associated with subjective
Kleinschmidt, A., Buchel, C., Zeki, S., & Frackowiak, R. S. visual perception. Proc Natl Acad Sci USA, 96(4),
(1998). Human brain activity during spontaneously revers- 1669–1673.
ing perception of ambiguous figures. Philos Trans R Soc Lond Mack, A. (2001). Inattentional blindness: Reply to commen-
B Biol Sci, 265(1413), 2427–2433. taries. Psyche, 7(16).
Knapen, T., Brascamp, J., Pearson, J., van Ee, R., & Blake, Mack, A., & Rock, I. (1998). Inattentional blindness. Cam-
R. (2011). The role of frontal and parietal brain areas in bridge, MA.: MIT Press.
bistable perception. J Neurosci, 31(28), 10293–10301. Macknik, S. L., & Martinez-Conde, S. (2009). The role of
Koch, C. (2004). The quest for consciousness: A neurobiological feedback in visual attention and awareness. In M. S.
approach. Denver, CO: Roberts. Gazzaniga (Ed.), The cognitive neurosciences (4th ed.,
Koch, C., & Tsuchiya, N. (2007). Attention and conscious- pp. 1165–1175). Cambridge, MA: MIT Press.
ness: Two distinct brain processes. Trends Cogn Sci, 11(1), Maier, A. V., Cox, M. A., Reavis, E. A., Adams, G. K., &
16–22. Leopold, D. A. (2011). Perceptual awareness and selective
Koch, C., & Tsuchiya, N. (2008). Phenomenology without attention differentially modulate neuronal responses in
conscious access is a form of consciousness without top- primary visual cortex. Paper presented at the Society for
down attention. Behav Brain Sci, 30(5/6), 509–510. Neuroscience, Washington, DC.
Koch, C., & Tsuchiya, N. (2011). Attention and conscious- Maier, A., Wilke, M., Aura, C., Zhu, C., Ye, F. Q., & Leopold,
ness: Related yet different. Trends Cogn Sci, 16(2), 103–105. D. A. (2008). Divergence of fMRI and neural signals in V1
Kok, P., Rahnev, D., Jehee, J. F., Lau, H. C., & de Lange, F. during perceptual suppression in the awake monkey. Nat
P. (2012). Attention reverses the effect of prediction in Neurosci, 11(10), 1193–1200.
silencing sensory signals. Cereb Cortex, 22(9), 2197–2206. Marchetti, G. (2012). Against the view that consciousness
Komura, Y., Nikkuni, A., Hirashima, N., Uetake, T., & and attention are fully dissociable. Front Psychol, 3, 36.
Miyamoto, A. (2013). Responses of pulvinar neurons Meng, M., & Tong, F. (2004). Can attention selectively bias
reflect a subject’s confidence in visual categorization. Nat bistable perception? Differences between binocular rivalry
Neurosci, 16(6), 749–755. and ambiguous figures. J Vis, 4(7), 539–551.
Kouider, S., de Gardelle, V., Sackur, J., & Dupoux, E. Metzinger, T. (2000). Neural correlates of consciousness:
(2010). How rich is consciousness? The partial awareness Empirical and conceptual questions. Cambridge, MA: MIT
hypothesis. Trends Cogn Sci, 14(7), 301–307. Press.
Lak, A. (2008). Attention during adaptation weakens negative Milner, B., Squire, L. R., & Kandel, E. R. (1998). Cognitive
afterimages of perceptually colour-spread surfaces. Can J neuroscience and the study of memory. Neuron, 20(3),
Exp Psychol, 62(2), 101–109. 445–468.
Lamme, V. A. (2010). How neuroscience will change our view Mitchell, J. F., Sundberg, K. A., & Reynolds, J. H. (2009).
on consciousness. Cogn Neurosci, 1(3), 204–220. Neuron, 63, 879–888.
Landman, R., Spekreijse, H., & Lamme, V. A. (2003). Large Mole, C. (2008). Attention in the absence of consciousness?
capacity storage of integrated objects before change blind- Trends Cogn Sci, 12(2), 44; author reply 44–45.
ness. Vision Res, 43(2), 149–164. Monti, M. M., Vanhaudenhuyse, A., Coleman, M. R., Boly,
Lau, H., & Rosenthal, D. (2011). Empirical support for M., Pickard, J. D., Tshibanda, L., … Laureys, S. (2010).
higher-order theories of conscious awareness. Trends Cogn Willful modulation of brain activity in disorders of con-
Sci, 15(8), 365–373. sciousness. N Engl J Med, 362(7), 579–589.
tsuchiya and koch: on the relationship between consciousness and attention 851
Morrone, M. C., Denti, V., & Spinelli, D. (2002). Color and Serre, T., Oliva, A., & Poggio, T. (2007). A feedforward
luminance contrasts attract independent attention. Curr architecture accounts for rapid categorization. Proc Natl
Biol, 12(13), 1134–1137. Acad Sci USA, 104(15), 6424–6429.
Naccache, L., Blandin, E., & Dehaene, S. (2002). Uncon- Simons, D. J., & Rensink, R. A. (2005). Change blindness:
scious masked priming depends on temporal attention. Past, present, and future. Trends Cogn Sci, 9(1), 16–20.
Psychol Sci, 13(5), 416–424. Sligte, I. G., Scholte, H. S., & Lamme, V. A. (2008). Are
Nishimoto, S., Vu, A. T., Naselaris, T., Benjamini, Y., Yu, B., there multiple visual short-term memory stores? PLoS ONE,
& Gallant, J. L. (2011). Reconstructing visual experiences 3(2), e1699.
from brain activity evoked by natural movies. Curr Biol, Sperling, G. (1960). The information available in brief visual
21(19), 1641–1646. presentations. Psychol Monogr, 74, 1–29.
Norman, L. J., Heywood, C. A., & Kentridge, R. W. (2013). Summerfield, C., & Egner, T. (2009). Expectation (and
Object-based attention without awareness. Psychol Sci, 24(6), attention) in visual cognition. Trends Cogn Sci, 13(9),
836–843. 403–409.
Noudoost, B., Chang, M. H., Steinmetz, N. A., & Moore, Suzuki, S., & Grabowecky, M. (2003). Attention during
T. (2010). Top-down control of visual attention. Curr Opin adaptation weakens negative afterimages. J Exp Psychol Hum
Neurobiol, 20(2), 183–190. Percept Perform, 29(4), 793–807.
Oh, S., Harris, J. A., Ng, L., Winslow, B., Cain, N., … Zeng, Tallon-Baudry, C. (2012). On the neural mechanisms
H. (2014). A mesoscale connectome of the mouse brain. subserving consciousness and attention. Front Psychol, 2,
Nature, 508, 207–214. 397.
Oizumi, M., Albantakis, L., & Tononi, G. (2014). From the Tapia, E., Breitmeyer, B. G., Jacob, J., & Broyles, E. C.
phenomenology to the mechanisms of consciousness: Inte- (2012). Spatial attention effects during conscious and non-
grated Information Theory 3.0. PLoS Comput Biol, 10(5): conscious processing of visual features and objects. J Exp
doi: e1003588. Psychol Hum Percept Perform, 39(3), 745–756.
O’Regan, J. K., & Noe, A. (2001). A sensorimotor account of Tononi, G. (2012). Integrated information theory of con-
vision and visual consciousness. Behav Brain Sci, 24(5), 939– sciousness: An updated account. Arch Ital Biol, 150(2–3),
973; discussion 973–1031. 56–90.
Otazu, G. H., Tai, L. H., Yang, Y., & Zador, A. M. (2009). Tononi, G., & Koch, C. (2008). The neural correlates of
Engaging in an auditory task suppresses responses in audi- consciousness: An update. Ann NY Acad Sci, 1124,
tory cortex. Nat Neurosci, 12(5), 646–654. 239–261.
Owen, A. M., Coleman, M. R., Boly, M., Davis, M. H., Tse, P. U., Martinez-Conde, S., Schlegel, A. A., & Macknik,
Laureys, S., & Pickard, J. D. (2006). Detecting awareness S. L. (2005). Visibility, visual awareness, and visual masking
in the vegetative state. Science, 313(5792), 1402. of simple unattended targets are confined to areas in the
Persaud, N., McLeod, P., & Cowey, A. (2007). Post-decision occipital cortex beyond human V1/V2. Proc Natl Acad Sci
wagering objectively measures awareness. Nat Neurosci, USA, 102(47), 17178–17183.
10(2), 257–261. Tsuchiya, N., Block, N., & Koch, C. (2012). Top-down atten-
Polonsky, A., Blake, R., Braun, J., & Heeger, D. J. (2000). tion and consciousness: Comment on Cohen et al. Trends
Neuronal activity in human primary visual cortex correlates Cogn Sci, 16(11), 527.
with perception during binocular rivalry. Nat Neurosci, Tsuchiya, N., & Braun, J. (2007). Contrast thresholds for
3(11), 1153–1159. component motion with full and poor attention. J Vis, 7(3),
Rao, R. P. (2005). Bayesian inference and attentional 1.
modulation in the visual cortex. NeuroReport, 16(16), Tsuchiya, N., & Koch, C. (2005). Continuous flash suppres-
1843–1848. sion reduces negative afterimages. Nat Neurosci, 8(8),
Rao, R. P., & Ballard, D. H. (1999). Predictive coding in the 1096–1101.
visual cortex: A functional interpretation of some extra- Tsuchiya, N., & van Boxtel, J. J. (2013). Introduction to
classical receptive-field effects. Nat Neurosci, 2(1), 79–87. research topic: Attention and consciousness in different
Raymond, J. E., Shapiro, K. L., & Arnell, K. M. (1992). senses. Front Psychol, 4, 249.
Temporary suppression of visual processing in an RSVP Van Boxtel, J. J., Tsuchiya, N., & Koch, C. (2010a). Con-
task: An attentional blink? J Exp Psychol Hum Percept Perform, sciousness and attention: On sufficiency and necessity. Front
18(3), 849–860. Psychol, 1, 217.
Reddy, L., Reddy, L., & Koch, C. (2006). Face identification Van Boxtel, J. J., Tsuchiya, N., & Koch, C. (2010b). Oppos-
in the near-absence of focal attention. Vision Res, 46(15), ing effects of attention and consciousness on afterimages.
2336–2343. Proc Natl Acad Sci USA, 107(19), 8883–8888.
Reddy, L., Wilken, P., & Koch, C. (2004). Face-gender dis- Van den Bussche, E., Hughes, G., Humbeeck, N. V., &
crimination is possible in the near-absence of attention. Reynvoet, B. (2010). The relation between consciousness
J Vis, 4(2), 106–117. and attention: An empirical study using the priming para-
Rees, G., & Lavie, N. (2001). What can functional imaging digm. Conscious Cogn, 19(1), 86–97.
reveal about the role of attention in visual awareness? Neu- VanRullen, R., Reddy, L., & Koch, C. (2004). Visual search
ropsychologia, 39(12), 1343–1353. and dual tasks reveal two distinct attentional resources.
Rosenholtz, R., Huang, J., & Ehinger, K. A. (2012). Rethink- J Cogn Neurosci, 16(1), 4–14.
ing the role of top-down attention in vision: Effects attribut- Walker, S., Stafford, P., & Davis, G. (2008). Ultra-rapid
able to a lossy representation in peripheral vision. Front categorization requires visual attention: Scenes with multi-
Psychol, 3, 13. ple foreground objects. J Vis, 8(4), 1–12.
852 consciousness
Wang, Q., Gao, E., & Burkhalter, A. (2011). Gateways of Wunderlich, K., Schneider, K. A., & Kastner, S. (2005).
ventral and dorsal streams in mouse visual cortex. J Neurosci, Neural correlates of binocular rivalry in the human lateral
31(5), 1905–1918. geniculate nucleus. Nat Neurosci, 8(11), 1595–1602.
Watanabe, M., Cheng, K., Murayama, Y., Ueno, K., Wyart, V., & Tallon-Baudry, C. (2008). Neural dissociation
Asamizuya, T., Tanaka, K., & Logothetis, N. (2011). between visual awareness and spatial attention. J Neurosci,
Attention but not awareness modulates the BOLD signal 28(10), 2667–2679.
in the human V1 during binocular suppression. Science, Yeshurun, Y., & Carrasco, M. (1999). Spatial attention
334(6057), 829–831. improves performance in spatial resolution tasks. Vision Res,
Wede, J., & Francis, G. (2007). Attentional effects on afterim- 39(2), 293–306.
ages: Theory and data. Vision Res, 47(17), 2249–2258. Yizhar, O., Fenno, L. E., Davidson, T. J., Mogri, M., &
Werner, S., & Thies, B. (2000). Is “change blindness” attenu- Deisseroth, K. (2011). Optogenetics in neural systems.
ated by domain-specific expertise? An expert–novices com- Neuron, 71(1), 9–34.
parison of change detection in football images. Vis Cognit, Zhang, P., Jamison, K., Engel, S., He, B., & He, S. (2011).
7(1–3), 163–173. Binocular rivalry requires visual attention. Neuron, 71(2),
Wilke, M., Logothetis, N. K., & Leopold, D. A. (2003). 362–369.
Generalized flash suppression of salient visual targets. Zhaoping, L. (2008). Attention capture by eye of origin sin-
Neuron, 39(6), 1043–1052. gletons even without awareness—a hallmark of a bottom-up
Wilke, M., Mueller, K. M., & Leopold, D. A. (2009). Neural saliency map in the primary visual cortex. J Vis, 8(5),
activity in the visual thalamus reflects perceptual suppres- 1–18.
sion. Proc Natl Acad Sci USA, 106(23), 9465–9470. Zhaoping, L. (2012). Gaze capture by eye-of-origin single-
Wolfe, J. M. (1999). Inattentional amnesia. In V. Coltheart tons: Interdependence with awareness. J Vis, 12(2), 17.
(Ed.), Fleeting memories (pp. 71–94). Cambridge, MA: MIT
Press.
tsuchiya and koch: on the relationship between consciousness and attention 853
73 Consciousness and Its Access
Mechanisms
SID KOUIDER AND JÉRÔME SACKUR
ABSTRACT Consciousness is a fundamental dimension of our are daunting challenges that consciousness researchers
mental life that involves both cognitive functions (attention, face no less than in other fields of cognitive neurosci-
verbalization, working memory, and so on), and subjective, ences. How then, could any progress be achieved in the
experiential aspects. During the past two decades, thanks to
conceptual and methodological progress, a cognitive neuro-
field? Two strategies have eased these issues and led to
science of consciousness has emerged and gained full respect- considerable progress over the last two decades, and
ability. However, this science remains challenged regarding both critically depend on a psychological operational-
whether the subjective dimension of experience can be fully ization of “consciousness.” The contrastive approach put
accounted for by the neuronal and cognitive mechanisms forward by Bernard Baars (1989) allows moving on
underlying conscious access. In this chapter, we first review
without formal definition, while the search for a neural
the progress and challenges of the cognitive neuroscience of
consciousness. We then discuss recent proposals that vindi- correlate of consciousness (hereafter NCC) put forward by
cate specific approaches to the subjective, phenomenal Francis Crick and Christof Koch (1995) allows moving
dimension of consciousness while denying the importance of on without focusing too much, at least for now, on the
access mechanisms. In contrast to these proposals, we argue necessity of a reduction to elementary brain structures
for a unified approach to consciousness, whereby experiential
and processes. The combination of these two approaches
and cognitive dimensions of consciousness rely on the same
set of core neural mechanisms. has constituted the core of most recent successes in the
scientific study of consciousness.
Despite all the progress made recently in the scientific
study of consciousness, there are still intense controver- The contrastive approach
sies regarding what a theory of consciousness should be.
In particular, the lack of consensus concerning the psy- The contrastive approach is based on the idea that even
chological definition of consciousness has rendered the if we don’t know what consciousness is, not to mention
study of its neural basis somewhat inconclusive. In this why we are conscious in the first place, we at least know
chapter, we review the progress made so far in uncover- when it happens. Consciousness is thus considered as
ing the neurocognitive mechanisms of conscious access, an outcome variable (absent/present), allowing us to
and we emphasize the dependence of such progress on compare situations where it occurs to close situations
precise, operational definitions of consciousness. We where, all other things being equal, it doesn’t. This
outline some of the main issues that research on this approach enables researchers not only to delineate the
topic faces today, in particular the issue of whether conditions for a stimulus to access consciousness, but
consciousness can be envisioned independently of its also to specify the extents of unconscious processes.
access mechanisms. This strategy has been most successfully applied to
perceptual consciousness, that is, consciousness about
Moving on without definition an external event. Many experiments use very brief
visual stimuli that are sometimes visible and sometimes
Nowadays, the vast majority of scientists reject dualist invisible. Subjects have to report whether they saw a
interpretations of consciousness that imply a separation stimulus, which is taken as an index of whether they
between mind (consciousness, thoughts) and matter were conscious of it or not. By comparing the two situ-
(the brain, neurons). Consciousness is amenable to a ations, one can tell apart which cognitive mechanisms
materialistic approach, with a biological perspective: we are shared, and which are specific to conscious process-
will understand how and why we are conscious by study- ing. Using this methodology, it was shown that some
ing the cerebral and neuronal features of the brain. Yet high-level processes are triggered in the absence of per-
the lack of a consensus definition of consciousness and ceptual consciousness (for instance, extracting the
the reduction of mental states to neuronal structures semantic information of a word or digit; Marcel, 1983a,
856 consciousness
Figure 73.1 Contrast of unconscious and conscious process- restricted to occipitotemporal regions (left panel), conscious
ing in functional MRI (Dehaene et al., 2001). While the perception is associated with the involvement of a parietofron-
neural activation induced by invisible words is primarily tal network (right panel). (See color plate 63.)
other words, the neural mechanisms that are specifi- share several common properties: they are triggered
cally implicated in consciousness are only involved at automatically, they are encapsulated (their internal
the final stage of a long chain of unconscious events. computations are not available to other processors)
Further, these studies pointed to the prefrontal cortex and, importantly, they operate unconsciously.
as an area where neural information converges, creat- Consciousness involves a second type of element,
ing global brain activity and allowing sensory areas to namely, the cortical “workspace” neurons that are par-
interact with other, task-relevant regions. ticularly dense in prefrontal, cingulate, and parietal
regions. These neurons send and receive projections to
The global neuronal workspace many distant areas through long-range excitatory axons,
breaking the modularity of the nervous system and
Such findings have been integrated in a theoretical allowing the domain-specific processors to exchange
framework by Dehaene and his colleagues, Jean- information. The global workspace provides a common
Pierre Changeux and Lionel Naccache (Dehaene & communication protocol by allowing the broadcasting
Changeux, 2011; Dehaene et al., 2006; Dehaene & of information to multiple neural targets. A mental
Naccache, 2001). Called the global neuronal work- state is conscious if two conditions are met. First, the
space, it is a neurobiological extension of the cognitive content of the mental state must be represented as
global workspace theory originally proposed by the psy- an explicit neuronal firing pattern that can reach
chologist Bernard Baars (1989). workspace neurons. Second, top-down amplification
According to this theory, the cerebral architecture is mechanisms mobilizing the long-distance workspace
composed of two qualitatively distinct types of elements. connections must render the representation accessed,
The first type is represented by a large network of sharpened, and maintained. A mental state, even if it
domain-specific processors, in both cortical and subcor- respects the first condition (explicit firing pattern avail-
tical regions, that are each attuned to the processing of able to workspace neurons), will remain unconscious
a particular type of information. For instance, the occip- until its neural signal is amplified. This amplification is
itotemporal cortex is constituted of many such domain- the neural counterpart of top-down attention, which,
specific processors, or “cerebral modules” (movement in this framework, is a necessary condition for con-
processing in MT/V5, face processing in the fusiform sciousness. Whether consciousness requires top-down
face area, etc.). Although these neural processors can attention is a highly debated issue (Cohen, Cavanagh,
differ widely in complexity and domain specificity, they Chun, & Nakayama, 2012; Dehaene et al., 2006; Koch
858 consciousness
Neural purity and the overflow argument 73.2). Using short presentation times and a pioneering
cued report method, Sperling showed that the informa-
Two main empirical arguments, which we termed the tion available for a short period of time after stimulus
overflow argument and the neural purity argument (see presentation vastly exceeded the information subjects
Kouider, de Gardelle, Sackur, & Dupoux, 2010), have could spontaneously report. This has been taken as an
been offered by proponents of the access/phenomenal indication that phenomenal consciousness does indeed
consciousness dissociation. The overflow argument is overflow access (Block, 2007). Yet, as we discuss below,
rooted in the intuition that we are conscious of much it remains controversial whether the large amount of
more than we can describe and manipulate. This intu- available information in cued reports reflects phenom-
ition was operationalized by Sperling over half a century enally conscious representations or unconscious pro-
ago (Sperling, 1960), who used letter arrays to quantify cessing that becomes reportable by virtue of the cues
the amount of information available at a given time (Block, 2007; de Gardelle, Sackur, & Kouider, 2009;
after presentation of a complex visual scene (see figure Dehaene et al., 2006; Sergent et al., 2012).
Figure 73.2 The Sperling paradigm (Sperling, 1960) and its row, performance increased to 3 out of 4 items. This suggests
interpretations. (A) Experimental procedure for the cued that a large amount of information is available but decays by
report. A brief array of letters is shown, followed by a random the time of reporting. (B) Two interpretations of the results.
tone cue (high tone in this example). The pitch of the cue Interpretation 1 assumes that subjects are phenomenally con-
(low, medium, high) instructs subjects to report one of the scious of the whole content in iconic memory demonstrated
three rows (lower, middle, or higher row, respectively). When by the high-level performances at short delays. Interpretation
participants are not cued and have to report all letters in the 2 hypothesizes that subjects access both high- and low-level
array, performance is restricted to about 4 out of 12 items. information from iconic memory. Low-level information is
However, when using the post-stimulus cue to report a specific reconstructed at higher levels. (See color plate 64.)
860 consciousness
observe internal states prior to access will necessarily be of the notions of partial awareness, confidence evalua-
contaminated by access mechanisms themselves. tion, and expectations, access mechanisms can mecha-
A potential escape from the problems outlined above nistically account for phenomenality without reliance on
might be to accept the neural purity argument, accord- specific and dedicated mechanisms for phenomenal
ing to which phenomenological consciousness can be consciousness.
probed regardless of reportability, through neural The notion of “levels of representation” is one of the
indices. However, this strategy is circular, since validat- most venerable notions in cognitive psychology: for
ing the neural index in the first place necessarily instance, a written word might be encoded at the level
requires reliance on access mechanisms. Indeed, dem- of nonspecific geometrical features, letter fragments,
onstrating that a specific neural mechanism (e.g., local specific letter shapes, or abstract letters, and then at
recurrence) is sufficient for consciousness initially lexical, phonological, and semantic levels (Vinckier et
requires the assessment of neural events while probing al., 2007). We know from numerous psycholinguistic
whether the subject is conscious. As the sole uncontro- tasks that these levels of representation are somewhat
versial way to prove consciousness relies on access independent, in the sense that some tasks can require
mechanisms, it appears impossible to map neural and access to one specific level. This kind of representa-
phenomenal states without depending on access. This tional hierarchy is implicit and basic in most areas of
is not to say that we cannot, in some situation, infer cognitive psychology, but has been largely ignored for
conscious contents from brain states. As we gain more consciousness. Recently, we proposed that different
insights into the nature of the brain mechanisms associ- levels of representation of one and the same stimulus
ated with conscious experience, we can reapply this might be separately consciously accessed and lead to
knowledge in cases where reports are impossible, for global broadcasting independently from one another
instance, in cases of patients with locked-in syndrome (Kouider et al., 2010). For instance, because of some
and in vegetative states (Laureys et al., 2005; Owen et degradation, a visual stimulus may only be accessed at
al., 2006), as well as in the case of preverbal infants some lower levels, making it only partially conscious.
(Kouider et al., 2013). But clearly, this extrapolation Thus a word might be accessed at the level of letter
beyond the domain of reportability is justified, because features, while remaining unaccessed at higher levels
we had first relied on conscious reports, and thus on having to do with the whole word form (which does not
access mechanisms. preclude unconscious processing at these higher levels).
Finally, the neural purity argument largely reflects a But conscious contents are not simply stimulus driven:
theoretical confusion: it merely shows that the brain the cognitive system has some a priori knowledge about
processes information without consciousness, but not the world, with some confidence level about the likeli-
that there is phenomenal experience associated with hood of sensory signals. Hence, access to partial infor-
these processes. A supposed neural index of phenom- mation is combined with prior knowledge of what
enal consciousness in the absence of access may thus should be perceived: if participants expect to be shown
simply reflect unconscious processes (Dehaene et al., letters and are partially conscious of letter fragments,
2006; Kouider, Dehaene, Jobert, & Le Bihan, 2007). Yet, they illusorily see letters (de Gardelle et al., 2009). The
because one cannot demonstrate whether phenomenal intuition of a rich, elusive phenomenality comes from
experience is involved or not, the neural purity argu- real-life situations, where stimuli are complex and span
ment becomes unfalsifiable: if, say, local recurrence is a large portion of the visual field. Thus, at each moment,
observed in the absence of conscious access, stipulating various parts of the scene are accessed at different
alternative forms of consciousness, instead of uncon- levels, with restricted levels for eccentric and crowded
scious processing, cannot be verified and simply stimuli. Since the pioneering work of McConkie and
becomes a matter of faith. Rayner (1975), who used eye-tracking methodology to
blur a text beyond a window centered at fixation, it has
Partial awareness and the illusion been known that we do not need rich and detailed
of phenomenal richness information over the entire visual field to produce a
visual consciousness with the impression of richness.
Nevertheless, while phenomenal consciousness seems More recently, Freeman and Simoncelli (2011), using
dubious both from methodological and epistemologi- more controlled methods, constructed stimuli that
cal standpoints, phenomenality in itself is a reality. Our looked exactly alike in spite of systematic distortions at
conscious mental content does seem to exceed all pos- the periphery. Again, this suggests that our visual system
sible reports, and it has a qualitative and subjective accesses only low-level geometrical information in the
“feel” that is private. Here, we explain how, by means periphery of the visual field, and creates on this basis a
862 consciousness
M. Velmans & S. Schneider (Eds.), The Blackwell companion O’Regan, J. K., & Noe, A. (2001). A sensorimotor account of
to consciousness (pp. 616–627). Oxford, UK: Blackwell. vision and visual consciousness. Behav Brain Sci, 24(5), 939–
Koch, C. (2004). The quest for consciousness: A neurobiological 973; discussion 973–1031.
approach. Denver, CO: Roberts. Owen, A. M., Coleman, M. R., Boly, M., Davis, M. H.,
Koch, C. (2012). Consciousness: Confessions of a romantic reduc- Laureys, S., & Pickard, J. D. (2006). Detecting awareness
tionist. Cambridge, MA: MIT Press. in the vegetative state. Science, 313(5792), 1402.
Koch, C., & Tsuchiya, N. (2007). Attention and conscious- Sackur, J. (2013). Two dimensions of visibility revealed
ness: Two distinct brain processes. Trends Cogn Sci, 11(1), by multidimensional scaling of metacontrast. Cognition,
16–22. 126(2), 173–180.
Kouider, S. (2009). Neurobiological theories of conscious- Sackur, J., & Dehaene, S. (2009). The cognitive architecture
ness. In W. Banks (Ed.), Encyclopedia of consciousness, Vol. 2 for chaining of two mental operations. Cognition, 111,
(pp. 87–100). Oxford, UK: Elsevier. 187–211.
Kouider, S., Dehaene, S., Jobert, A., & Le Bihan, D. (2007). Sergent, C., Baillet, S., & Dehaene, S. (2005). Timing of
Cerebral bases of subliminal and supraliminal priming the brain events underlying access to consciousness during
during reading. Cereb Cortex, 17(9), 2019–2029. the attentional blink. Nat Neurosci, 8(10), 1391–1400.
Kouider, S., de Gardelle, V., Sackur, J., & Dupoux, E. Sergent, C., Wyart, V., Babo-Rebelo, M., Cohen, L.,
(2010). How rich is consciousness? The partial awareness Naccache, L., & Tallon-Baudry, C. (2012). Cueing atten-
hypothesis. Trends Cogn Sci, 14(7), 301–307. tion after the stimulus is gone can retrospectively trigger
Kouider, S., Sackur, J., & de Gardelle, V. (2012). Do we conscious perception. Curr Biol, 23, 150–155.
still need phenomenal consciousness? Trends Cogn Sci, Sklar, A. Y., Levy, N., Goldstein, A., Mandel, R., Maril, A.,
16(3), 140–141. & Hassin, R. R. (2012). Reading and doing arithmetic
Kouider, S., Stahlhut, C., Gelskov, S., Barbosa, L., de nonconsciously. Proc Natl Acad Sci USA, 109, 19614–19619.
Gardelle, V., Dutat, M., … Dehaene-Lambertz, G. Sperling, G. (1960). The information available in brief visual
(2013). A neural marker of perceptual consciousness in presentation. Psychol Monogr, 74, 1–29.
infants. Science, 340(6130), 376–380. Tong, F., Nakayama, K., Vaughan, J. T., & Kanwisher, N.
Lamme, V. A. (2006). Towards a true neural stance on con- (1998). Binocular rivalry and visual awareness in human
sciousness. Trends Cogn Sci, 10(11), 494–501. extrastriate cortex. Neuron, 21(4), 753–759.
Laureys, S., Pellas, F., van Eeckhout, P., Ghorbel, S., Tononi, G., & Koch, C. (2008). The neural correlates of
Schnakers, C., Perrin, F., … Goldman, S. (2005). The consciousness: An update. Ann NY Acad Sci, 1124,
locked-in syndrome: What is it like to be conscious but 239–261
paralyzed and voiceless? Prog Brain Res, 150, 495–511. Vinckier, F., Dehaene, S., Jobert, A., Dubus, J. P., Sigman,
Leopold, D. A., & Logothetis, N. K. (1996). Activity changes M., & Cohen, L. (2007). Hierarchical coding of letter
in early visual cortex reflect monkeys’ percepts during bin- strings in the ventral stream: Dissecting the inner organiza-
ocular rivalry. Nature, 379(6565), 549–553. tion of the visual word-form system. Neuron, 55, 143–156.
McConkie, G. W., & Rayner, K. (1975). The span of the Zeki, S. (2007). A theory of micro-consciousness. In M.
effective stimulus during a fixation in reading. Percept Psy- Velmans & S. Schneider (Eds.), The Blackwell companion to
chophys, 17, 578–586. consciousness (pp. 580–588). Oxford, UK: Blackwell.
Milner, A. D., & Goodale, M. A. (1995). The visual brain in Zeki, S., & Bartels, A. (1999). Toward a theory of visual
action. New York, NY: Oxford University Press. consciousness. Conscious Cogn, 8(2), 225–259.
Nagel, T. (1974). What is it like to be a bat? Philos Rev, 83(4),
435–450.
ABSTRACT Recent data have linked self-consciousness to the used video, virtual reality, and robotics technologies to
processing of multisensory bodily signals in temporoparietal study bodily self-consciousness. These findings in
and premotor cortex. Studies in which subjects receive ambig- healthy subjects will be compared with evidence from
uous multisensory information about the location and appear-
ance of their own body have shown that activity in these brain
abnormal states of bodily self-consciousness in two neu-
areas reflects the conscious experience of identifying with the rological conditions, out-of-body experiences and heau-
body (self-identification), with the experience of where “I” am toscopy (Brugger, 2002; Devinsky, Feldmann, Burrowes,
in space (self-location), and with the experience of the per- & Bromfield, 1989). Recent data about three aspects of
spective from where “I” perceive the world (first-person bodily self-consciousness will be discussed, and in par-
perspective). I argue that these data may form the basis for
ticular how these three aspects relate to the processing
a neurobiological model of self-consciousness, grounding
higher-order notions of self-consciousness and personhood in of bodily signals and which functional and neural mech-
multisensory brain mechanisms. anisms they may share. These are self-identification with
the body (i.e., the experience of owning a body), self-
Humans experience a “real me” that “resides” in “my” location (i.e., the experience of where I am in space),
body and is the subject or “I” of experience and thought. and the first-person perspective (i.e., the experience
This is self-consciousness, the feeling that conscious from where I perceive the world).
experiences are bound to the self. Thus, experiences Starting with the breakdown of bodily self-conscious-
happen for someone, the subject or the self, in an ness, the results of detailed observations in neurological
immediate way. They are generally felt as “my” experi- patients who have had out-of-body experiences (char-
ences and seem to be present whenever “I” am perceiv- acterized by abnormal self-identification, self-location,
ing a color, have a thought, or feel pain. Experiences and first-person perspective) will be described. Next,
are felt as belonging to “somebody,” and it is this unitary some of the major experimental paradigms, behavioral
entity, the “I,” that is often considered to be one of the results, and neuroimaging findings will be reviewed. In
most astonishing features of the human mind. So far a final section, I will highlight how the principles of
self-consciousness has mainly been approached by phil- bodily self-consciousness extend to interoceptive signals,
osophical inquiry, and has led to an impressive body describe interactions with visual consciousness, and
of work. However, there is an overabundance of diverg- sketch some limitations of the present approach.
ing theories about the nature of the self and self-
consciousness that are not data-driven (i.e., Aikins, When the self leaves the body: The out-of-body
2005; Bermudez, 1998; Gallagher & Shear, 1999). experience
A powerful approach to investigate self-consciousness
experimentally has been to target brain mechanisms If you ever while lying in bed suddenly had the distinct
that process bodily signals (i.e., bodily self-consciousness; impression of floating up near the ceiling and looking
Blanke & Metzinger, 2009; Christoff, Cosmelli, Legrand, back down at your body on the bed, then it is likely that
& Thompson, 2011; Damasio & Meyer, 2009; Jeannerod, you had an out-of-body experience (OBE). OBEs are
2003; Knoblich, 2002; Legrand, 2007; de Vignemont, striking phenomena because they challenge our every-
2011). The study of such bodily signals is complex since day experience of the spatial unity of self and body: they
bodily input is continuously present and updated challenge our experience of a “real me” that “resides”
and conveyed by many senses, including tactile, pro- in my body and is the subject, or “I,” of experience and
prioceptive, and visual signals about the body. Although thought. As Sylvan Muldoon, one of the first modern
prominent accounts of self-consciousness have also authors to describe an OBE, wrote: “I was floating in
highlighted the importance of motor and interocep- the very air, rigidly horizontal, a few feet above the bed.
tive/homeostatic signals (Craig, 2009; Damasio, 1999; … I was moving toward the ceiling, horizontal and pow-
Frith, 2005; Jeannerod, 2003; Seth, Suzuki, & Critchley, erless. … I managed to turn around and there … was
2012), the present review will focus on multisensory another ‘me’ lying quietly upon the bed” (Muldoon &
mechanisms and highlight recent research that has Carrington, 1929).
866 consciousness
Figure 74.1 Brain damage leading to changes in bodily self- Voxel-based lesion symptom mapping (VLSM; bottom left) of
consciousness (out-of-body experiences, or OBE). Brain focal brain damage leading to OBEs. The violet-to-red cluster
damage and results of lesion overlap analysis (top) in nine shows the region that VLSM analysis associated statistically
patients with OBEs due to focal brain damage are shown. with OBEs as compared to control patients. The color code
Maximal lesion overlap centers at the right temporoparietal indicates significant Z scores of the respective voxels, showing
junction (TPJ) at the angular gyrus (red). Overlap color code maximal involvement of the right TPJ. (Modified from Ionta
ranges from violet (one patient) to red (seven patients). et al., 2011.) (See color plate 65.)
an online video-delay or offline prerecorded data), gen- different visuotactile conflicts, and that these changes
erating synchronous and asynchronous visuotactile can also be achieved in the supine position (Ionta et
stimulation, respectively. al., 2011; Lenggenhager, Mouthon, & Blanke, 2009).
Under these conditions, subjects self-identified with Changes in bodily self-consciousness are also associ-
the seen virtual body, and such illusory self-identifica- ated with an alteration in how visual stimuli interfere
tion with the virtual body was stronger during synchro- with the perception of tactile stimuli that are applied
nous than during asynchronous stroking conditions while stroking is applied to induce full-body and out-of-
(Lenggenhager et al., 2007; for a similar approach, see body illusions. Such visuotactile interference is a behav-
Ehrsson, 2007). Concerning self-location, a forward ioral index of whether visual and tactile stimuli are
drift (as predicted) was found. Thus, self-location was functionally perceived to be in the same spatial location
experienced at a position that was closer to the virtual (i.e., visuotactile cross-modal congruency effect; Iga-
body, as if subjects were located “in front” of the posi- rashi, Kimura, Spence, & Ichihara, 2008; Pavani, Spence,
tion where they had been standing during the experi- & Driver, 2000; Shore, Barnes, & Spence, 2006; Spence,
ment, compatible with a breakdown of the experienced Pavani, & Driver, 2004). When testing such cross-modal
spatial unity between self and body. Later work con- congruency effects during the full-body illusion, it was
firmed that self-location toward and self-identification found that visual stimuli seen at a position 2 meters in
with the virtual body are systematically influenced by front of the subject’s back and tactile stimuli (applied
868 consciousness
mu rhythm suppression is thought to reflect increased vestibular cues from the physical body (indicating
cortical activation in sensorimotor and/or premotor downward gravity directed toward the physical body)
cortices (Oakes et al., 2004). Another functional MRI than on visual gravitational cues from the virtual body
study (Ionta et al., 2011) found that self-identification (indicating downward gravity directed away from the
with a virtual body is associated with activation in the physical body), whereas participants from the Down
right middle-inferior temporal cortex (partially overlap- group show the opposite pattern (relying more strongly
ping with the extrastriate body area), a region that is on visual gravitational cues from the virtual body than
like the premotor cortex involved in the multisensory on vestibular cues from the physical). The role of ves-
processing of human bodies (Astafiev, Stanley, Shulman, tibular processing in the first-person perspective and
& Corbetta, 2004; Downing et al., 2001; Grossman and bodily self-consciousness is supported by the proximity
Blake, 2002). Future work is needed to distinguish of the vestibular cortex to TPJ (Lopez & Blanke, 2011),
regions associated with ownership for the entire body related changes in first-person perspective during the
from those related to ownership for body parts such as so-called inversion illusion (Lackner, 1992; Lackner &
an arm or the face (for review see Blanke, 2012; Makin DiZio, 2002), and frequent vestibular disturbances
et al., 2008). during OBEs. Accordingly, it has been argued that
Recently changes in the experienced direction of the changes in the experienced direction of the first-person
first-person perspective have also been achieved using perspective are due to abnormal signal integration of
robotic stimulation while participants were in a supine otolithic vestibular cues and visual cues (Blanke, 2012;
position and viewed a virtual body that was filmed from for a more extensive review of visuovestibular-somato-
an elevated position (Ionta et al., 2011; Pfeiffer et al., sensory interactions, see Blanke, 2012; Lackner & DiZio,
2013). Despite identical visuotactile stimulation, half of 2002).
the participants experienced looking upward toward
the virtual body (Up group), and half experienced Bodily self-consciousness
looking down on the virtual body (Down group), and
these subjective perspectival changes were associated It has been argued elsewhere (Blanke & Metzinger,
with consistent changes in self-location (Ionta et al., 2009) that the three aspects of bodily self-consciousness
2011; Pfeiffer et al., 2013). In addition, subjective —self-identification, self-location, and first-person per-
reports of elevated self-location and sensations of flying, spective—are the necessary constituents of the simplest
floating, rising, and lightness (which are common in form of self-consciousness, which arises when the brain
out-of-body experiences) were frequent in the Down encodes the origin of the first-person perspective from
group and rare in the Up group. These data show that within a spatial frame of reference (i.e., self-location)
self-location depends on the experienced direction of that is associated with self-identification. The reviewed
the first-person perspective. It also shows that self- data reveal that self-identification depends on somato-
location versus self-identification are associated with sensory and visual signals and likely involves bimodal
different multisensory mechanisms, as self-identification visuotactile neurons that have also been described for
was not found to depend on the first-person arm-related manipulations (i.e., Ehrsson et al., 2004;
perspective. Graziano, Cooke, & Taylor, 2000; Iriki, Tanaka, &
Changes in self-location and the first-person perspec- Iwamura, 1996; Maravita & Iriki, 2004), whereas self-
tive were reflected in bilateral TPJ activity (Ionta et al., location and the first-person perspective depend on the
2011). TPJ activity peaked in the posterior superior additional integration of these bodily signals with ves-
temporal gyri, differed between synchronous and asyn- tibular cues, probably in trimodal visuotactile-vestibular
chronous stroking conditions, and depended on the neurons in posterior parietal and temporoparietal
experienced direction of the first-person perspective. cortex (i.e., Bremmer, Klam, Duhamel, Ben Hamed,
More recent work was able to extend these findings and & Graf, 2002; Duhamel, Colby, & Goldberg, 1998;
show that these subjective changes in the first-person Grüsser, Pause, & Schreiter, 1990). These differences
perspective are associated with interindividual differ- between self-identification versus self-location and first-
ences of visuovestibular integration (Pfeiffer et al., person perspective are corroborated by neuroimaging
2013) such as visual verticality judgments using the and neurological data, showing that self-identification
rod-and-frame test (Isableu, Ohlmann, Cremieux, & recruits primarily bilateral premotor and parietal
Amblard, 1997; Lopez, Lacour, Magnan, & Borel, 2006; regions, whereas self-location and the first-person
Young, Oman, Watt, Money, & Lichtenberg, 1984). perspective recruit posterior parietal–TPJ regions
Based on these data, it can be concluded that partici- (including the insula; Ionta et al., 2014) with a right
pants from the Up group rely more strongly on hemispheric predominance.
870 consciousness
target detection, binocular rivalry, bistable perception,
continuous flash suppression, and other perceptions in
well-defined psychophysical settings (i.e., Tsuchiya &
Koch, 2005). Roy Salomon has taken first steps in this
direction and was able to show that proprioceptive and
vestibular signals as well as changes in self-identification
and self-location (in the full-body illusion) modulate
visual consciousness as quantified in a continuous flash
suppression task (Salomon et al., 2013). These last
experiments suggest that bodily self-consciousness may
turn out to be an important component for conscious-
ness generally (Damasio, 1999). As Gerald Edelman
stated, “It is not enough to say that the mind (and con-
sciousness) is embodied. You also have to say how”
(1992). Bodily self-consciousness may provide this link,
bringing the “I” of conscious perception with conscious-
ness studies.
Conclusion
The “I” of conscious experience is one of the most
astonishing features of the human mind. The reviewed
investigations of self-identification, self-location, and
first-person perspective have described some of the
multisensory brain processes that give rise to bodily
self-consciousness. As argued elsewhere (Blanke &
Metzinger, 2009), these three aspects are the necessary
constituents of the simplest form of self-consciousness,
which arises when the brain encodes the origin of the
first-person perspective from within a spatial frame of
reference (i.e., self-location) associated with self-
identification. Cognitive psychologists and neuroscien-
tists have studied many different aspects of self-related
processes (e.g., Arzy, Thut, Mohr, Michel, & Blanke,
2006; Christoff et al., 2011; Esslen, Metzler, Pascual-
Marqui, & Jancke, 2008; Gillihan & Farah, 2005; Heath-
erton et al., 2006; Jeannerod, 2003; Knoblich, 2002;
Legrand & Ruby, 2009; Macrae, Moran, Heatherton,
Banfield, & Kelley, 2004; Northoff et al., 2006; Perrin et
al., 2005; Platek et al., 2006; Vogeley & Fink, 2003).
Mechanisms of bodily self-consciousness overlap with
several of these self-related processes, and it will be an
872 consciousness
fascicularis): Visual and neck receptor responses. J Physiol, Lopez, C., & Blanke, O. (2011). The thalamocortical vestibu-
430, 559–583. lar system in animals and humans. Brain Res Rev, 67,
Hänsel, A., Lenggenhager, B., von Kanel, R., Curatolo, 119–146.
M., & Blanke, O. (2011). Seeing and identifying with a Lopez, C., Halje, P., & Blanke, O. (2008). Body ownership
virtual body decreases pain perception. Eur J Pain, 15, and embodiment: Vestibular and multisensory mecha-
874–879. nisms. Neurophysiol Clin, 38, 149–161.
Heatherton, T. F., Wyland, C. L., Macrae, C. N., Demos, Lopez, C., Lacour, M., Magnan, J., & Borel, L. (2006).
K. E., Denny, B. T., & Kelley, W. M. (2006). Medial pre- Visual field dependence-independence before and after
frontal activity differentiates self from close others. Soc Cogn unilateral vestibular loss. NeuroReport, 17, 797–803.
Affect Neurosci, 1, 18–25. Macrae, C. N., Moran, J. M., Heatherton, T. F., Banfield,
Heydrich, L., & Blanke, O. (2013). Distinct illusory own- J. F., & Kelley, W. M. (2004). Medial prefrontal activity
body perceptions caused by damage to posterior insula and predicts memory for self. Cereb Cortex, 14, 647–654.
extrastriate cortex. Brain, 136, 790–803. Maillard, L., Vignal, J. P., Anxionnat, R., Taillandier, L.,
Igarashi, Y., Kimura, Y., Spence, C., & Ichihara, S. (2008). & Vespignani, H. (2004). Semiologic value of ictal autos-
The selective effect of the image of a hand on visuotactile copy. Epilepsia, 45, 391–394.
interactions as assessed by performance on the crossmodal Makin, T. R., Holmes, N. P., & Ehrsson, H. H. (2008). On
congruency task. Exp Brain Res, 184, 31–38. the other hand: Dummy hands and peripersonal space.
Ionta, S., Heydrich, L., Lenggenhager, B., Mouthon, M., Behav Brain Res, 191, 1–10.
Fornari, E., Chapuis, D., … Blanke, O. (2011). Multisen- Maravita, A., & Iriki, A. (2004). Tools for the body (schema).
sory mechanisms in temporo-parietal cortex support Trends Cogn Sci, 8, 79–86.
self-location and first-person perspective. Neuron, 70, Muldoon, S. J., & Carrington, H. (1929). The projection of
363–374. the astral body. London: Rider.
Ionta, S., Martuzzi, R., Salomon R., & Blanke, O. (2014). Northoff, G., Heinzel, A., de Greck, M., Bermpohl, F.,
The brain network reflecting bodily self-consciousness: A Dobrowolny, H., & Panksepp, J. (2006). Self-referential
functional connectivity study. Soc Cogn Affect Neurosci. processing in our brain—a meta-analysis of imaging studies
Advance online publication. doi:10.1093/scan/nst185 on the self. NeuroImage, 31, 440–457.
Iriki, A., Tanaka, M., & Iwamura, Y. (1996). Coding of modi- Oakes, T. R., Pizzagalli, D. A., Hendrick, A. M., Horras,
fied body schema during tool use by macaque postcentral K. A., Larson, C. L., Abercrombie, H. C., … Davidson, R.
neurones. NeuroReport, 7, 2325–2330. J. (2004). Functional coupling of simultaneous electrical
Isableu, B., Ohlmann, T., Cremieux, J., & Amblard, B. and metabolic activity in the human brain. Hum Brain
(1997). Selection of spatial frame of reference and postural Mapp, 21, 257–270.
control variability. Exp Brain Res, 114, 584–589 Pacherie, E. (2008). The phenomenology of action: A con-
Jeannerod, M. (2003). The mechanism of self-recognition in ceptual framework. Cognition, 107, 179–217.
humans. Behav Brain Res, 142, 1–15. Pavani, F., Spence, C., & Driver, J. (2000). Visual capture of
Kannape, O. A., & Blanke, O. (2013). The self in motion: touch: Out-of-the-body experiences with rubber gloves.
Sensorimotor and cognitive mechanisms in gait agency. Psychol Sci, 11, 353–359.
J Neurophysiol, 110, 1837–1847. Perrin, F., Maquet, P., Peigneux, P., Ruby, P., Degueldre,
Kannape, O. A., Schwabe, L., Tadi, T., & Blanke, O. (2010). C., Balteau, E., … Laureys, S. (2005). Neural mechanisms
The limits of agency in walking humans. Neuropsychologia, involved in the detection of our first name: A combined
48, 1628–1636. ERPs and PET study. Neuropsychologia, 43, 12–19.
Knoblich, G. (2002). Self-recognition: Body and action. Petkova, V. I., Björnsdotter, M., Gentile, G., Jonsson, T.,
Trends Cogn Sci, 6, 447–449. Li, T. Q., & Ehrsson, H. H. (2011). From part- to whole-
Lackner, J. R. (1992). Spatial orientation in weightless envi- body ownership in the multisensory brain. Curr Biol, 21,
ronments. Perception, 21, 803–812. 1118–1122.
Lackner, J. R., & DiZio, P. (2002). Somatosensory and pro- Petkova, V. I., & Ehrsson, H. H. (2008). If I were you:
prioceptive contributions to body orientation, sensory Perceptual illusion of body swapping. PLoS ONE, 3,
localization, and self-calibration. Adv Exp Med Biol, 508, e3832.
69–78. Pfeiffer, C., Lopez, C., Schmutz, V., Duenas, J. A.,
Legrand, D. (2007). Pre-reflective self-as-subject from expe- Martuzzi, R., & Blanke, O. (2013). Multisensory origin of
riential and empirical perspectives. Conscious Cogn, 16, the subjective first-person perspective: Visual, tactile, and
583–599. vestibular mechanisms. PLoS ONE, 8, e61751.
Legrand, D., & Ruby, P. (2009). What is self-specific? Theo- Pineda, J. A. (2005). The functional significance of mu
retical investigation and critical review of neuroimaging rhythms: Translating “seeing” and “hearing” into “doing.”
results. Psychol Rev, 116, 252–282. Brain Res Brain Res Rev, 50, 57–68.
Lenggenhager, B., Halje, P., & Blanke, O. (2011). Alpha Platek, S. M., Loughead, J. W., Gur, R. C., Busch, S.,
band oscillations correlate with illusory self-location Ruparel, K., Phend, N., … Langleben, D. D. (2006).
induced by virtual reality. Eur J Neurosci, 33, 1935–1943. Neural substrates for functionally discriminating self-face
Lenggenhager, B., Mouthon, M., & Blanke, O. (2009). from personally familiar faces. Hum Brain Mapp, 27,
Spatial aspects of bodily self-consciousness. Conscious Cogn, 91–98.
18, 110–117. Salomon, R., Lim, M., Herbelin, B., Hesselmann, G., &
Lenggenhager, B., Tadi, T., Metzinger, T., & Blanke, O. Blanke, O. (2013). Posing for awareness: Proprioception
(2007). Video ergo sum: Manipulating bodily self- modulates access to visual consciousness in a continuous
consciousness. Science, 317, 1096–1099. flash suppression task. J Vis, 13, 2.
874 consciousness
75 Intention and Agency
PATRICK HAGGARD
ABSTRACT Cognitive neuroscience makes an important dis- relevant for the diagnosis and treatment of the many
tinction between actions that are triggered by a specific stimu- neuropsychiatric disorders that affect capacity for vol-
lus in the external world (i.e., reactions) and actions that are untary action. Third, and perhaps most importantly,
relatively independent of any individual stimulus. These
internally-generated actions instead result in a combination neuroscientific findings may have important social
of motivation to act and an internal decision about what implications, because human society depends on a
action to make and when to make it. Some internally-gener- concept of individual autonomy and responsibility for
ated actions are accompanied by a characteristic conscious action.
experience of “willing” or intending to move, which is clearly
linked to the ability to initiate the action. This chapter focuses
on the brain mechanisms and computations underlying inten- What is intentional action?
tional action. The medial frontal cortex plays a key role in
generating intentional actions and, in conjunction with the Ludwig Wittgenstein famously asked, “What is left over
parietal cortex, in the subjective experience of intention. The if I subtract the fact that my arm goes up from the fact
second part of the chapter focuses on the sense of agency—
that I raise my arm?”(Wittgenstein, 1953, §621). We all
the experience that our actions aim at external outcomes and
are the causes of those outcomes. The same frontal brain know that intentionally lifting one’s arm and having
circuits that initiate intentional action are also responsible for one’s arm lifted feel quite different, but cognitive psy-
linking actions to outcomes. Some evidence suggests the chology has nevertheless struggled to provide a good
action-outcome link is a prospective prediction of outcomes definition of intentional actions, where there is a direct
within action-related circuits, rather than a retrospective
and immediate correspondence between stimulus and
causal inference within outcome-related circuits. The chapter
ends with a brief discussion of the implications of cognitive action. Intentional actions are not driven by any obvious
neuroscience for social and moral responsibility. or immediate external stimulus. For example, my leg
may kick forward because of a sharp tap on the tendon
Intentions are mental states that represent one’s future just below the kneecap (a reflex), or because I just
actions. However, not all actions are intentional. The decide to kick my leg forward. In essence, this approach
key tasks of the cognitive neuroscience of human inten- defines intentional actions as internally-generated,
tional action are to identify the specific features of rather than externally-triggered (Passingham, Bengts-
intentional actions as opposed to other kinds of actions, son, & Lau, 2010). This definition has dominated recent
to investigate the specific mechanisms of intentional thinking about intentional action, and has both advan-
action in the brain, and to consider the implications of tages and disadvantages. The main disadvantage is that
these mechanisms for human cognition in general. the definition says what intentional actions are not, but
The cognitive neurosciences’ interest in intentional does not say what they are in a positive sense. In particu-
action emerges from two sources: the classic philosophi- lar, the causes of intentional action remain largely
cal interest in free will and the increasing mechanistic undefined. On the other hand, this definition gives a
understanding of the motor circuits in the brain. clear operational definition of intentional action that
Healthy human adults generally have the experience can be used in experimental designs. Indeed, many
that they can choose their actions for themselves on at studies have compared the brain mechanisms for inter-
least some occasions. This makes them responsible for nally-generated versus externally-triggered actions.
an action and its outcomes. However, the neurosciences These studies are reviewed in the next section.
view actions as events in the motor system, caused by Several studies have compared the brain processes
neural mechanisms in cortical and subcortical cognitive for internally-generated versus externally-triggered
and motor areas. The first view is person-centered and actions. The design of these studies aims to make the
subjective, while the second is system-centered and physical movement, and any physical stimulation, bal-
mechanistic. Reconciling these two very different views anced across experimental conditions. Differences
of human action represents an important frontier in between conditions can then be linked to the different
our scientific understanding of ourselves. Moreover, sci- causes of action. For example, an early positron emis-
entific investigation of intention and agency may be sion tomography study (Deiber et al., 1991) asked
876 consciousness
direct response to the $20 stimulus than an internally-
generated action. Because of this ambiguity, many
human experiments used arbitrary, valueless action
decisions. Such experiments clearly cannot capture
how everyday intentional actions follow from goals and
reasons, but they may be able to study the process of
generating intentional action.
A third critical element of intention is the timing of
initiation of action itself. The combination of a motiva-
tion to act and a decision about which action to make
is not always sufficient to trigger action. For example,
action may be physically prevented by external con-
straint, or not appropriate given current circumstances,
competing actions, or competing goals. For these
reasons, the brain needs a mechanism to maintain
intentions and release them only at the appropriate
time: another form of “freedom from immediacy.” The
temporal aspect of intention has received more experi-
mental attention than any other, and forms the topic of
the next section.
The combination of decision, motivation, and timing
of initiation gives intentional actions a level of flexibility
and complexity that stimulus-driven actions lack. The Figure 75.1 Computational motor control model for goal-
traditional definition of intentional action can some- directed action. (Reprinted with permission from Frith et al.,
times be misinterpreted as implying an action with no 2000. Copyright © 2000, The Royal Society.)
apparent cause—something that is clearly nonsensical.
Perhaps it is better to think of internally generated quite different ways within cognitive neuroscience. In
actions as responses where a combination of several cognitive psychology, intentional action goals are often
factors jointly cause action (Schüür & Haggard, 2011). considered as prospective memories to perform a
desired action at a later, appropriate time. In senso-
Mental chronometry of intentions rimotor control, the goal is a state of the body, such as
getting the hand to a particular target position in space.
Mental chronometry has been a rich line of investiga- The main difference between the two approaches is
tion in many areas of psychology, and no more so than how close the person is to the time of action. Prospec-
in intentional action. Recent computational models tive memory involves long-term or prior intentions:
(see figure 75.1) provide a useful framework for con- other events and other actions may occur before the
sidering intentions and time. intended action. Sensorimotor control models, in con-
In these models (see chapter 39), actions begin with trast, begin with the immediate current intention or
specification of a goal. A “planner,” or inverse model, goal and are assumed to operate over the short term.
converts the goal into a set of motor commands to They therefore deal with intentions-in-action rather
achieve the goal. These commands are sent from the than long-range intentions (Searle, 1983). An external
brain to the spinal cord and the muscles, causing physi- triggering event, or an internal representation of
cal movement and finally bringing about the goal state. elapsed time, is required to retrieve a relevant prior
A copy of the motor command is sent to an internal intention from long-term memory and transform it to
forward model, which predicts the effects of the motor an intention-in-action: this process marks the trigger or
command. Therefore, the system potentially has access decision to make the action now. This progression
to several different kinds of information about a current toward triggering action is associated with a gradient
action, each relating to a different time: a description across the prefrontal cortex: more anterior regions deal
of the future or goal state, an estimate of the current with more abstract, longer-range intentions, while more
state, and a delayed description based on feedback posterior regions deal with specific action details
from the body and the external world. (Koechlin & Summerfield, 2007; Pacherie, 2008).
This model of intentional action begins with goals, Neuroimaging studies identify the lateral part of
but the concept of action goal has been viewed in two Brodmann’s area (BA) 10 in the prefrontal cortex as a
878 consciousness
However, since numerous brain processes contribute to may be part of the normal regulatory process that allows
preparation and control of actions, an important scien- us to make an action only when it is appropriate, and
tific question is to identify those that underlie the expe- not at other times when it would be inappropriate.
rience of intention. Fried and colleagues (Fried, Recent work in healthy humans (van den Wildenberg
Mukamel, & Kreiman, 2011) recorded from 1,019 indi- et al., 2010) and monkeys (Wardak, 2011) confirms that
vidual neurons in the brains of patients undergoing deciding whether to act and deciding when to act are
preoperative monitoring before epilepsy surgery, while closely linked. Even in a simple reaction task, cognitive
the patients performed the Libet experiment. They motor areas maintain a tonic inhibitory influence on
found several neurons in the medial frontal cortex that the primary motor cortex until just before the time of
progressively modulated their firing rate before the the expected “go” signal (Duque et al., 2010).
patients’ average reported time of conscious intention. Intentional inhibition has sometimes been viewed
Interestingly, these neurons could either increase dualistically, as a form of “conscious veto” underlying
or decrease their firing rates as the action onset self-control (Libet, 2005). However, the neuropsycho-
approached, suggesting that medial frontal areas may logical cases described above show that a specific frontal
initiate action by a balance of excitatory and inhibitory brain mechanism is responsible for inhibiting actions
drives onto motor-execution areas. We shall return to that might, in principle, be executed. Neuroimaging
this point later. They also showed that they could accu- studies of inhibition broadly agree with this view. The
rately predict the time of the conscious intention to classic experimental paradigms for studying inhibition
move from a population of a few hundred such medial generally involve an external stop or no go signal—yet
frontal neurons. Neurons in other brain areas, such as in everyday life, healthy adults are expected to
the temporal lobe, showed much less association with inhibit inappropriate actions endogenously, without
conscious intention. any explicit instruction (Aron, 2011; Filevich, Kühn, &
Haggard, 2012). Therefore, intentional paradigms such
“Negative volition” and inhibition of action as the Libet experiment have also been used to investi-
gate inhibition (Brass & Haggard, 2007). Participants
An interesting recent debate has focused on decisions were asked to prepare voluntary key-press actions, but
to inhibit intentional actions. Behavioral sciences have then to cancel their action at the last possible moment
often struggled to study inhibition of action, largely on a freely chosen subset of trials. These intentional
because it produces no measurable behavioral output. inhibition trials present an unusual experimental chal-
As a result, the contribution of inhibitory processes to lenge, because the input is not directly controlled by
intentional action has generally been underestimated. the experimenter’s instructions, nor is there any behav-
Nevertheless, classical neuropsychological studies ioral output to measure! Crucially, however, partici-
showed that medial frontal lobe damage could cause a pants indicated the time at which they experienced
form of excessive, compulsive action, in which the willing the action using a rotating clock hand, even if
patient’s actions would be triggered automatically by the action was subsequently canceled. This subjective
objects that happened to be present in the environ- marker allowed the authors to investigate brain activity
ment, even when no particular reason or desire to time-locked to intending actions and then inhibiting
perform the action could be detected. These were them. A region of BA 9 in the dorsomedial prefrontal
termed utilization behaviors (Lhermitte, 1983). This cortex was found to be activated in inhibition trials, but
finding suggests that one normal function of the healthy deactivated in action trials. A later study, using a differ-
medial frontal cortex may be to inhibit those actions ent experimental paradigm, additionally showed that
that we currently do not wish to make, and not simply this area had a strong effective connectivity with the
to initiate those actions we do wish to make. A dramatic preSMA during inhibition trials, relative to action trials
example comes from the neurological syndrome of (Kühn, Haggard, & Brass, 2009).
anarchic hand. This unilateral form of utilization behav- Based on this evidence, it was suggested that this
ior sometimes follows a medial frontal and callosal dorsomedial frontal area may express the intention to
lesion. Della Sala and colleagues (Della Sala, Marchetti, inhibit actions, and may do so by exerting control over
& Spinnler, 1991) report the case of a patient whose action-preparation circuits elsewhere in the frontal
anarchic right hand reached out involuntarily to grab cortex. Interestingly, a similar prefrontal region, with
a hot beverage immediately after the patient announced similar connectivity to preSMA, was activated when par-
that she had decided to wait for the drink to cool before ticipants intentionally decided to resist feeling the
taking it. This compelling example suggests that the emotion suggested by unpleasant visual stimuli (Kühn,
maintenance, and then eventual release, of inhibition Haggard, & Brass, 2013). This region may therefore
880 consciousness
relative imperception of external events caused by one’s participants’ feeling of control reduced, but only for
own agency. incompatibly primed trials. Crucially, the primes in this
A recent study resolved some of these uncertainties experiment did not predict the color patches that
with a novel interpretation of the role of the angular appeared after each action. Therefore, the influence of
gyrus in sense of agency. Chambon and colleagues priming on sense of agency was interpreted as a pro-
(Chambon, Wenke, Fleming, Prinz, & Haggard, 2013; spective fluency effect: people feel a stronger sense of
see figure 75.3) used subliminal arrow stimuli to prime control when it is easy to select which action to take,
participants’ actions, so that selecting the appropriate irrespective of the statistical relation between their
action in response to a left or right target arrow was action and outcome. People mistake the ease of choos-
either facilitated or inhibited. After the participants ing what to do for actually achieving something. Inter-
responded, one of several color patches appeared on estingly, the angular gyrus showed increasing levels of
the screen after a short delay. Participants reported how activation as the subjective sense of control over the
much control they felt they had over the color appear- color path decreased, but only for incompatible trials.
ing on the screen. The results showed a higher sense of This result confirms the negative coding for nonagency
control over action outcomes when participants’ in the angular gyrus. However, this code may be pro-
responses had been facilitated by subliminal priming, spectively based on what we think we may achieve, and
relative to when they had been impaired. Interestingly, not just retrospectively based on what we have actually
the angular gyrus showed increasing activation as the achieved.
Figure 75.3 Subliminal priming of actions influences sense panel: activation of the angular gyrus associated with the
of agency prospectively. Upper panel: subliminal primes facilitate target-response event varies negatively with perceived level of
or impair actions in response to target arrow stimuli. Responses control, but only following incompatible priming. (Repro-
are followed by appearance of a color patch, and participants duced from Chambon et al., 2013, by permission of Oxford
judge how much control they have over the color patch. Lower University Press.) (See color plate 67.)
882 consciousness
This evidence can only be reviewed briefly here. First, Responsibility for action
intentional binding effects were reduced after inactiva-
tion of the preSMA caused by repetitive transcranial All human societies appear to hold individuals respon-
magnetic stimulation (Moore, Ruge, Wenke, Rothwell, sible for their actions. In many systems of law, theories
& Haggard, 2010). Second, Parkinsonian patients of responsibility are explicitly dualist. The agents’
showed significantly stronger intentional binding after capacity for rational thought, their conscious choice
taking their dopaminergic medication than immedi- over their actions, and their freedom to do otherwise—
ately before, suggesting that dopaminergic drive in the all make them responsible for what they do. In dramatic
basal-ganglia-thalamocortical loop strongly contributes contrast, neuroscience ultimately views intentional
to sense of agency (Moore, Schneider, et al., 2010). actions as consequences of specific, deterministic elec-
Finally, an fMRI study showed that activation of the trical and chemical brain processes. Modern cognitive
caudal SMA correlated more strongly with the per- neuroscience rejects dualist notions of brain-indepen-
ceived interval between voluntary actions and tones dent consciousness, and instead views an individual’s
than between passive movements and identical tones choices, actions, and even his or her character as mech-
(Kühn, Brass, & Haggard, 2012). anistic functions of his or her brain. For the neurosci-
entist, the conscious, rational thought is simply an
additional brain process, which is presumably just as
Context and belief in the conscious experience determined as all others. These opposing views form
of intentional action the basis of a lively debate between law and neurosci-
ence. For example, if psychopathy is a neuropsychiatric
Taken together, these results suggest that the processes disorder, are psychopaths truly responsible for their
of preparing and initiating voluntary actions in the actions? Can neuroscience help to guide such individu-
frontal lobes also involve a prediction of the anticipated als toward more prosocial behaviors (Anderson & Kiehl,
consequences of action, much as the computational 2013)? Greene and Cohen argue that if neurobiological
motor control model suggests. Importantly, the experi- determinism is true, then retributive justice and punish-
ence of agency, or causing the effects of one’s action, ment are misguided. To the extent that a criminal’s
appears to derive from the same circuits that develop brain “made them do it,” then there is little point in
action itself. Thus, the sense of agency may not simply punishing criminals for their actions (Greene & Cohen,
be an inference or confabulation of authorship, but a 2004). Of course, the same determinist neuroscientific
measurable internal signal within the motor system. theories of action offer ready support for a rehabilita-
There is currently a lively controversy regarding the tive view of justice. Rehabilitation of offenders could, in
division of labor between the frontal and parietal lobes principle, provide learning experiences that change the
in both intention and agency. The conscious urge to brain, and thus behavior.
move that was previously identified with medial frontal Humans are social animals, and our brain mecha-
cortex was also recently reported after direct electrical nisms have coevolved with our increasingly large and
stimulation of parietal cortex (Desmurget et al., 2009), diverse societies. To participate in such a society requires
while patients with parietal lesions had a delayed expe- a brain that is able to follow, or learn to follow, the social
rience of intention to move (Sirigu et al., 2004). The and moral codes by which societies work. Equally, soci-
involvement of parietal cortex in agency is also clear, eties will generally try to protect themselves against pos-
since the angular gyrus is routinely activated in judg- sible harm from individuals who do not follow these
ments of (non)agency, as we have seen. However, frontal codes. The methods of protection are costly, since they
activations did correlate with implicit markers of sense involve either costly care for those who are incapable
of agency, whereas parietal activations did not (Kühn of following such codes or costly punishment for those
et al., 2012). Therefore, one interim hypothesis sug- who are judged capable, but who transgress. For a strict
gests that the frontal lobe houses the core machinery neurobiological determinist, individuals are punished
for voluntary action, including the prospective con- not so much for the action that they performed, but for
struction of experience of goal-directed action and the having the kind of brain they have—since it was their
low-level experience of agency. The parietal lobes, in brain that caused their action. The consequences of this
contrast, would monitor the motor instructions gener- view are highly unpalatable and morally unacceptable!
ated frontally. By comparing these instructions with However, even a determinist may agree that responsibil-
later sensory inputs, a parietal comparator could detect ity and punishment are more acceptable in a society
nonagency and contribute to explicit judgments of where all individuals have an equal opportunity to learn
agency. to follow legal and moral codes of behavior than in a
884 consciousness
Libet, B., Gleason, C. A., Wright, E. W., & Pearl, D. K. Shadlen, M. N., & Gold, J. I. (2004). The neurophysiology
(1983). Time of conscious intention to act in relation to of decision-making as a window on cognition. In M. S.
onset of cerebral activity (readiness-potential). The uncon- Gazzaniga (Ed.), The cognitive neurosciences (3rd ed.,
scious initiation of a freely voluntary act. Brain, 106(Pt. 3), pp. 1229–1241). Cambridge, MA: MIT Press.
623–642. Sirigu, A., Daprati, E., Ciancia, S., Giraux, P., Nighoghos-
Momennejad, I., & Haynes, J.-D. (2012). Human anterior sian, N., Posada, A., & Haggard, P. (2004). Altered aware-
prefrontal cortex encodes the “what” and “when” of future ness of voluntary action after damage to the parietal cortex.
intentions. NeuroImage, 61(1), 139–148. Nat Neurosci, 7(1), 80–84.
Moore, J. W., Cambridge, V. C., Morgan, H., Giorlando, F., Sirigu, A., Daprati, E., Pradat-Diehl, P., Franck, N., &
Adapa, R., & Fletcher, P. C. (2013). Time, action and Jeannerod, M. (1999). Perception of self-generated move-
psychosis: Using subjective time to investigate the effects ment following left parietal lesion. Brain, 122(Pt. 10),
of ketamine on sense of agency. Neuropsychologia, 51(2), 1867–1874.
377–384. Smith, A. B., Taylor, E., Brammer, M., Toone, B., & Rubia,
Moore, J. W., Ruge, D., Wenke, D., Rothwell, J., & Haggard, K. (2006). Task-specific hypoactivation in prefrontal and
P. (2010). Disrupting the experience of control in the temporoparietal brain regions during motor inhibition and
human brain: Pre-supplementary motor area contributes to task switching in medication-naive children and adoles-
the sense of agency. Philos Trans R Soc Lond B Biol Sci, cents with attention deficit hyperactivity disorder. Am J Psy-
277(1693), 2503–2509. chiatry, 163(6), 1044–1051.
Moore, J. W., Schneider, S. A., Schwingenschuh, P., Synofzik, M., Vosgerau, G., & Newen, A. (2008). Beyond the
Moretto, G., Bhatia, K. P., & Haggard, P. (2010). Dopa- comparator model: A multifactorial two-step account of
minergic medication boosts action-effect binding in Parkin- agency. Conscious Cogn, 17(1), 219–239.
son’s disease. Neuropsychologia, 48(4), 1125–1132. Thaler, D., Chen, Y. C., Nixon, P. D., Stern, C. E., & Pass-
Moore, J., & Haggard, P. (2008). Awareness of action: Infer- ingham, R. E. (1995). The functions of the medial premo-
ence and prediction. Conscious Cogn, 17(1), 136–144. tor cortex. I. Simple learned movements. Exp Brain Res,
Moretto, G., Walsh, E., & Haggard, P. (2011). Experience 102(3), 445–460.
of agency and sense of responsibility. Conscious Cogn, 20(4), Van den Wildenberg, W. P. M., Burle, B., Vidal, F., van der
1847–1854. Molen, M. W., Ridderinkhof, K. R., & Hasbroucq, T.
Mueller, V. A., Brass, M., Waszak, F., & Prinz, W. (2007). (2010). Mechanisms and dynamics of cortical motor inhibi-
The role of the preSMA and the rostral cingulate zone in tion in the stop-signal paradigm: A TMS study. J Cogn Neu-
internally selected actions. NeuroImage, 37(4), 1354–1361. rosci, 22(2), 225–239.
Pacherie, E. (2008). The phenomenology of action: A con- Voss, M., Moore, J., Hauser, M., Gallinat, J., Heinz, A., &
ceptual framework. Cognition, 107(1), 179–217. Haggard, P. (2010). Altered awareness of action in schizo-
Passingham, R. E., Bengtsson, S. L., & Lau, H. C. (2010). phrenia: A specific deficit in predicting action conse-
Medial frontal cortex: From self-generated action to reflec- quences. Brain, 133(10), 3104–3112.
tion on one’s own performance. Trends Cogn Sci, 14(1), Wardak, C. (2011). The role of the supplementary motor
16–21. area in inhibitory control in monkeys and humans. J Neuro-
Rubia, K., Overmeyer, S., Taylor, E., Brammer, M., Wil- sci, 31(14), 5181–5183.
liams, S. C., Simmons, A., & Bullmore, E. T. (1999). Hypo- Wegner, D. M. (2003). The illusion of conscious will. Cambridge,
frontality in attention deficit hyperactivity disorder during MA: MIT Press.
higher-order motor control: A study with functional MRI. Wegner, D. M., & Wheatley, T. (1999). Apparent mental
Am J Psychiatry, 156(6), 891–896. causation. Sources of the experience of will. Am Psychol,
Schüür, F., & Haggard, P. (2011). What are self-generated 54(7), 480–492.
actions? Conscious Cogn, 20(4), 1697–1704. Wittgenstein, L. (1953). Philosophical investigations. Hoboken,
Searle, J. R. (1983). Intentionality. Cambridge, UK: NJ: Blackwell.
Cambridge University Press.
78 ASSAF 921
83 MARCUS 981
Introduction
B. A. WANDELL
ABSTRACT Functional MRI is increasingly used to study the possible noninvasively in vivo by diffusion MRI (dMRI)
connectivity in large-scale brain networks that supports per- and computational tractography analysis (Johansen-
ception, action and cognition. This chapter focuses on the
Berg & Behrens, 2009; Johansen-Berg & Jbabdi, 2011;
modeling of causality (directed influence or effective con-
nectivity) and how this can be applied to fMRI data obtained Jones, 2010). Functional connectivity refers to the correla-
in well-designed cognitive, perceptual, or motor tasks. This tion structure (or, more generally, any order of statisti-
kind of modeling entails the estimation of multivariate math- cal dependency) in the data such that brain areas can
ematical models from the data record which embed assump- be grouped into interacting networks. Finally, effective
tions that allow us to move beyond mere correlations between connectivity modeling moves beyond statistical depen-
activity signals. We discuss the main challenges of causal fMRI
data modeling: the missing region problem, the missing time dency to measures of directed influence and causality
problem, and the missing model problem. We then suggest within networks (Friston, 1994).
how greatly increased spatial and temporal resolution of This chapter focuses on the latter: modeling of
acquisition can assist in addressing these challenges and, directed influence, effective connectivity, or causality
potentially, move computational causal fMRI modeling into and how this can be applied to fMRI data obtained in
the realm of cortical microcircuit dynamics.
well-designed cognitive, perceptual, or motor tasks.
Causal modeling of this kind entails the estimation of
Understanding how interactions between brain struc- multivariate mathematical models from the data record.
tures support the performance of specific cognitive These models embed assumptions (implicitly or explic-
tasks or perceptual and motor processes is a prominent itly) that allow us to move beyond correlations to direc-
goal in cognitive neuroscience. Neuroimaging methods, tions of information flow. Statistical inference on
such as electroencephalography (EEG), magnetoen- parameters estimated in the context of such models
cephalography (MEG), and functional magnetic reso- then provides quantification of directed connectivity,
nance imaging (fMRI), are ever more employed to and inference can take place on individual parameters
address questions of functional connectivity, inter- or on a group of parameters together; the latter is often
region coupling, and networked computation that go referred to as model comparison. We can partition
beyond the “where” and “when” of task-related activity models of causal influence into two parts, each neces-
(Friston, 1994; Horwitz, Friston, & Taylor, 2000; McIn- sitating choices and assumptions: the structural model
tosh, 2004; Salmelin & Kujala, 2006; Valdes-Sosa, Kotter, and the dynamical model (figure 76.1).
& Friston, 2005). A network perspective onto the paral- The structural model contains (1) a selection of the
lel and distributed processing in the brain—even on the regions of interest (ROIs) in the brain that are assumed
scale accessible by neuroimaging methods—is a promis- to be of importance in the cognitive process or task
ing approach to enlarge our understanding of percep- under investigation, (2) the possible interactions
tual, cognitive, and motor functions. Functional MRI, between those structures, and (3) the possible effects
in particular, is increasingly used, not only to localize of exogenous inputs onto the network. Note that
structures involved in cognitive and perceptual pro- although ideally the structural model within a causal
cesses, but also to study the connectivity in large-scale modeling framework is guided by measurement or
brain networks that support these functions. knowledge of direct structural connectivity (as defined
Generally, a distinction is made between three types above), it is not identical to the concept of structural
of brain connectivity (Sporns, 2010). Structural connec- connectivity. For instance, a structural model can
tivity (or anatomical connectivity) refers to the physical contain connections between ROIs that are hypothe-
presence of an axonal projection from one brain area sized or thought to exist only indirectly via several relay
to another. Identification of large axon bundles con- stations. The exogenous inputs may be under control
necting remote regions in the brain has recently become of the experimenter and often have the form of a simple
Figure 76.2 The neuronal, physiological, and physical pro- cerebral metabolic rate of oxygen, TE: time to echo, TR: time
cesses (top row) and variables and parameters involved to repeat, SE: spin echo, GRE: gradient echo, SNR: signal-to-
(middle row) in the complex causal chain of events that leads noise ratio). The bottom row lists some mathematical models
to the formation of the fMRI signal (ePSP, excitatory postsyn- of the subprocesses that play a role in the analysis and causal
aptic potential; iPSP, inhibitory postsynaptic potential; FIR, modeling of fMRI signals. (Adapted from Roebroeck, Seth,
finite impulse response; B0, main magnetic field, CBF: cere- et al., 2011.)
bral blood flow, CBV: cerebral blood volume, CMRO2:
that BOLD fMRI at UHF has enough spatial specificity Complementary to tangential, and especially radial,
to discriminate (small contiguous sets of) layers and specificity afforded by very high spatial resolution
sample information about their connectivity and func- fMRI, there are other ways of deriving directionality
tional roles in cortical activity dynamics. Zimmermann from space that rely on sophistication of analysis. A
and colleagues (2011) show an example of a human case in point is the study from Heinzle, Kahnt, and
visual system study with high spatial resolution UHF Haynes (2011), who use a support vector machine–
fMRI that illustrates the possibility of combining the based regression to explain extrastriate human response
cortical tangential and radial specificity. In their study, from those in V1. The support vector machine helps
these authors distinguished column-level and layer-level reduce the dimensionality of the large number of can-
axis-of-motion selectivity in MT simultaneously (figure didate voxels in V1 by selecting a sparse set of best-
76.1C). predicting voxels. They find a clear, topographically
BOLD fMRI responses after whisker stimulation show an we call static techniques). Within the class of dynamic
earlier response onset at layer IV in the barrel cortex. In techniques, two separate developments have been
contrast, BOLD responses in motor cortex through most used: Granger causality analysis (GCA; Goebel,
cortico-cortical somatomotor connections had a simul- Roebroeck, Kim, & Formisano, 2003; Roebroeck,
taneous early onset in layer II–III and layer V. These Formisano, & Goebel, 2005; Valdes-Sosa, 2004) and
results show strong agreement of laminar fMRI onset dynamic causal modeling (DCM; Daunizeau, David, &
(i.e., both spatially and temporally) with input projec- Stephan, 2009; Friston, Harrison, & Penny, 2003).
tions into the cortex. Despite the common goal, there are differences between
the two methods. Whereas GCA explicitly models tem-
Causal time-series modeling of fMRI poral precedence and uses the concept of Granger cau-
sality (or G-causality), usually formulated in a discrete
Recently, causal modeling techniques that make use of time-series analysis framework, DCM employs a
the temporal dynamics in the fMRI signal and employ biophysically motivated generative model formulated
time-series analysis and systems identification theory in a continuous-time, dynamic-system framework.
have become popular. This modeling of temporal fMRI And although these approaches have recently started
signal structure, and its use for causal direction estima- developing in an integrated single direction (Valdes-
tion, sets these dynamic techniques apart from earlier Sosa, Roebroeck, Daunizeau, & Friston, 2011), initially
methods that do not use temporal structure (and that each was focused on separate issues that pose
in the coefficient matrices (A, Bj, and C in figure 76.5) population dynamics can be captured without error
from measurements y[t] and, possibly, the inputs u[t]. from the chosen inputs and the transformation of that
The first important difference in modeling signal input in its “flow” through the DCM network. The exog-
dynamics is that LSMs employ linear stochastic transi- enous inputs mostly have a very simple form, such as a
tion equations, whereas those in the most-used forms stimulus function that represents the presence or
of DCM are bilinear and deterministic. The stochastic absence of a visual stimulus or level of experimental
term in the LSM transition equation allows for variation manipulation, such as attention left versus right. The
in the state variables that cannot be explained by inability of DCM to model signal variations beyond
the inputs u[t]. In fact, in the case of a purely autore- those implied by the exogenous inputs makes its con-
gressive model, exogenous inputs are absent and all nectivity estimation highly dependent on the exact
signal variation is modeled as driven by uncorrelated number and form of the assumed inputs and the form
stochastic processes (called “innovations”). This forces of the structural model. Stochastic extensions to DCM
all dynamic and spectral complexity in the observed have been developed (Daunizeau, Friston, & Kiebel,
signals to be represented in the model parameters. It is 2009; Friston, Stephan, Li, & Daunizeau, 2010). These
exactly this property of comprehensive and flexible rep- developments clearly have the potential to eliminate
resentation of signal dynamics and spectral properties one of the differentiating aspects of LSMs and deter-
that has made autoregressive models a popular tool in ministic DCMs and bring the models even closer
analyzing complex biophysical signals (Bernasconi & together. Interestingly, the inclusion of noise in the
Konig, 1999; Brovelli et al., 2004; Ding, Bressler, Yang, state equations makes inference on stochastic DCMs
& Liang, 2000; Harrison, Penny, & Friston, 2003; Kamin- usefully interpretable in the stochastic framework of
ski, Ding, Truccolo, & Bressler, 2001). In contrast, the WAGS influence (Valdes-Sosa et al., 2011).
transition equation in DCM for fMRI, in its most used The second important difference in modeling signal
form, does not have a stochastic term. As a consequence, dynamics is that, in DCM, the state variables are given
any and all signal dynamics that it can capture are a definite physical interpretation within a generative
limited to the signal subspace spanned by the assumed model of the data. For every selected region, a single
inputs. In other words, it assumes that all neural state variable represents the neuronal or synaptic
Since the total chain of temporal causality in social com- well, we call the model unidentifiable. If we are faced
munication can span intervals of seconds (figure 76.6A), with an unidentifiable model, we have only a few
it can be very informative to investigate time-delayed options: give up (cf. part one of the missing model
information between the brains of the communicators. problem), simplify the model (and inevitably make it
The time-lagged flow of information was resolved with less realistic), try to acquire data that supports its iden-
G-causality analysis and, as expected, the between-brain tification better, or increase the number and weights of
G-causality from gesturer (information sender) to assumptions or Bayesian priors that force parameters to
guesser (information receiver) was found to be signifi- a priori values. It could be supposed that part two of
cant, in contrast to that in the anticausal direction the missing model problem is largely solved in model
(figure 76.6B). More importantly as a finding in social comparison frameworks that use a mixture of model fit
neuroscience research, the brain areas found to be and model complexity penalty as a criterion, such as the
communicating were found to be part of the putative Bayesian information criterion (BIC) or free energy.
human mirror system in both brains. This illustrates the Indeed, the use of such model comparison criteria is to
use of the nongeneratively modeled G-causality concept be preferred over model fit alone in any model selec-
to answer interesting neuroscience questions that (1) tion or comparison step. However, the weights of com-
are not easily formulated by a generative model, for plexity and model fit in the balance are dictated by the
instance, incorporating the entire interpersonal com- criterion chosen by the researcher. What is the “best”
munication chain, and (2) are concerned with the iden- model (among those considered) according to one cri-
tification of the unknown areas that interact to perform terion could be ranked lower by another criterion that,
a perceptual, cognitive, or motor task. for example, weights model fit more and complexity
Part two of the missing model problem is more subtle, less. Moreover, it is common to perform model com-
and states that many existing realistic models contain parison between dynamical model parameters within a
too many interrelated parameters to estimate from the confined structural model (cf. the missing region
available data records. This stipulates the interaction problem). This means complexity is balanced with
between models and data in model identifiability. A model fit in a situation where the above choices have
model is identifiable from a given data record if there already been made, since the data have already been
is a unique setting of its parameter values that fits the recorded and structural and dynamical models have
data best. If there is a number (or even an infinity) of been preselected. The missing model problem is about
possible parameter settings that all fit the data equally general choices to be made before this phase.
modality while using the same underlying neuronal results. In F. Palumbo, C. N. Lauro, & M. J. Greenacre
model (Deneux & Faugeras, 2010; Riera, Jimenez, Wan, (Eds.), Studies in classification, data analysis, and knowledge
organization: Data analysis and classification (pp. 435–443).
Kawashima, & Ozaki, 2007; Valdes-Sosa et al., 2009).
Berlin: Springer.
This approach holds great potential to fruitfully combine Attwell, D., Buchan, A. M., Charpak, S., Lauritzen, M.,
the superior spatial resolution of fMRI with the superior Macvicar, B. A., & Newman, E. A. (2010). Glial and neu-
temporal resolution of EEG. In Valdes-Sosa et al. (2009), ronal control of brain blood flow. Nature, 468, 232–243.
structural connectivity information obtained from dMRI Attwell, D., & Iadecola, C. (2002). The neural basis of func-
is also incorporated. We look in anticipation to what tional brain imaging signals. Trends Neurosci, 25, 621–625.
Barrett, A. B., Barnett, L., & Seth, A. K. (2010). Multivari-
high spatial and temporal resolution fMRI, perhaps ate Granger causality and generalized variance. Phys Rev E
in conjunction with other neuroimaging modalities, Stat Nonlin Soft Matter Phys, 81, 041907.
can bring for causal computational modeling. If the Bastos, A. M., Usrey, W. M., Adams, R. A., Mangun, G. R.,
missing model problem is sufficiently considered in this Fries, P., & Friston, K. J. (2012). Canonical microcircuits
endeavor, we expect human cognitive neuroscience for predictive coding. Neuron, 76, 695–711.
Bernasconi, C., & Konig, P. (1999). On the directionality of
stands to gain very important research tools to answer cortical interactions studied by structural analysis of elec-
new questions that are currently out of reach. trophysiological recordings. Biol Cybern, 81, 199–210.
Bressler, S. L., Tang, W., Sylvester, C. M., Shulman, G. L.,
& Corbetta, M. (2008). Top-down control of human visual
REFERENCES
cortex by frontal and parietal cortex in anticipatory visual
Aalen, O. O. (1987). Dynamic modeling and causality. Scand spatial attention. J Neurosci, 28, 10056–10061.
Actuar J, 1987(3–4), 177–190. Breuer, F. A., Blaimer, M., Mueller, M. F., Seiberlich, N.,
Aalen, O. O., & Frigessi, A. (2007). What can statistics con- Heidemann, R. M., Griswold, M. A., & Jakob, P. M. (2006).
tribute to a causal understanding? Scand J Stat, 34, Controlled aliasing in volumetric parallel imaging (2D
155–168. CAIPIRINHA). Magn Reson Med, 55, 549–556.
Aguirre, G. K., Zarahn, E., & D’Esposito, M. (1998). The Brovelli, A., Ding, M., Ledberg, A., Chen, Y., Nakamura,
variability of human, BOLD hemodynamic responses. Neu- R., & Bressler, S. L. (2004). Beta oscillations in a large-
roImage, 8, 360–369. scale sensorimotor cortical network: Directional influences
Akaike, H. (1968). On the use of a linear model for the iden- revealed by Granger causality. Proc Natl Acad Sci USA, 101,
tification of feedback systems. Ann Inst Stat Math, 20, 9849–9854.
425–439. Buxton, R. B., Uludag, K., Dubowitz, D. J., & Liu, T. T.
Amendola, A., Niglio, M., & Vitale, C. (2010). Temporal (2004). Modeling the hemodynamic response to brain acti-
aggregation and closure of VARMA models: Some new vation. NeuroImage, 23(Suppl. 1), S220–S233.
ABSTRACT The central goal of cognitive neuroscience is to of MVPA methods over individual voxel–based methods
understand how information is processed in the brain. To is increased sensitivity for detecting this information.
accomplish this goal, researchers studying human cognition Conventional fMRI analysis methods try to find voxels
are increasingly relying on multivoxel pattern analysis
(MVPA); this method involves analyzing spatially distributed that show a statistically significant response to the
(multivoxel) patterns of functional MRI activity, with the goal experimental conditions. To increase sensitivity to a
of decoding the information that is represented across the particular condition, these methods spatially average
ensemble of voxels. In this chapter, we describe the major across voxels that respond significantly to that condi-
subtypes of MVPA, we provide examples of how MVPA has tion. Although this approach reduces noise, it also
been used to study neural information processing, and we
highlight recent technical advances in MVPA.
reduces signal in two important ways: first, voxels with
weaker (i.e., nonsignificant) responses to a particular
Cognitive neuroscience theories deal with information condition might carry some information about the pres-
processing: what information is represented in differ- ence or absence of that condition. Second, spatial aver-
ent brain structures, how is this information trans- aging blurs out fine-grained spatial patterns that might
formed over time, and how is it used to guide behavior? discriminate between experimental conditions.
Functional MRI (fMRI) constitutes a powerful tool for Like conventional methods, the MVPA approach also
addressing these questions: while a subject performs a seeks to boost sensitivity by looking at the contributions
cognitive task, we can obtain estimates of local blood of multiple voxels. However, to avoid the signal-loss
flow (a proxy for local neural processing) from tens of issues mentioned above, MVPA does not routinely
thousands of distinct neuroanatomical locations (voxels, involve uniform spatial averaging of voxel responses.
or volumetric pixels) within a matter of seconds. Instead, the MVPA approach uses a weighted average
Traditional univariate fMRI analysis methods have of responses, treating each voxel as a distinct source of
focused on characterizing how cognitive variables mod- information about the participant’s cognitive state. The
ulate the activity of individual brain voxels (or small technique finds ways to optimize these weights, and
clusters of voxels; e.g., Gonsalves & Cohen, 2010). The then aggregates this (possibly weak) information across
goal of this chapter is to describe a different approach voxels to derive a more precise sense of what the par-
to fMRI analysis that focuses on extracting information ticipant is thinking. The multivoxel response can be
about a person’s cognitive state (i.e., a snapshot of the thought of as a combinatorial code for representing
person’s current internal thought space) from spatially distinctions between cognitive states (see figure 77.1).
distributed, multivoxel patterns of fMRI activity. This Because MVPA analyses focus on high spatial-frequency
approach is referred to as multivoxel pattern analysis (and often idiosyncratic) patterns of response, they are
(MVPA; Haxby et al., 2001; Haynes & Rees, 2006; typically conducted within individual subjects, although
Kamitani & Tong, 2005; Norman, Polyn, Detre, & recent advances in data alignment procedures have
Haxby, 2006; Pereira, Mitchell, & Botvinick, 2009). paved the way for the expansion of classification analy-
Over the past decade, this approach has become ubiq- ses beyond the individual subject (Haxby et al., 2011;
uitous in fMRI research, and its adoption has led to see the section “New developments,” below).
novel discoveries about the brain bases of perception, Broadly speaking, the term MVPA has come to
attention, imagery and working memory, episodic encompass two distinct methods. The first method
memory, semantic knowledge, language processing, involves using pattern classification methods imported
and decision making (see Rissman & Wagner, 2012; from machine learning to learn a mapping between
Tong & Pratte, 2012). multivoxel brain states and cognitive state information.
Given the goal of detecting the presence of a particu- This approach flips standard univariate fMRI analysis
lar cognitive state in the brain, the primary advantage on its head: standard voxel-based analysis uses multiple
lewis-peacock and norman: multivoxel pattern analysis of functional mri data 911
Voxels blood oxygen level–dependent, or BOLD, data, includ-
v1 v2 v3 Avg ing temporal and spatial realignment, noise filtering,
and z-scoring of the data (over time, within each voxel)
A within each run. Next, feature selection chooses which
Cognitive voxels will be included in the analysis. All voxels in the
states B
brain can be used, but it is often advantageous to limit
C the analysis to certain voxels. One way to select features
is to limit the analysis to specific anatomical regions
(e.g., Haxby et al., 2001, focused on ventral temporal
Single voxel
{A,C} A {A,B} None cortex in their study of visual object processing). Uni-
vs. vs. vs. variate statistics used in conventional fMRI analysis
distinctions B {B,C} C
(e.g., Mitchell et al., 2004) and newer multivariate
}
“wrapper methods” (Guyon & Elisseeff, 2003) can also
Multivoxel
A vs. B vs. C be used for feature selection (e.g., one can discard the
distinctions
voxels that—taken on their own—do the worst job of
Figure 77.1 Observing the multivoxel response pattern discriminating between conditions). Finally, pattern
allows us to distinguish all three cognitive states, A, B, and C. assembly involves sorting the data into discrete “brain
Considering each voxel in isolation provides only partial
discrimination. patterns” corresponding to the pattern of activity across
the selected voxels at a particular time in the experi-
ment. Patterns can be assembled using the prepro-
regression to predict the activity of individual voxels cessed fMRI signal for each trial or, alternatively, by
based on the participant’s cognitive state. By contrast, using multiple regression to estimate the unique neural
classification-based MVPA uses multiple regression to response in each voxel for each trial (Mumford, Turner,
predict the participant’s cognitive state based on the Ashby, & Poldrack, 2012). Brain patterns are labeled
activity of multiple voxels. The second major subtype of according to which cognitive state (or experiment con-
MVPA does not use pattern classifiers; rather, it exam- dition, stimulus, response, etc.) generated the pattern;
ines the similarity structure of multivoxel patterns (i.e., this labeling procedure needs to account for the fact
which patterns are similar to one another) and uses this that the hemodynamic response measured by the
similarity structure information to draw conclusions scanner is delayed and smeared out in time, relative to
about what information is reflected in these patterns. the instigating neural event. Once the patterns have
In the section “Mechanics of multivoxel pattern anal- been assembled, MVPA can proceed along two main
ysis” below, we provide an overview of these two sub- branches of analysis: classifier-based MVPA and pattern-
types of MVPA. In the section “Applications of multivoxel similarity MVPA. We will now discuss both methods in
pattern analysis,” we illustrate how MVPA has been used turn.
to study information representation and processing in
the brain. Finally, in the section “New developments,” Classifier-Based Multivoxel Pattern Analysis
we discuss recent advances in MVPA that allow for finer- There are two steps to classifier-based MVPA. The first
grained mappings between brain activity and cognitive step, classifier training, involves feeding a subset of labeled
states within individuals, and also new methods for patterns into a multivariate pattern-classification algo-
aligning and combining brain data across individuals. rithm. Based on these patterns, the classification algo-
Importantly, while this chapter is focused on fMRI, rithm learns a function that maps between voxel activity
we should emphasize that most of the MVPA methods patterns and cognitive states. As illustrated in figure
described here can be applied to other imaging modali- 77.2, brain patterns can be viewed as points in a multi-
ties as well (for applications to electroencephalography dimensional voxel space; the goal of the classifier is
and magnetoencephalography data, see, e.g., Jafarpour, to find a decision boundary in this space that best
Horner, Fuentemilla, Penny, & Duzel, 2013; for applica- separates the patterns associated with the to-be-
tions to direct neural recording data, see, e.g., Hung, discriminated cognitive states. The second step is gen-
Kreiman, Poggio, & DiCarlo, 2005). eralization testing: given a new pattern of brain activity
(not previously presented to the classifier), can the
Mechanics of multivoxel pattern analysis trained classifier correctly determine the cognitive state
associated with that pattern?
Here, we will review the basic procedures of MVPA. All The most commonly used classifiers are linear classi-
pattern analyses start with preprocessing of the raw fMRI fiers, which derive a linear decision boundary between
faces scenes
Stimulus
categories
Multivoxel
patterns of
brain activity
n voxels
scene? C
house
5
faces
µ1= [−2 1]
B
silos
Activity in voxel 2
Similarity matrices
Neural Other
scenes A B C A B C
µ2= [2 – 1]
A A
–5
–10 0 10 B B
Activity in voxel 1 C C
Figure 77.2 The two main types of multivoxel pattern analysis of functional MRI data. (See color plate 72.)
classes. At training, the classifier learns a weight for between these algorithms relates to which features of the
each voxel, plus an intercept term; collectively, these data they use when modeling the data. The multidimen-
weights determine the equation of the hyperplane (in sional clouds of data for each class can be characterized
two dimensions, a line) that forms the decision bound- in terms of their mean value and also their covariance
ary. At test, the classifier uses these weights (and inter- matrix. This matrix specifies the spread of the cloud
cept) to compute a weighted sum of voxel activity along each voxel dimension (i.e., how tall or wide the
values, and it uses this weighted sum to determine cloud is) and also the covariance between each pair of
whether the test pattern falls on one side or the other dimensions (i.e., the tilt of the cloud). The simplest
of the decision boundary. There is a wide range of classifier is the minimum distance classifier (e.g., Haxby
linear classification algorithms; the main difference et al., 2001), which estimates the mean value for each
lewis-peacock and norman: multivoxel pattern analysis of functional mri data 913
class based on the training data and then classifies new of parameters to estimate from n2 to n. Support vector
points based on their proximity to these means. machines (SVMs; Cox & Savoy, 2003) achieve complex-
However, figure 77.2 demonstrates how ignoring the ity control by defining the category boundary in terms
covariance matrix can produce nonoptimal decision of a small number of support vectors (i.e., training
boundaries. More complex linear classifiers (e.g., Fish- exemplars close to the decision boundary). Another
er’s linear discriminant; Duda, Hart, & Stork, 2012) can way to limit the number of free parameters is to limit
converge on the optimal decision boundary by creating the number of voxels used for classification (e.g.,
a more sophisticated model of the data: in addition to restricting classification of oriented gratings to low-level
estimating the mean, they also model the class-condi- visual cortex; Kamitani & Tong, 2005; or restricting clas-
tional probability densities (i.e., they estimate the full sification of faces and scenes to ventral temporal cortex;
covariance matrix within and between voxel dimen- Kuhl, Rissman, Chun, & Wagner, 2011). This is a useful
sions for each class). Figure 77.2 shows how the bound- approach when there is a priori knowledge of strong
ary learned by a linear-discriminant classifier factors in selectivity for the classes in particular brain regions.
the “tilt of the ellipse” for each data cloud. Nonlinear An effective way to reduce the complexity of linear
classifiers (e.g., k nearest neighbor, multilayer neural classifiers is to add a regularization parameter to the
networks) can form even more complex decision model that punishes undesirable properties of the solu-
boundaries. tion (e.g., large weights on individual voxels). Common
Classification of fMRI data is a challenging problem, forms of regularization are L2 regularization, which
for several reasons: first, the number of data points penalizes the sum of squares of the voxel weights, and
(brain patterns) that are available for training tends to L1 regularization, which penalizes the absolute value of
be small relative to the number of parameters in the the weights. As the regularization parameter is increased,
model. For example, Polyn, Natu, Cohen, and Norman L2 regularization pulls in extreme voxel weights (result-
(2005) trained a classifier on 450 total brain patterns ing in a smoother distribution of weights), whereas L1
(150 for each of three stimulus classes) per participant, regularization causes some weights to be driven to zero
where each brain pattern consisted of approximately (a sparser solution). In both cases, the regularization
7,000 voxels. In this situation, the covariance matrix has parameter limits the space of possible solutions, thereby
millions of unique entries (corresponding to all of the reducing the flexibility of the classifier and reducing
voxels, plus all of the unique pairings of voxels); each overfitting.
of these entries is a parameter that needs to be esti- Practically speaking, all of the above forms of com-
mated based on only a few hundred training patterns. plexity control (GNBs, SVMs, voxel reduction, and
Further adding to the complexity of this problem, the regularization) have been shown to improve generaliza-
brain patterns are very noisy (i.e., the clouds are highly tion performance, relative to linear classifiers that do
dispersed). In this kind of situation, where the data are not incorporate complexity control. The only excep-
noisy and the number of parameters being estimated tion is when the number of voxels is very small or the
by the classifier dwarfs the number of training patterns, number of training patterns is very large, at which point
classifiers are prone to overfitting the noise in the train- it becomes feasible to estimate the full covariance
ing data: that is, the classifier may learn idiosyncratic matrix. Importantly, with fMRI data, nonlinear classifi-
features of the training examples rather than the actual ers virtually never outperform linear classifiers on gen-
distinction between the classes, thereby leading to poor eralization tests—the added flexibility of these classifiers
generalization. leads to overfitting.
Overfitting is the main obstacle to achieving good
fMRI classification. One way to combat overfitting is to Pattern-Similarity Multivoxel Pattern Analy-
collect more data, but there are practical limits on col- sis The second major form of MVPA is pattern-simi-
lecting more data per participant. In the final section larity analysis (e.g., Kriegeskorte, Mur, & Bandettini,
of this chapter, we will discuss new developments in 2008). Here, brain patterns are viewed as points in high-
MVPA that allow us to obtain more data by combining dimensional voxel space, where the distance between
across subjects. The other way to combat overfitting is points indicates the similarity of the patterns. Rather
to try to limit the complexity of the classifier. For than specifying which features of the data to separate
example, Gaussian naive Bayes classifiers (GNB; Pereira with a classifier, pattern-similarity analysis summarizes
et al., 2009) simplify the modeling of the covariance the space using a matrix that records the distance
matrix by treating the n dimensions of the data as inde- between each pair of points. This matrix can be viewed
pendent (such that the off-diagonal elements of the as a neural “fingerprint” of the representational space.
covariance matrix are zero), thus reducing the number Although information about the exact positions of the
Classifier-Based Thought Tracking The goal of Cautionary notes for thought-tracking studies The ideal
the classifier-based thought tracking approach is to situation for thought-tracking is to get independent
measure participants’ thoughts on a trial-by-trial basis, readouts of the relevant cognitive states, but achieving
to characterize the dynamics of these thoughts, and to this goal can be difficult. Classifiers are opportunistic:
assess how they relate to behavior. This approach is used if two categories are anticorrelated in the training set
when the main concern is tracking a particular latent (e.g., all training patterns are either faces or scenes,
cognitive state, and there is relatively less concern about never both), the classifier will learn this negative cor-
how that cognitive state is represented in the brain relation, and it will come to treat the lack of scene
(although this approach can be applied to specific activity as strong evidence for the presence of faces (see
regions of interest to localize cognitive representations). Kuhl et al., 2011, for discussion of this issue). Training
Compared to univariate methods, MVPA squeezes on additional categories alleviates this problem by
more information about the participant’s cognitive reducing the size of the negative correlation between
state out of each snapshot of fMRI data, thereby increas- categories at study (e.g., if there are faces, scenes, and
ing the effective temporal resolution of fMRI analysis objects, then the absence of faces does not perfectly
and making it possible to record trajectories of cogni- predict the presence of scenes).
tive states over time. However, even with the added
sensitivity of MVPA, not all cognitive states are equally Classifier-Based Information Mapping A second
“visible” to fMRI. In this situation, researchers often application of MVPA is less concerned with getting a
find it useful to take the cognitive state of interest and useful readout of information processing during indi-
link it to something that we know is highly visible with vidual trials, and more concerned with assessing whether
fMRI: stimulus category information (e.g., faces and a particular fine-grained distinction is represented in a
lewis-peacock and norman: multivoxel pattern analysis of functional mri data 915
particular brain region (e.g., Pereira & Botvinick, 2011). similarity of mental concepts based on the similarity of
This analysis is similar in concept to the mass-univariate patterns of brain activity elicited by those concepts. The
approach, in that the goal is to determine which brain strength and versatility of this approach comes from the
regions are responsive to a particular cognitive process. many different ways that similarity matrices can be com-
However, rather than considering how the activity in puted and thus permits many types of comparisons. For
each individual voxel is predicted by a person’s (pre- example, Kriegeskorte, Mur, Ruff, et al. (2008) showed
sumed) cognitive state, classifier-based information that the similarity structure of neural patterns in human
mapping uses information from multiple voxels simul- IT cortex (measured using fMRI) resembles the similar-
taneously to predict the person’s cognitive state. ity structure of neural patterns in monkey IT cortex
This analysis can be done using many different a (measured using electrophysiology). Furthermore, they
priori regions of interest, or it can be done using the showed that the similarity structure of neural patterns
searchlight method (e.g., Kriegeskorte, Goebel, & Ban- in both human and monkey IT (specifically, clustering
dettini, 2006). This method consists of constructing a into animate vs. inanimate objects) could not be
“searchlight” of voxels and sliding this searchlight all explained purely in terms of the low-level visual features
around the three-dimensional brain volume. For each of the stimuli.
placement of the searchlight, you consider the multi- Pattern-similarity analysis is frequently used to study
voxel pattern of activity within that searchlight. A clas- the representation of item-specific information—the
sifier is trained on these patterns and then used to logic here is that regions that differentiate items within
assess how informative these patterns are about the a category should show greater pattern similarity
cognitive states of interest. between two instances of the same item, compared to
This approach has been used to discover new insights two distinct items from the same category. For example,
into cognition and the localization of function in the Ritchey, Wing, Labar, and Cabeza (2012) found evi-
brain. For example, Soon, Brass, Heinze, and Haynes dence that encoding-retrieval similarity at the individ-
(2008) used the searchlight technique to discover brain ual item level predicted memory success. Similarity
regions whose activity patterns were predictive of future between an item’s neural representation and the neural
decisions. They found that the outcome of a simple representations of other studied items has also been
decision (to press a left or right button) could be used to predict memory performance. For example,
decoded from prefrontal and parietal cortices up to 10 LaRocque et al. (2013) found that greater levels of
seconds prior to this decision entering awareness. across-item pattern similarity in perirhinal and parahip-
pocampal cortices were associated with better recogni-
Cautionary notes for classification-based information tion memory performance.
mapping An important caveat for the information-
mapping approach is that above-chance decoding— Cautionary notes for pattern-similarity analysis As
which signals that a brain region contains information described above, classifiers compute weighted combina-
about a particular cognitive distinction—does not nec- tions of features that discriminate between classes;
essarily imply that this region is involved in guiding uninformative or noisy features may be effectively “fil-
behavior based on that distinction (e.g., Williams, Dang, tered out” by being assigned small weight values. In
& Kanwisher, 2007). Furthermore, information mapping contrast, pattern-similarity analyses do not compute
is opportunistic and may produce false-positive results; weights for each voxel—these analyses treat all voxels
see Todd, Nystrom, and Cohen (2013) for discussion of as equally important. For this reason, pattern-similarity
how MVPA can be more susceptible than univariate analyses are more susceptible to contamination from
analysis to experimental confounds (e.g., task diffi- uninformative or noisy features than classifiers. Another
culty). Finally, multivariate decoding is not necessarily concern is that pattern-similarity results can be influ-
more sensitive than univariate decoding (Jimura & Pol- enced by univariate effects. For example, imagine that
drack, 2012). If the underlying signal has a coarse a 33-voxel searchlight contains a 10-voxel subregion
spatial scale, then univariate approaches using spatial that tracks memory strength, such that all 10 of these
smoothing at this scale will outperform MVPA. In this voxels activate together for remembered (but not for-
case, the extra parameters used to model the data in gotten) items. This will increase the average pattern
MVPA can lead to overfitting (Kriegeskorte et al., 2006). similarity between remembered items. Naively, one
might interpret this effect in terms of neural represen-
Pattern Similarity Analysis The goal of pattern- tations “converging” in representational space, when
similarity analysis of fMRI data (e.g., Kriegeskorte, Mur, (in fact) it is merely due to a univariate effect being
& Bandettini, 2008) is to make inferences about the superimposed on the searchlight region.
lewis-peacock and norman: multivoxel pattern analysis of functional mri data 917
of anatomical alignment alone leading to good across- accomplish (Davis & Poldrack, 2013). Going forward, it
subject classification. There are two likely reasons for will be beneficial to compare MVPA to other measure-
this: first, the transformations result in spatial blurring, ment and analysis techniques to get a better sense of
which might erase high-spatial-frequency information which aspects of neural processing we can and cannot
in the data that would otherwise be useful for the clas- detect with MVPA. Comparisons of MVPA with univari-
sifier. In addition, people have different experiential ate analysis (Jimura & Poldrack, 2012) and fMRI adap-
histories that shape how concepts are represented in tation (Epstein & Morgan, 2012) suggest that these
their brains—no amount of anatomical alignment will methods are interrogating different aspects of the
correct for such differences. neural code. It will also be useful to compare MVPA to
To address these issues, Haxby and colleagues have neurophysiology data from human and nonhuman pri-
developed a new across-subject alignment procedure mates (e.g., Kriegeskorte, Mur, Ruff, et al., 2008) to
called hyperalignment (Haxby et al., 2011) that aligns better understand the strengths and limitations of this
brains not based on anatomical landmarks but rather powerful, but relatively recent, advance in the cognitive
based on the functioning of those brains (i.e., aligning neuroscience toolkit.
parts of the brain that behave similarly, regardless of
“where” exactly in the brain these parts came from).
The basic hyperalignment algorithm performs a Pro-
crustes transformation that rotates, scales, and shifts REFERENCES
temporal trajectories of voxels to best align data sets
Cox, D. D., & Savoy, R. L. (2003). Functional magnetic reso-
from different brains. Haxby and colleagues hyper- nance imaging (fMRI) “brain reading”: Detecting and clas-
aligned a data set of 21 participants viewing the movie sifying distributed patterns of fMRI activity in human visual
Raiders of the Lost Ark and performed across-subject cortex. NeuroImage, 19(2), 261–270.
classification for movie segments, faces and objects, Davis, T., & Poldrack, R. A. (2013). Measuring neural rep-
resentations with fMRI: Practices and pitfalls. Ann NY Acad
and animal species. They found that hyperalignment
Sci, 1296, 108–134.
produced far superior classification performance com- Detre, G. J., Natarajan, A., Gershman, S. J., & Norman,
pared to alignment based purely on anatomy; hyper- K. A. (2013). Moderate levels of activation lead to forgetting
alignment even matched the accuracy of within-subject in the think/no-think paradigm. Neuropsychologia, 51(12),
classification. Good classification indicates good align- 2371–2388.
ment of patterns across participants, which suggests Duda, R. O., Hart, P. E., & Stork, D. G. (2012). Pattern clas-
sification. New York, NY: Wiley.
that adding more participants to the training set should Epstein, R. A., & Morgan, L. K. (2012). Neural responses to
help generalization even further. Therefore, hyper- visual scenes reveals inconsistencies between fMRI adapta-
alignment is an extremely promising approach to mini- tion and multivoxel pattern analysis. Neuropsychologia, 50(4),
mizing the “data starvation” problem of MVPA. 530–543.
Gonsalves, B. D., & Cohen, N. J. (2010). Brain imaging,
cognitive processes, and brain networks. Perspect Psychol Sci,
Conclusion 5(6), 744–752.
Guyon, I., & Elisseeff, A. (2003). An introduction to variable
Multivoxel pattern analysis allows us to detect informa- and feature selection. J Mach Learn Res, 3, 1157–1182.
tion in the brain that was not visible using previously Haxby, J. V., Gobbini, M. I., Furey, M. L., Ishai, A., Schouten,
developed methods of fMRI analysis. Getting a better J. L., & Pietrini, P. (2001). Distributed and overlapping
representations of faces and objects in ventral temporal
handle on the informational contents of a person’s
cortex. Science, 293(5539), 2425–2430.
brain puts researchers in a better position to test theo- Haxby, J. V., Guntupalli, J. S., Connolly, A. C., Halchenko,
ries of how information processing works in the brain Y. O., Conroy, B. R., Gobbini, M. I., & Ramadge, P. J.
and how cognitive states shape behavior. (2011). A common, high-dimensional model of the repre-
MATLAB software for performing a classifier analysis sentational space in human ventral temporal cortex. Neuron,
72(2), 404–416.
(the Princeton Multi-Voxel Pattern Analysis toolbox)
Haynes, J. D., & Rees, G. (2006). Decoding mental states
can be found at www.pni.princeton.edu/mvpa. Alterna- from brain activity in humans. Nat Rev Neurosci, 7(7),
tively, there is a separately developed Python version 523–534.
of the toolbox (PyMVPA) available at http://neuro Hung, C. P., Kreiman, G., Poggio, T., & DiCarlo, J. J. (2005).
.debian.net. MATLAB software for performing pattern- Fast readout of object identity from macaque inferior tem-
similarity MVPA can be found at www.mrc-cbu.cam poral cortex. Science, 310(5749), 863–866.
Jafarpour, A., Horner, A. J., Fuentemilla, L., Penny, W. D.,
.ac.uk/methods-and-resources/toolboxes. & Duzel, E. (2013). Decoding oscillatory representations
Although MVPA offers advantages over other forms and mechanisms in memory. Neuropsychologia, 51(4),
of analysis, there are limitations to what it can 772–780.
lewis-peacock and norman: multivoxel pattern analysis of functional mri data 919
78 Quantitative Magnetic Resonance
Imaging
YANIV ASSAF
ABSTRACT Quantitative measures of biological tissues such as We can divide MRI contrast into three categories:
measuring relaxation times (T1, T2, and T1*), magnetization
transfer, diffusion, flow, and exchange have been studied for 1. contrast that originates from an MRI physical
almost 30 years. But only recently the acquisition of these parameter
measurements became fast enough, and the analytical 2. contrast that represents a biophysical property of
methods robust enough, to enable large population studies. the tissue
With the great investment in technology development of
these methods, parameters such as the myelin water fraction 3. contrast that is induced by external chemicals that
(Deoni, Rutt, Arun, et al., 2008) and the axon diameter are transported into the tissue (contrast agent).
(Assaf, Blumenfeld-Katzir, Yovel, & Basser, 2008; Alexander et
al., 2010) distribution can now be estimated in vivo and for The acquisition of each contrast necessitates the use of
the whole brain. This is a quantum leap in the ability of MRI different pulse programs, and as a result each MRI
to dive into the microstructure of the brain and provide the experiment should be planned to achieve the suitable
researcher with invaluable parameters that will provide a contrast to the research. Thus, the researcher faces a
more comprehensive characterization of the tissue. difficult problem when performing an MRI experi-
ment: what is the best contrast mechanism to investi-
With these new approaches to studying the brain, the gate the phenomenon of interest? To answer this
link between structure and function can be revisited. A question, one should understand the origins of each
full description of quantitative magnetic resonance contrast mechanism.
imaging (MRI) methods requires a whole book (e.g., MRI is not a specific modality in the sense that the
Tofts, 2003); in this chapter, we will focus on two vast majority of MRI experiments measure water, which
methods that are being intensively explored and used, is not specific to a molecule, an organ, a tissue, or any
T1 mapping and diffusion MRI. Both methods provide cellular type (Hashemi et al., 2004; Stark & Bradley,
an unprecedented view of the brain, especially the 1999). Therefore, increasing the specificity of MRI is
white matter, allowing exploration of the microstruc- usually achieved by manipulating the MRI signal (by
ture, in vivo, at a scale that could have been achieved changing the pulse sequence) so that it will be more
thus far only by microscopes. sensitive and specific to certain parts of the tissue
(Hashemi et al., 2004).
What is quantitative magnetic resonance imaging? The most basic MRI sequences provide contrast that
is sensitive to the different relaxation times of MRI: T1,
Contrast Mechanisms of Magnetic Resonance T2, and T2*. T1 contrast provides excellent distinction
MRI is a powerful imaging modality for several reasons: among the three tissue types in the brain (gray matter,
it is performed in vivo, noninvasively, and is highly sensi- white matter, and cerebrospinal fluid); T2 is highly
tive to tissue composition. MRI is also a very flexible sensitive to the water content of the tissue; T2* is influ-
modality, permitting acquisition of a multitude of dif- enced by the magnetic susceptibility of the tissue. More
ferent contrasts, each sensitive to a different aspect of advanced techniques can extract information that is
the tissue (Hashemi, Bradley, & Lisanti, 2004; Stark & specific to biophysical parameters of the tissue. In
Bradley, 1999). Either representing a MRI physical this category, information about diffusion, flow, macro-
parameter (e.g., relaxation) or a biophysical property molecular content, and chemical exchange can be
of the tissue (e.g., diffusion), MRI has become central extracted. Additional approaches combining several
in neuroscience, enabling scientists to explore the brain contrast mechanisms and mathematical models allow
in many ways. us to relate the MRI signal into specific tissue
and are generally used for segmentation and parcella- both processes lead to similar consequences (i.e.,
tion of the brain (figure 78.1). dephasing), the latter can be experimentally separated
The initial RF excitation causes a coherent precision and is called the true spin-spin interaction relaxation,
of the spins. This coherence declines over time, and T2 T2. This is the basic contrast mechanism used for
refers to the time constant of the coherence loss diagnosis purposes since spin-spin interactions are
process (also termed dephasing). As indicated by its extremely sensitive for water content, which regularly
additional names, spin-spin or transverse relaxation, T2 changes in pathology or following insult to the tissue
mechanism differs from T1, and the T2 value depends (Stark & Bradley, 1999).The combination of both spin
on other factors. The loss of spin coherence occurs coherence loss processes is called T2*. This is the con-
due to interactions of spins with other spin magnetiza- trast mechanism used to measure the blood oxygen-
tion (either of water or other molecules). Conse- ation level that is at the heart of fMRI.
quently, T2 depends on the chemical content and The term MRI relaxometry refers to acquisition and
mobility of the spins. Here we differentiate between analysis methods that quantify the relaxation parame-
two sources of interactions: (1) spin coherence loss ters (mainly T1 and T2) for each image pixel in the
due to magnetic field inhomogeneities, typically caused brain (or any other part of the body).
by the chemical composition of the tissue (e.g., high
concentrations of iron ions); (2) spin coherence loss Inversion Recovery: The Measurement of T1 After
due to interaction with other hydrogen atom spins RF excitation, spins exchange energy with the lattice
(Hashemi et al., 2004; Stark & Bradley, 1999). While and return to their equilibrium state at an exponential
Figure 78.4 Representative sagittal slices (at approximately the same location) through the whole brain. 3D T1(A) and T2(B)
map volumes for each of six volunteers.
When barriers to the molecule’s motion exist (e.g., direction perpendicular to the fibers compared to the
cell membrane), the displacement will be affected (as parallel direction. The displacement of water molecules
the net displacement will be confined to the barrier in neural tissue is anisotropic—that is, not equal in all
geometry), while the diffusion coefficient itself remains directions. When considering the white matter, the
the same (in theory). Since in diffusion MRI we measure interpretation is straightforward—intuitively, water
the displacement rather than the diffusion, the exact molecules should experience more displacement paral-
diffusion coefficient (according to the Stejskal-Tanner lel to the long axis of the neuronal fibers (due to the
relation above) will be miscalculated. Therefore, in a high packing of the neuronal fibers and their myelin
biological system, the computed diffusion coefficient is lamellas) than perpendicular to them. Based on this
called the apparent diffusion coefficient (ADC) to indicate concept, it has been proposed that characterization of
that the measured value includes the effects of barriers white matter can be achieved by measuring the ADC
to motion. This is also called hindered diffusion (Le perpendicular and parallel to the fibers’ orientation
Bihan, 1995). and extracting an anisotropy index reflecting the differ-
The diffusion MRI measurement is directional. By ences between the two measures (e.g., D⊥/D//). Such
specifying the direction of the applied pulsed gradient, measurement can be performed on very specific fiber
the diffusion or displacement will be acquired along systems, where the orientation of the fibers is such that
that direction. In an aqueous environment, such as the it allows easy prescription of diffusion gradients in the
CSF, the signal loss is the same in all directions. But in parallel and perpendicular directions (e.g., the spinal
fibrous and cellular tissue, the signal loss is highly direc- cord, optic nerve, and corpus callosum). However, for
tional. In seminal works from the early 1990s, it was most fiber systems in the brain, defining the parallel
shown that the contrast and signal decay at various and perpendicular directions is impossible. Moreover,
brain regions depends dramatically on the orientation even if such information were available, due to the
of the applied diffusion gradient, especially in white heterogeneity in fiber orientations in the brain, each
matter (figure 78.6; Moseley et al., 1990). This is the system would have to be acquired separately.
basis for the diffusion tensor imaging (DTI) method. In the mid-1990s, the concept of DTI was proposed
(Basser, 1995; Basser, Mattiello, & Le Bihan, 1992, 1994;
Diffusion Tensor Imaging As indicated in figure Basser & Pierpaoli, 1998; Pierpaoli & Basser, 1996; Pier-
78.6, the displacement and ADC in neural tissue, espe- paoli et al., 1996). In DTI, the whole brain is sampled
cially in white matter, may differ when measured in with diffusion imaging in multiple directions (the same
different directions. There is greater signal loss in the directions for all brain voxels, in at least six directions),
and for each direction the ADC is computed. The direc- The principal diffusivities (axial and radial diffu-
tional ADC maps are then analyzed using a diffusion sivities) can be used to compute various summation
tensor model. Mathematically, there are several ways to parameters. The most commonly used are the mean
apply the diffusion tensor model, yet in all of them the diffusivity (MD) and the fractional anisotropy (FA;
result is a three-dimensional ellipsoid that best describes Basser & Pierpaoli, 1998; Pierpaoli & Basser, 1996; Pier-
the multiple measures of the ADC in the different direc- paoli et al., 1996). The MD is the average of the three
tions. The DTI acquisition and analysis pipeline is sum- eigen values (three axes of the ellipsoid), while the FA
marized in figure 78.7. is the normalized variance between them. The formula-
Once the ellipsoid is calculated in each voxel, it is tions for both indices are
possible to define the principal directions that will cor-
respond to the diffusivity parallel to the fibers (largest MD = (λ1 + λ2 + λ3)/3
axis of the ellipsoid) and diffusivity perpendicular to 3 (λ1 − D )2 + (λ 2 − D )2 + (λ 3 − D )2
FA = ⋅ .
the fibers (mean of the two shortest axes of the ellip- 2 (λ12 + λ 22 + λ 23 )
soid). The extracted diffusivities are often called axial
diffusivity, D// or λ1 (largest eigenvalue), for the largest The size, shape, and direction of the ellipsoid char-
axis of the ellipsoid; and radial diffusivity, D⊥ or λ2/λ3 acterize the diffusion tensor and can be extracted for
(smallest eigenvalues), for the mean of the two shortest each voxel individually. The following figure summa-
axes of the ellipsoid (figure 78.7). rizes the different options of the ellipsoid varying in
size (figure 78.8A) indicating on the MD, varying in specific to this method (Jones & Cercignani, 2010). For
shape (figure 78.8B, rounded sphere → elongated example, as in any MRI method, and especially in dif-
ellipsoid) indicating on the FA, and varying in color fusion MRI when the signal is attenuated, DTI suffers
(figure 78.8C) indicating on the orientation. The latter from signal-to-noise limitations. Other acquisition-
code, where ellipsoid (fibers) crossing in the left-right related issues include gradient stability, eddy current
direction will be colored in red, ellipsoids that lie in the artifacts, gradient sampling schemes, and optimized b-
anterior-posterior direction will be colored in green, value, which have been studied in recent years.
and ellipsoids that are orientated from inferior to pos- In addition to acquisition artifacts, there are inher-
terior position will be colored in blue, is extremely ent flaws in the processing of DTI images that are
widely used and is termed the color-coded FA map manifested in errors in the computed DTI indices. The
(Pajevic & Pierpaoli, 1999). It should be noted that any most common problems arise in voxels that contain
arbitrary orientation of the ellipsoid can be achieved by more than a single tissue type, such as voxels at the
combinations of the three colors (red-green-blue). boundary between white matter and ventricles. Another
Figure 78.9 summarizes the most widely used parame- very problematic case arises in white matter regions
ters extracted from DTI analysis. The information in that contain several intersecting fiber populations.
the color-coded maps is enormous, since it provides In such areas, the DTI model fits only a single direc-
three-dimensional information into two-dimensional tion, and this may be a compromise direction that
maps, and the anatomy of the fiber systems becomes fails to represent the underlying architecture (see
apparent. figure 78.10). This has implication on FA and MD cal-
DTI provided unprecedented information about culation in those areas as well as on tractography (see
white matter, its physiology, and architecture. Even so, below).
there are artifacts and error in DTI’s acquisition (Jones
& Cercignani, 2010). Some of the artifacts in DTI are Imaging Microstructure DTI’s apparent inabil-
common to any MRI measurement, and some are ity, in some cases, to characterize the underlying
Figure 78.9 Summary of quantitative indices extracted from with section at the genu of the corpus callosum enlarge in
DTI. (A) Reference T1-weighted scan; (B) Axial diffusivity; (H). Color scale for (B–D) is given in diffusivity color scale
(C) Radial diffusivity; (D) mean diffusivity; (E) fractional (bottom left). Color scale for FA map is given in the FA
anisotropy; (F) color-coded FA map; (G) ellipsoid glyph map color scale. (See color plate 75.)
Figure 78.10 Example for partial volume artifact in DTI. ((C) enlarged at (F)) where one can follow the direction of
(A) T1-weighted anatomical image with the right frontal the fiber (red for the corpus callosum and green for the tha-
region enlarged at (D), including line drawings of the differ- lamic radiation). In the crossing fiber region, the observed
ent fibers system the passes in that region: one arriving from color is a mix of the two-fiber system. Since two crossing fibers
the genu of the corpus callosum and one from the thalamic reside within the same pixel in the region, the diffusion is
radiation of the internal capsule, and both projecting into the hindered in all measured directions, which artifactually can
frontal lobe. (B) FA maps enlarged at (E) showing low anisot- be characterized by isotropic diffusion and thus reduced FA.
ropy area (marked by yellow circle) in the area of the crossing (See color plate 76.)
fiber system. This is also shown in the color-coded FA maps,
architecture of white matter initiated directed research complex signal dependency, and it is now agreed that
to overcome this problem. Indeed, technical and com- DTI is a good first approximation of the diffusion pro-
putational solutions to some issues (e.g., crossing fibers) cesses in neural tissue, but certainly does not represent
were proposed (Assaf & Basser, 2005; Descoteaux, or characterize it completely. For more comprehensive
Deriche, Knosche, & Anwander, 2009; Parker & Alexan- characterization of the diffusion MRI signal, alternative
der, 2003; Tournier, Calamante, & Connelly, 2007). models to DTI had to be formulated.
However, the same studies indicated that DTI fails to Studying the origins and characteristics of the multi-
represent the underlying tissue even in homogenous exponential decay in diffusion experiments suggested
white matter, since DTI’s model (Gaussian tensor) is that diffusion in neural tissue is complicated and that
not appropriate for neural tissue (Assaf, Mayk, & Cohen, molecules at different subvoxel components (cellular
2000). The main indication for that was the measure- compartments) have different diffusion characteris-
ment of diffusion signal decay at a large range of tics. For example, it was found that one can measure
b-values. According to the Stejskal-Tanner equation two diffusion processes in neural tissue: hindered and
above, the relation between the signal decay and the restricted (Assaf, Freidlin, Rohde, & Basser, 2004).
diffusion coefficient is exponential, but it was shown While restricted diffusion refers to water that is con-
that this is not the case for neural tissue, and the depen- fined within specific environments, hindered diffusion
dency of the signal decay on the b-value seems to be implies water molecules move significantly but encoun-
composed of multiple exponents whose number cannot ter an obstacle to their motion every now and then.
be determined easily and thus cannot be assigned to Further research suggested that the origin of restricted
physiological compartments (Assaf & Cohen, 1998, diffusion stems from water molecules that are trapped
2000). DTI’s Gaussian model is not able to analyze such within axons, and that if one can design an acquisition
Figure 78.11 Cluster analysis of the axon diameter distribu- different clusters are shown in the middle row; note that the
tion along the corpus callosum: (top row, left). A midsaggital colors of the graphs match the clusters’ colors. In the bottom
T2-weighted MRI with the AxCaliber clusters superimposed, row are the histological ADDs of the same clusters. (See color
enlarged at right. The AxCaliber averaged ADDs for the plate 77.)
ABSTRACT Network science provides tools that can be used to between functions like memory, attention, and move-
understand the structure and function of the human brain in ment? How do we control the interactions between dif-
novel ways using simple concepts and mathematical represen- ferent neural circuits in our brains?
tations. Network neuroscience is a rapidly growing field with
implications for systems neuroscience, cognitive neurosci-
To answer these questions, we can use tools from
ence, and clinical medicine. In this chapter, we describe the systems neuroscience to perform (1) data analysis to
methodology of network science as applied to neuroimaging extract characteristic or predictive patterns in the data
data. We cover topics in constructing networks, probing and (2) forward modeling to build mathematical
network structure, generating network “diagnostics,” and models of the system from first principles. The distinc-
experimental design. We discuss several current frontiers and
tion here is important: data analytical approaches lead
the associated methodological challenges and considerations.
We aim to provide a practical introduction to the field: we to descriptions of an observed process; for example,
supplement the explanations and examples with pointers to brain areas A and B tend to be alternately activated
resources for students or researchers interested in using these during a visual task. In contrast, modeling involves the
methods to address their own questions in empirical and creation of a set of mathematical descriptions (i.e.,
theoretical neuroscience.
equations) that describe how components of the system
behave, given certain inputs or conditions. For example,
the activity in brain area A could be described by an
Why network neuroscience? equation that captures its behavior, including its depen-
dency on the activity of brain area B, and vice versa.
Each area of the human brain plays a unique role in Crucially, the mathematical descriptions in a model can
processing information gleaned from the external then be used to predict the behavior of the system given
world and in driving our responses to that external a different set of inputs, or in a different context. Natu-
world via behavior. Mapping these roles has led to enor- rally, models are much more difficult to create than
mous insights into the complex and varied contribu- descriptions.
tions of different brain regions to our mental function. In this chapter, we focus on newly developed tools for
However, the brain is far from a set of disconnected data analysis, which we refer to under the broad term
building blocks. Instead, at each moment throughout network neuroscience, that we envision will dramatically
the day, parts of the brain communicate with one inform the efforts in forward modeling in the coming
another in complex spatiotemporal patterns, like evolv- years. Network neuroscience provides a simple and
ing dance partners in a multifarious choreography, elegant systems approach to understanding how neural
which enable the formation of creative thoughts, the circuits function, how they constrain one another, and
acquisition of new skills, and the adaptation of human how they differ across individuals. A network represen-
behavior. Understanding this spatiotemporal complex- tation of a biological system (e.g., a genome, proteome,
ity requires a paradigmatic shift in our conceptual or connectome) treats individual components (e.g.,
approaches, empirical goals, and quantitative methods: genes, proteins, brain regions) as network nodes and
in short, in the way that we design and interpret our treats interactions between these components as
models and experiments. network edges. Network science, an interdisciplinary
Systems neuroscience addresses this complexity by approach spanning biology, economics, sociology, lin-
seeking to understand the structure and function of guistics, and computer science, provides a battery of
large-scale neural circuits and systems. How do indi- quantitative diagnostics that enable us to describe the
vidual brain areas interact with one another to enable architecture of this network in a statistically principled
cognitive function? How is cognition constrained by manner. One can then study these properties of
white matter pathways? How does the brain transition the network to gain insight into organizational
Connectivity strength
1
Brain regions
20 0.8
40
(Sagittal)
DSI/DTI
0.6
60
0.4
80
0.2
100 No clustering Clustering
0
20 40 60 80 100
Brain regions C.2.
Network
(Axial)
fMRI
C.3.
Embedded network
EEG/MEG
Figure 79.1 From data to diagnostics: The stages of a added to close the connected triples (i.e., three nodes con-
network study. (A) Data acquisition. Neuroimaging data can nected by two edges; green) to form triangles (i.e., three
capture structural connectivity (e.g., diffusion spectrum nodes connected by three edges; brown), thereby leading to
imaging, DSI; diffusion tensor imaging, DTI) or functional higher clustering. (C.2) The average shortest path length is a
connectivity (e.g., functional magnetic resonance imaging, diagnostic of global network structure. The left panel contains
fMRI; electroencephalography, EEG; or magnetoencephalog- a network with a relatively long average path length. For
raphy, MEG). (B) Representations of a network. Top: A con- example, to move from the purple node (top left) to the
nectivity matrix in which matrix elements (or pixels in the orange node (bottom right) requires one to traverse at least
grid) represent one connection between two brain regions, four edges. The right panel contains a network in which addi-
and the color indicates the strength of that connection. Center: tion edges have been added (green) to form triangles or to
A topographical network visualization in which brain regions link distant nodes (peach), thereby leading to a shorter
that are strongly (weakly) connected to one another lie close average path length by comparison. (C.3) Mesoscale network
to (far from) each other in the plane. Bottom: An embedded structure can take many forms. The left panel contains a
network visualization in which nodes are placed in anatomi- network with a core of densely connected nodes (red circles;
cally accurate locations. (C) Network diagnostics. (C.1) The green edges) and a periphery of sparsely connected nodes
clustering coefficient is a diagnostic of local network struc- (blue circles; gray edges). The right panel contains a network
ture. The left panel contains a network with zero connected with four densely connected modules (red circles; green
triangles and therefore no clustering, while the right panel edges) and a connector hub (blue circle; gray edges) that
contains a network in which additional edges have been links these modules to one another. (See color plate 78.)
Weighted
network
Density: 0.1 0.3 0.5 0.7 0.9
10% strongest 90% strongest
edges retained edges retained
Windowed thresholding
Figure 79.2 Probing multiscale architecture in network above a certain weight value or threshold. Windowed thresh-
edge weights. Multiscale structure in a weighted network (left) olding (bottom) is the procedure whereby a connectivity
can be probed using thresholding techniques. Cumulative matrix is separated into a set of graphs that contain edges
thresholding (top) is the procedure whereby a connectivity whose weight values lie within a given weight range or window
matrix is separated into a set of graphs that contain edges (Bassett et al., 2012; Schwarz & McGonigle, 2011).
that are within versus between hemispheres (e.g., sparse graphs and large weight ranges produce dense
Bassett, Nelson, Mueller, Camchong, & Lim, 2012; Her- graphs.
mundstad et al., 2013). Simple binary networks (where Cumulative and windowed thresholding techniques
edges are treated as either present or absent) and non- can provide insight into the multiscale nature of a
embedded networks (where edge locations in the brain network’s connectivity patterns, embedded in edge
are ignored) necessarily neglect this multiscale struc- weights. The windowed thresholding approach has the
ture, and in doing so dismiss potentially important bio- advantage of isolating the network structure of strong
logical signatures present in the data. versus weak edge weights. Strong edges can provide
For simplicity, we will use the remainder of this sub- insight into the organization of energetically costly links
section to discuss the role of edge strength in multiscale in anatomical brain networks and of heavily utilized
brain structure. To study the network topology (the coordination links in functional brain networks. Weak
arrangement of the elements in a network) and puta- edges, while technically less significant according to
tive biological utility of edge strength, we can probe some statistical tests (Achard et al., 2006), can imply
weighted networks (where edges maintain estimated strongly correlated network states (Schneidman, Berry,
strengths) using a variety of thresholding techniques. Segev, & Bialek, 2006) and can distinguish diseased
Cumulative thresholding is the procedure whereby a network structures in schizophrenia (Bassett et al.,
family of graphs is created from a single connectivity 2012). Observed weak correlations between regional
matrix; each graph in the family contains edges above activity could be driven at least in part by the variability
a certain weight value or threshold (see figure 79.2, of neuronal activity signals, which plays an important
top). The threshold for each graph in the family is role in cognitive function (Misic, Mills, Taylor, &
unique, and specifies the density of edges in the graph: McIntosh, 2010), development (McIntosh et al., 2010),
small values of the threshold produce dense graphs and and recovery from injury (Raja Beharelle, Kovacevic,
large values of the threshold produce sparse graphs. McIntosh, & Levine, 2012).
Windowed thresholding is the procedure whereby a
different family of graphs is created; each graph in the Generating Network Diagnostics A network
family contains edges whose weight values lie within a “diagnostic” is simply a named measurement used to
given weight range or window (see figure 79.2, bottom; describe a network’s properties. These properties can
Bassett et al., 2012; Schwarz & McGonigle, 2011). The then be compared across different networks. For
width of the weight range specifies the density of each example, a patient’s network diagnostics can be com-
graph in the family: small weight ranges produce pared with those of a healthy participant; or a human
A 1 2 B 1 C
Clustering coefficient
Group 1
lo610 (Network efficiency)
75.2
Clustering coefficient
Clustering coefficient
Group 2 57.5
0.5 1 0.5
Network efficiency
Area 21.2
0 0 0 9.4
0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1
Density Density
Figure 79.3 Using diagnostics to probe brain network struc- used to determine whether the area between the two curves
ture. (A) Clustering coefficient (left y-axis, black) and network is significant in comparison to a null model. (C) Area under
efficiency (right y-axis, light gray) as a function of network the clustering coefficient versus density curve (top) and the
density as estimated using cumulative thresholding tech- network efficiency versus density curve (bottom) for 112
niques. (B) Schematic of mean clustering coefficient versus regions of the brain, defined according to the Harvard-
density curves for two sets of participants: group 1 (thin, light- Oxford atlas (see Bassett, Wymbs, et al., 2011, and Bassett,
gray dashed line) and group 2 (thick, dark-gray dashed line). Porter, et al., 2013, for additional details on this data set).
Functional data analysis is a statistical framework that can be
REFERENCES
Achard, S., & Bullmore, E. (2007). Efficiency and cost of
Box 79.1: Reviews and Resources economical brain functional networks. PLoS Comput Biol, 3,
e17.
• Several reviews address the general techniques Achard, S., Salvador, R., Whitcher, B., Suckling, J., &
used in applying network science to neuroscience data: Bullmore, E. (2006). A resilient, low-frequency, small-
Bassett and Bullmore (2006); Reijneveld, Ponten, world human brain functional network with highly con-
Berendse, and Stam (2007); Bullmore and Sporns nected association cortical hubs. J Neurosci, 26(1), 63–72.
(2009); He and Evans (2010); Bullmore and Bassett Allen, E. A., Damaraju, E., Plis, S. M., Erhardt, E. B.,
(2011); Wig, Schlaggar, and Petersen (2011); Sporns Eichele, T., & Calhoun, V. D. (2014). Tracking whole-
(2011); Kaiser (2011); Bullmore and Sporns (2012); brain connectivity dynamics in the resting state. Cereb Cortex,
Sporns (2012); Stam and Straaten (2012); Sporns 24(3), 663–676.
(2012). Bassett, D. S., Brown, J. A., Deshpande, V., Carlson, J. M.,
• A smaller number of reviews focus on network & Grafton, S. T. (2011). Conserved and variable architec-
applications in disease: Bassett and Bullmore (2009); ture of human white matter connectivity. NeuroImage, 54(2),
He, Chen, Gong, and Evans (2009); Xie and He (2011); 1262–1279.
Fornito et al. (2012). Bassett, D. S., & Bullmore, E. T. (2006). Small-world brain
• Networks: An Introduction is an excellent network networks. Neuroscientist, 12, 512–523.
science textbook (Newman, 2010). Bassett, D. S., & Bullmore, E. T. (2009). Human brain
• Networks of the Brain is a wonderful book about the networks in health and disease. Curr Opin Neurol, 22(4),
application of network science to neuroscience (Sporns, 340–347.
2010). Bassett, D. S., Greenfield, D. L., Meyer-Lindenberg, A.,
Weinberger, D. R., Moore, S., & Bullmore, E. (2010).
ABSTRACT Computational models of neurons allow us to is that the basic principles generalize: given appropriate
understand how cellular components interact to generate experimental measurements, this model and its more
neuronal activity. Model activity, and consequently model elaborate cousins that include additional currents can
explanatory power, depends on model parameter values. How
do we choose parameters for neuron models in a principled
describe the excitable electrical behavior of all types of
way? Some parameters (such as neuronal resting potential or neurons in any nervous system (Armstrong & Hille,
average spiking rate) can be estimated directly from experi- 1998).
mental data; however, many other parameters (such as density Experimental work over many decades has revealed
of ion channels in the neuronal membrane) cannot be con- how other components of neurons and networks, such
strained easily unless they are explicitly measured. More
as synapses, dendrites, and other membrane conduc-
importantly, many parameters in biological systems are highly
variable, so even with careful measurements it is inaccurate tances, operate in quantitative terms. With modern
to represent a distribution of measured values with a single imaging and electrophysiological and molecular tech-
model parameter value. Here, we describe several techniques niques, it is becoming possible to obtain data in living
to overcome these problems in the context of conductance- nervous systems from the level of individual biochemi-
based neuron models. Rather than building one, representa-
cal reactions to the activity patterns in large neural
tive model, we construct a family of models, each with
parameters drawn from a distribution of values based on circuits. Similarly, computer power is increasing at a
experimental observations. From this population, we select rate that makes detailed simulations of these phenom-
those models that match observed activity and discard the ena possible (Alivisatos et al., 2012; de Schutter,
rest. Overall, the techniques we describe result in computer Ekeberg, Kotaleski, Achard, & Lansner, 2005; Markram,
models that explicitly capture the variability observed in 2006). Even with these advances, a major challenge
neurons and overcome the problem of manually choosing
parameters for these models. when building neuron models is turning experimental
measurements into appropriate model parameters.
To fully understand nervous system function, we need Thus, in principle, we can construct biologically real-
to build quantitative models that incorporate experi- istic models of neuronal networks as a means of under-
mentally measured properties of neurons and networks. standing nervous system function at any level of detail
Models allow us to understand how different compo- that can be accessed experimentally. In practice, it is
nents and processes interact to generate emergent not possible to fully estimate all the parameters of a
behavior and therefore provide a means of articulating neural system by experimental observation because
and testing theories of how nervous systems work. The even for a single neuron a detailed model can have tens
most famous example of this is the Hodgkin-Huxley or even hundreds of parameters representing morphol-
model of action potential generation in the squid giant ogy, channel densities, and the kinetics of the resulting
axon (Hodgkin & Huxley, 1952). This model explains membrane conductances. In addition, when attempt-
how action potentials, or “spikes,” in neurons are gener- ing to estimate values of model parameters from experi-
ated by an interaction between two different membrane mental data, one finds that the measured values are
currents, a fast inward (depolarizing) sodium current widely variable, even in neurons with virtually identical
and a slower outward (hyperpolarizing) potassium overall behavior (Marder & Goaillard, 2006). We
current. As is the case with all good models, its power emphasize that, in this context, these differences
are not due to measurement error; they represent
genuine biological differences. For example, figure
*These authors contributed equally to this work. 80.1 shows measurements of membrane currents in two
g max(mS/cm 2)
in models 300
1 200
Figure 80.2 (Golowasch et al., 2002) shows the behavior
100
of a computational model of a single cell with several
0
different types of conductances, including a fast sodium
(Na+) conductance and a slower, delayed-rectifier K+
conductance (Kd). Three different parameter sets are 400
g max (mS/cm 2)
used to generate three different models (figure 80.2A, 300
left panels 1–3). Each parameter set has distinct values 2 200
of the Na+ and Kd conductances (figure 80.2A, right 100
panels 1–3). These values were chosen to lie in an 0
experimentally determined range. The behavior of the
model is very similar in all three cases, in spite of large
400
g max (mS/cm 2)
variation in these two conductances. Significantly, when
a model is constructed from the average values of both 300
30 mV
simple example. First, very different underlying param-
eters can produce very similar model behavior. This 200 ms
means that the task of choosing an “optimal” set of
parameters based on overall behavior does not have a B
well-defined solution (i.e., there are multiple solutions 400
g max(mS/cm 2 )
that generate a given behavior). In other words, a con- 300
sequence of neuronal variability is that many different 200
parameter combinations can produce functioning 100
neurons. Second, the average parameter values can give 0
qualitatively different behavior to three examples gNa gKd (× 4)
chosen within the experimental bounds. This indicates Figure 80.2 Different parameter combinations can produce
that the average parameter values are not necessarily equivalent model behavior. (A) Each row (1–3) shows a dif-
represented biologically in any single cell. ferent set of parameters for the same model neuron. Sample
This prompts caution in adopting strategies (1) and activity traces (left column) are qualitatively similar for each
(2) and suggests that model parameters might not be model (inset: expanded view of a single-spike burst, scale bar:
50 ms). The individual conductance values for the sodium
best described by single numerical values. Strategy (3)
and delayed-rectifier potassium conductances vary substan-
avoids this problem and allows us to address interesting tially over the three different models. (B) A model that uses
biological questions, such as how sensitive a biological the average values (right column) of the conductances used
system is to variation in specific parameters or how the in (A). The model does not produce the same qualitative
parameters might covary to give a particular global behavior observed in all three models in (A). Specifically, this
model has multiple spikes for each burst (left panel, inset:
behavior (Marder & Taylor, 2011). However, working
expanded view, scale bar: 50 ms). (Adapted from Golowasch
with populations of models introduces technical prob- et al., 2002.)
lems, most notably the computing power required to
run many instances of a model rather than a single
instance. We illustrate these concepts with single-cell clamp and current clamp recordings, from which we
examples; however, the methods are quite general and can estimate a subset of the biophysical parameters,
can be used for entire networks. such as the properties of different ionic conductances.
Across multiple cells, we would obtain a range of values
Capturing parameter variability using for each parameter that would provide an estimate of
families of models the mean value and the spread of this biological feature.
If we have a large number of data points, we can infer
Suppose we want to build a model neuron using experi- something of the shape of the distribution of parame-
mental data. The data consist of intracellular voltage- ters. For example, we may be able to infer a covariance
…
Neuron behavior
{Parameter set n}
Figure 80.3 Procedure for generating populations of model randomly sampling, parameter values are drawn according to
neurons. Left: Maximal conductance measurements from the probability distributions estimated from the experimental
hypothetical experiments are taken (black circles). These data (dotted lines). When systematically sampling, parameter
measurements allow the estimation of the distributions of values are drawn at regular intervals and all possible combina-
each property in the population (vertical curves; gray arrows tions of the intervals are used. Right: Resulting models are
indicate sample means; sample points fit with Gaussians). simulated. Many resulting neuron models have qualitatively
Below, the output of an example neuron shows periodic burst- similar bursting behavior to what is observed experimentally
ing behavior, which is typical in this type of neuron (scale bar: (rows 1, 3, 4). However, one of the neurons shows tonic
100 ms). Middle: Each model in the population is created spiking behavior (box), highlighting that different qualitative
either by randomly sampling the parameter space (left) or by behaviors are possible from these parameters (scale bar:
systematically sampling the space on a grid (right). When 100 ms).
structure and then sample from distributions that pre- exponential in the number of parameters. It is easy to
serve covariance (such as multidimensional Gaussian see that regardless of how powerful our computing
distributions with the same covariance). resources are, we only need to add a few parameters
We can then construct a family of models in one of before the simulation becomes impractical or unfeasi-
two ways, as illustrated in figure 80.3. We can randomly ble. For example, suppose it takes 1 second to simulate
sample the measured parameter distributions, choosing a population with 10 parameters. Each additional
values over the experimentally observed range—we parameter increases the simulation time 10-fold, so
refer to this as the random sampling method. Alterna- simulating this same population with 11 parameters
tively, we can systematically vary each parameter inde- would take 10 seconds, while a simulation with 20
pendently over the observed range using prespecified parameters takes 1010 seconds, or 317 years.
intervals. We call the latter approach systematic sampling Even single-neuron models can easily require tens of
(or “grid” sampling). parameters, so the curse of dimensionality is very real,
Systematic/grid sampling is attractive because it especially when one considers larger-scale simulations
explicitly traces the dependence of model behavior as of networks of neurons. Yet it is nonetheless possible to
each parameter is varied and will generate all combina- obtain useful information on how underlying parame-
tions of parameter values over the chosen intervals ter variability contributes to model behavior using
(Prinz et al., 2003). For example, we might be inter- random sampling in place of systematic sampling
ested in how the firing rate of a neuron changes as Na+ (Sobie, 2009; Taylor et al., 2009). While random sam-
conductance is increased (for examples, see below) for pling is never guaranteed to cover specific combina-
all possible combinations of other parameters. However, tions of parameters, finding how the behavior of the
the disadvantage of this approach is that it suffers from models depends on the distribution of parameters is
the so-called curse of dimensionality. Suppose we have not subject to the curse of dimensionality. This is
N different parameters and we want to vary each over because the computational cost of a single simulation
the experimentally observed range. If we wish to sample is fixed (it can of course, be significant); however, the
each parameter at M distinct values, then the total number of replicates in a family of simulations can be
number of models we need to simulate is MN. The chosen arbitrarily and will only be dependent on the
computational cost of performing this simulation is precision required to draw conclusions. For example,
50 mV
50
Frequency (Hz)
40
Frequency (Hz)
27 Hz 200 ms 60 Hz 20 ms 40
30
3x g Na
20
20
50 mV
0
10 0 1 2
Current (nA/nF)
0
0 2 4 6 8 10
C D E
250 400 Current (nA/nF)
300
200
Kd conductance
300
Count
Count
150 200
200
100
100
100
50
0 0 0
0 50 100 150 200 250 –0.2 –0.1 0 0.1 0.2 –6 –4 –2 0 2 4 6
Sodium conductance Rheobase shift (nA/nF) Voltage threshold shift (mV)
Axonal
D
B
Y
nP
nA
r
nP
oc
,p
F
a
ak
ak
a
ak
ak
KC
a
Kd
sy
sy
v½
Kd
Aa
pr
sy
le
le
P
C
le
le
N
h
E
g
A
Input conductance
Slow-wave amplitude
Slow-wave peak
Figure 80.4 Population simulations show that each neu- upper-right region is the model shown in (A). (D) Over the
ron’s behavior is sensitive to multiple intrinsic properties. whole population, tripling the sodium conductance always
(A) Example model neuron from a population in which the decreased the rheobase (the minimum current required to
sodium conductance was tripled (black: control; gray: 3x gNa). fire an action potential). All points in the histogram are below
The left column shows the model neuron at low current injec- zero (gray line). (E) Similarly, voltage threshold was hyperpo-
tion (firing rate increases with 3x gNa), and the right column larized for all 1,000 models when the sodium conductance
shows the model neuron at high current injection (firing rate was tripled. (Adapted from Kispersky et al., 2012.) (F) The
decreases with 3x gNa). (B) Frequency-current curves for this neuron-level properties (rows) of a population of lateral
example neuron with fit lines. (C) A population of 1,000 pyloric neurons are plotted relative to their sensitivities to
randomly generated models was considered in this study. biophysical parameters (columns). The size of each circle
Models were only included in the population if they fired represents how sensitive a given behavior was to that property.
tonically at low current injection level. Black points represent In general, each property was sensitive to several parameters
the sodium and delayed rectifier potassium conductance of at once. (Adapted from Taylor et al., 2009.)
single models in the population. The large gray dot in the
ABSTRACT One of the best-studied examples of the functional Clark, Armstrong, & Moore, 2011; Histed, Ni, &
organization of cerebral cortex is the visual system, where Maunsell, 2012). When applied to human subjects, elec-
distinct cortical areas that respond selectively to visual stimuli trical stimulation offers a unique opportunity to study
have been identified in the primate brain. However, the fact
that these areas respond to visual stimuli falls short of proving
the qualitative properties of stimulation-induced visual
that they are critical to visual perception. The long-recognized percepts, because human subjects can readily offer
observation that electrical stimulation of human occipital verbal descriptions of percepts. This can offer insights
cortex can make a patient experience a visual percept in the about how activity in visual cortex underlies visual per-
absence of an external visual stimulus offers perhaps the ception, and may advance efforts to restore vision with
strongest evidence that visual cortex has a causal role in visual
cortical prosthetics for blind patients.
perception. In this chapter, we review our recent efforts to
understand how electrical stimulation of cortex can produce
a visual percept and to compare the abilities of different Early studies of electrical stimulation of human
human visual areas to contribute to perception. We also con- visual cortex
sider implications of this work to understanding of the neural
basis of visual perception, and to the future development of
a cortical visual prosthetic to restore vision to the blind.
In 1870, Gustav F. Fritsch and Eduard H. Hitzig used a
thin metal probe to apply an electrical current to the
cortical surface of awake dogs, finding that stimulation
How does activity in the brain lead to visual perception, of the central part of the hemisphere resulted in con-
a term that we use as an umbrella for all of the mysteri- tralateral limb movements (Fritsch & Hitzig, 1870).
ous processes that result in conscious awareness of an This finding was subsequently corroborated in human
external visual stimulus? The most popular approach to patients by the neurosurgeons Fedor K. Krause and
this question is to measure the brain activity that occurs Ortfrid F. Foerster (Luders & Luders, 2001). These dis-
during presentation of natural stimuli, using techniques coveries helped overturn the previous dogma that cog-
ranging from microscopic (recordings of individual nitive functions are equally distributed across the
neurons using penetrating microelectrodes) to macro- cerebral cortex (equipotentiality), leading to a new
scopic (measurements of large populations of neurons understanding that the brain is divided into function-
using blood oxygen level–dependent functional mag- ally specialized regions. In the ensuing years, electrical
netic resonance imaging, or BOLD fMRI). These tech- stimulation to study human brain function was pio-
niques and others have been used to show that the neered by Canadian neurosurgeon Wilder Penfield
primate visual system contains over 30 different cortical in a series of landmark studies beginning in 1928
areas (Orban, Van Essen, & Vanduffel, 2004). However, (Penfield, 1958; Penfield & Rasmussen, 1950). Penfield
a correlation between the presentation of a visual stimu- used electrical stimulation to study the function of corti-
lus and activity in a specific neuronal population does cal areas in awake and cooperative patients undergoing
not necessarily indicate a causal relationship between brain operations under local anesthesia, usually for the
the measured neural activity and visual perception. To treatment of epilepsy. He extended the results of Fritsch
demonstrate such a relationship, it is necessary to acti- and Hitzig to map the organization of motor cortex, the
vate the relevant neurons and demonstrate that this motor homunculus, for which he is perhaps best
results in a visual percept (Parker & Newsome, 1998). remembered today. He also examined sensory cortex
Electrical stimulation is a technique that enables and was able to elicit reports from patients on percep-
researchers to artificially activate a specific brain region, tual experiences produced with stimulation, including
affording the opportunity to study how activity in that somatosensory, auditory, and visual percepts (Penfield
region influences or even produces a visual percept & Perot, 1963; Perot & Penfield, 1960). Penfield and
(Borchers, Himmelbach, Logothetis, & Karnath, 2011; co-workers found that stimulation of the occipital lobe
Percent correct
to early visual areas, and there is no sharp dichotomy
between early and late visual areas in their abilities to 50
support perception.
For the sites that did produce a detectable percept
0
with stimulation, subjects were asked to report on the 1 1.5 2.0 2.5
qualitative properties of the percept. Subjects usually Current (mA)
reported a consistent percept with repeated stimulation
D 3
of a specific individual electrode. Stimulation of many
sites in both early and late visual areas elicited descrip- 2.5
Threshold (mA)
tions of consistent simple phosphenes (small flashes of
2
white or silver light). Stimulation of some areas elicited
reports of more complex percepts, with descriptions of 1.5
color or more complex shapes. However, no systemic 1
difference was noted between percept complexity for
0.5
early and late sites. In contrast to previous reports
(Blanke, Landis, & Seeck, 2000; Lee et al., 2000; Perot 0
0 20 40 60 80 100 120
& Penfield, 1960; Puce, Allison, & McCarthy, 1999), no Distance from occipital pole (mm)
elaborate percepts such as images of faces or objects or
memories were described from any site where electrical Figure 81.3 Determining thresholds for detecting electrical
stimulation produced a detectable percept. This dis- stimulation. (A) Each trial contained two 300-msec epochs,
marked by the words “one” and “two.” The current amplitude
crepancy could reflect the fact that electrodes that
(and the epoch containing the stimulus) varied from trial to
produce complex visual percepts are relatively rare, and trial. The subject’s task was to detect the epoch in which the
our study tested a relatively small number of electrodes. stimulation was delivered. A sample trial is shown in which a
It is also possible that this discrepancy relates to the fact high-amplitude current train was delivered in the second
that previous studies did not objectively verify that the epoch, and the subject responded by pressing mouse button
subject actually detected stimulation of a particular two and received positive feedback. (B) A sample trial in
which a low-amplitude current was delivered in the first
electrode—perhaps the oft-cited accounts of subjects epoch, and the subject responded by pressing mouse button
reporting a complex visual perce