NeuroImage 127 (2016) 144–157

Contents lists available at ScienceDirect

NeuroImage

journal homepage: www.elsevier.com/locate/ynimg

Decisions about the past are guided by reinstatement of specific

memories in the hippocampus and perirhinal cortex
Michael L. Mack a,b,⁎, Alison R. Preston a,b,c,⁎
a
Department of Psychology, The University of Texas at Austin, Austin, TX 78712, United States
b
Center for Learning and Memory, The University of Texas at Austin, Austin, TX 78712, United States
c
Department of Neuroscience, The University of Texas at Austin, Austin, TX, 78712, United States

a r t i c l e i n f o a b s t r a c t

Article history: When faced with a new challenge, we often reflect on related past experiences to guide our behavior. The ability
Received 16 July 2015 to retrieve memories that overlap with current experience, a process known as pattern completion, is theorized
Accepted 10 December 2015 as a critical function of the hippocampus. Although this view has influenced research for decades, there is little
Available online 15 December 2015
empirical support for hippocampal pattern completion to individual memory elements and its influence on
behavior. We used pattern analysis of brain activity measured with functional magnetic resonance imaging to
Keywords:
Decision making
demonstrate that specific elements of past experiences are reinstated in the hippocampus, as well as perirhinal
Hippocampus cortex (PRC), when making decisions about those experiences. Linking neural measures of specific memory
Medial temporal lobe reinstatement in the hippocampus and PRC to behavior with computational modeling revealed that
Memory reinstatement predicts the speed of memory-based decisions. Moreover, hippocampal activation during retrieval
Perirhinal cortex was selectively coupled to regions of occipito-temporal cortex that showed content-specific item reinstatement.
Computational modeling These results provide evidence for hippocampal pattern completion and its role in the mechanisms of decision
making.
© 2015 Elsevier Inc. All rights reserved.

Decisions often rely on information that is not immediately Specifically, the hippocampus is thought to act as an auto-associative
available. Whether it is ordering an entrée at a favorite restaurant or network that reinstates complete memory representations from partial
deciding among treatment options for a medical condition, many of or degraded input. It has been further proposed that the content of these
our decisions are guided by what we recall from past experience. Most reinstated memories serve as internally-generated evidence that guides
research on the neural systems of decision making have focused on subsequent behavior (Lisman and Grace, 2005; Norman and O'Reilly,
perceptual decisions that depend on information directly or recently 2003). In the present study, our goal was to test the theoretical predic-
available to the sensory system (Heekeren et al., 2008). Although such tion that during memory-based decision making, specific memory
studies have characterized how external sensory information is repre- elements are reinstated in the human hippocampus and that this rein-
sented and interpreted by the brain to influence behavior (Bogacz statement predicts decisions.
et al., 2010; Bollimunta et al., 2012; Gold and Shadlen, 2007; Hanks Empirical evidence for hippocampal memory reinstatement and its
et al., 2014; Heekeren et al., 2008; Nienborg and Cumming, 2009), the influence on decision making has been established in human electro-
neural computations that support how we use internally-generated physiological studies. This work has demonstrated that firing patterns
content from past experiences to guide decision making remain poorly of single neurons in medial temporal lobe selectively code for specific
understood. contents of memory (Heit et al., 1988) and that reinstatement of these
It is well known that our ability to encode and remember past firing patterns during retrieval is related to memory performance
experiences depends critically on the medial temporal lobe (MTL) (Gelbard-Sagiv et al., 2008; Paz et al., 2010; Rutishauser et al., 2015).
(Eichenbaum, 2004; Squire, 2004). The prominent theoretical view However, such empirical evidence of pattern completion in human
(Marr, 1971; McClelland et al., 1995; O'Reilly and Rudy, 2001) proposes functional magnetic resonance imaging (fMRI) research has been limit-
that successful memory retrieval involves reinstatement of memory ed. Initial reports focused on differences in process rather than contents
representations through a process known as pattern completion. of memories, inferring pattern completion from a reduction in blood-
oxygen-level dependent (BOLD) response to repeated presentations of
objects and similar lures relative to initial encoding (Bakker et al.,
⁎ Corresponding authors at: Center for Learning and Memory, The University of Texas at
Austin, 100 East 24th Street, Austin, TX 78712-0805, United States.
2008; Lacy et al., 2011). Such an effect is consistent with pattern
E-mail addresses: [email protected] (M.L. Mack), [email protected] completion; however, it is unknown whether or not such reduced
(A.R. Preston). neural responding reflects reinstatement of specific memory elements.

http://dx.doi.org/10.1016/j.neuroimage.2015.12.015
1053-8119/© 2015 Elsevier Inc. All rights reserved.
 M.L. Mack, A.R. Preston / NeuroImage 127 (2016) 144–157 145

fMRI studies have demonstrated that hippocampal activation patterns demonstration of hippocampal cells representing memory evidence,
during spatial context encoding exhibit a bias towards existing spatial the kind of information carried by these cells was not item specific,
templates, a finding consistent with pattern completion processes and the memory-specific firing patterns were observed in response to
(Stokes et al., 2015). Further, it has been shown that hippocampal perceptual information currently available to the participant. The
activation patterns during retrieval differentiate between stored theoretically important question remains as to whether or not internal-
event representations (Chadwick et al., 2010, 2011) and are consistent ly generated hippocampal memory evidence is directly related to
with episode-specific templates of activation patterns recorded decisions about the specific mnemonic content.
after encoding and retrieval (Wimber et al., 2015). However, these Here, we sought direct evidence of pattern completion during
extant findings cannot speak to the central tenet of pattern completion: memory-based decision making by indexing reinstatement of the
hippocampal activation patterns representing specific memory specific components of individual memories from patterns of fMRI
elements are themselves reinstated during decisions that rely on activation in the human hippocampus and surrounding cortices. Partic-
memory. ipants learned arbitrary associations between pictures of objects and
A second theoretical aspect of pattern completion with limited famous faces and places before performing a delayed match-to-
empirical support is the proposal that reinstated mnemonic information memory test (Chen et al., 2011), in which objects cued memory
in the hippocampus serves as evidence for memory-based decision retrieval of the associated face or place (Fig. 1). We used neural pattern
making (Lisman and Grace, 2005; Norman and O'Reilly, 2003). It has similarity (Kriegeskorte et al., 2008; Xue et al., 2010) to assess the
been demonstrated that successful memory decisions are associated reinstatement of specific people and places during memory retrieval
with stronger reinstatement signatures in content-specific cortical in parahippocampal cortex (PHC) and perirhinal cortex (PRC), MTL
regions (Bosch et al., 2014; Johnson et al., 2009; Kuhl and Chun, 2014; cortical regions that have demonstrated content-specific coding
LaRocque et al., 2013; Polyn et al., 2005; Ritchey et al., 2012; Staresina (e.g., LaRocque et al., 2013; Liang et al., 2013; Staresina et al., 2012),
et al., 2012; Wing et al., 2015). However, the behavioral link between and hippocampus (Fig. 1). Item-specific patterns during retrieval were
memory decisions and hippocampal pattern completion has yet to be compared to item-specific neural patterns recorded during a pre-
observed. Furthermore, the link between reinstatement and behavior exposure phase, in which participants viewed each item in isolation
observed in these studies is based on the indirect comparison of cortical prior to associative encoding. This approach allowed us to go beyond
reinstatement signatures during successful versus unsuccessful memo- category level decoding of memory reinstatement (Johnson et al.,
ry choices, thus limiting the sensitivity of evaluating the direct influence 2009; Polyn et al., 2005), a method that has failed to provide evidence
of reinstatement on decision making on a trial-by-trial basis. Recent for reinstatement of hippocampal representations (Diana et al., 2008;
neuroimaging evidence shows that the magnitude of hippocampal Gordon et al., 2014; cf. Kuhl and Chun, 2014), to assess reinstatement
activation is linked to memory confidence judgments (Leiker and of specific memory elements.
Johnson, 2015; Thakral et al., 2015) and the speed of memory decisions Furthermore, we investigated the theoretical proposal that reinstat-
(Gordon et al., 2014), with greater retrieval-related activation for higher ed mnemonic information in hippocampus serves as evidence for
confidence and faster responses. Electrophysiological work in humans memory-based decision making (Lisman and Grace, 2005; Norman
has further identified single cells within the hippocampus that exhibit and O'Reilly, 2003). To this end, we assessed the link between neural
firing patterns consistent with a graded representation of memory reinstatement of specific memory content in MTL subregions, including
strength and confidence during recognition decisions (Rutishauser the hippocampus, and decisions about the memory probes with a math-
et al., 2015). However, these studies did not index hippocampal pattern ematical model of decision making commonly used to assess
completion per se. And, although the single cell finding is a compelling how perceptual evidence impacts decisions (Ratcliff, 1978). Critically,

A B

hipp

PHC x = -28

PRC
C
pre-exposure pair learning delayed match-to-memory test
study trial test trial

+
...
TAJ MAHAL BOOT TAJ MAHAL

1s stim 3.5s stim until response 1s cue 9s delay 1s probe

Fig. 1. Schematic of experimental task and depiction of brain regions of interest (ROIs). (A) Eight famous face and eight famous place images were used as stimuli in the experiment.
(B) Analyses were conducted in MTL subregions: PHC (light blue), PRC (yellow), and the hippocampus (red). (C) The experiment consisted of three phases. In the pre-exposure phase,
participants viewed images of faces, places, objects, and fixation points in isolation. During the pair learning phase, participants learned to criterion eighty paired associates consisting
of a common real-world object and either a famous face or place through repetitions of first studying all pairs followed 3AFC tests on associate memory. Finally, participants were tested
on each of the eighty pairs during a delayed match-to-memory (DMTM) test. Each test trial consisted of: an object cue (1 s), a delay period (9 s), a probe face or place (1 s) followed by an
inter-trial interval (5-11 s, mean 7.5 s). The probe item was either the correct paired associate (a match) or a same-category foil drawn from another pair (a mismatch). Participants in-
dicated match or mismatch status with a button response.
146 M.L. Mack, A.R. Preston / NeuroImage 127 (2016) 144–157

our model-based approach extends beyond correlation methods Task procedure

(e.g., Gordon et al., 2014) to test whether or not MTL neural reinstate-
ment signatures are predictive of subsequent behavior. Typical ap- After an initial screening and consent in accordance with the Univer-
proaches to relating brain measures to behavior are largely sity of Texas Institutional Review Board, participants were instructed on
discriminative, focusing on differences in neural activation across the pre-exposure, associative learning, and delayed match to memory
known conditions. For example, studies that compare neural measures (DMTM) tasks. Participants then performed all three tasks in the MRI
based on successful versus unsuccessful memory performance have re- scanner by viewing visual stimuli back-projected onto a screen through
vealed that cortical reinstatement is stronger for correct than incorrect a mirror attached onto the head coil. Foam pads were used to minimize
memory decisions (e.g., Kuhl and Chun, 2014). The discriminative ap- head motion. Stimulus presentation and timing was performed using
proach can reveal potentially important neural differences across condi- custom scripts written in Matlab (Mathworks) and Psychtoolbox
tions, but the resulting findings are ultimately correlational. In contrast, (www.psychtoolbox.org) on an Apple Mac Pro computer running OS
a generative approach attempts to describe the system of interest first X 10.7.
and then evaluate the system under different conditions. Such an
approach is considered generative because the modeled system can
Pre-exposure phase
generate predictions for novel situations (Ng and Jordan, 2002). Here,
During the pre-exposure, participants were instructed to respond as
we take a generative approach by using a modeling framework that
quickly as possible when a centrally-located fixation cross changed
makes explicit assumptions about how neural information available in
color to blue or yellow by pressing one of two buttons on a MRI-
the MTL influences memory decisions. By doing so, we provide a strong
compatible button box. On each trial, a face, place, or object image
and direct test of the hypothesis that internally generated memory
was presented for 1 s along with a black fixation cross (Fig. 1C). After
evidence in the hippocampus predicts mnemonic decision making on
a random delay (250-750 ms) from image and fixation onset, the
a trial-by-trial basis.
fixation cross changed color to blue or yellow. After 1 s had elapsed
We also tested the key hypothesis that reinstatement in the
from the onset of the image and fixation cross, a blank screen was
hippocampus is linked to reinstatement in cortex (Ciocchi et al., 2015;
presented for 3-6 s. Participants were instructed to respond only to
Tanaka et al., 2014). It has been demonstrated that the magnitude of
the fixation cross color change and that the image stimuli were irrele-
hippocampal BOLD activation during both encoding and retrieval scales
vant to the task. Trials were presented in a mixed design with blocks
with reinstatement signatures in neocortex (Gordon et al., 2014; Horner
of eight trials consisting of stimuli from the same visual category
et al., 2015; Ritchey et al., 2012; Wing et al., 2015). Such findings suggest
(faces, places, and objects). Within a block, trials were randomly jittered
that the hippocampus plays an important role in the re-experiencing of
with an average trial duration of 5.5 s and a range of 4-7 s. Each of the
mnemonic content by influencing cortical reinstatement. We extend
eight famous faces and places and eight objects were presented twice
this work by asking whether or not regions of the hippocampus and
across a pre-exposure run. Five fixation blocks, each lasting 15 s, were
the surrounding cortex that demonstrate item-specific reinstatement
presented interleaved with the stimulus blocks. Each run lasted 341 s,
are functionally coupled to regions of occipito-temporal cortex (OTC)
and participants completed four runs. The entire pre-exposure phase
that selectively reinstate specific mnemonic content. Such a functional
lasted approximately twenty-five minutes.
link would suggest that not only is item-specific memory evidence
reinstated in the hippocampus, but that the same underlying neural
substrate is also communicating with regions of neocortex that Associative learning phase
demonstrate item-specific reinstatement. Each of the eight famous faces and places were paired with five
objects for a total of eighty pairs. These pairings were randomly selected
across participants. Participants performed repetitions of study and test
Materials and methods blocks to learn the paired associates. Although participants performed
the associative learning phase in the scanner, we did not collect fMRI
Participants data for this experimental phase. On each trial of a study block, a face-
object or place-object stimulus pair was presented for 3.5 s followed
Twenty-five volunteers (15 females, mean age 20.2 years old, by 0.5 s of fixation (Fig. 1C). The stimulus images were presented at
ranging from 18 to 29 years) participated in the experiment; one partic- the same time as the object image. The object image appeared 8.1° to
ipant was excluded from analysis due to a failure in the button-response the left of the screen center, and the face/place image 8.1° to the right
recording device and an additional participant was removed from of the screen center. Labels for the faces/places and objects were
further analysis due to substantial head motion. All subjects were presented in black, 32-point, Helvetica text below the images. Each
right handed, had normal or corrected-to-normal vision, and were image subtended 7.3° × 7.3° of visual space. Participants were
compensated $75 for participating. instructed to view and attempt to remember the pairs. All 80 pairs
were shown in random order during a study block.
After study, participants were tested on the eighty paired associates.
Materials On each test trial, an object stimulus from one of the pairs was present-
ed above three faces or places, with one face/place being the correct
Eight color images of famous faces (Oprah Winfrey, Taylor Swift, paired associate and two faces/places from different pairs serving as
Meghan Fox, Julia Roberts, Barack Obama, Steve Martin, Owen Wilson, lures. Lures were always from the same category as the correct face or
and Morgan Freeman) and famous places (Mount Fuji, Florida place. Participants responded by pressing one of three buttons on a
Everglades, Yosemite National Park, Monument Valley, Eiffel Tower, MRI button box to indicate which of the three face/place stimuli was
Las Vegas Strip, Taj Mahal, and London Bridge) as well as eighty-eight associated with the object. Test trials were self-paced with a 1 s feed-
color images of everyday objects were used in the experiment (Fig. back screen that presented the object along with the correct face/
1A). Face and object images consisted of a cropped face or object placed place. Test trials were separated by a 500 ms fixation. All 80 pairs
on a white background. All stimuli were sized to 300 × 300 pixels. Eight were tested in random order. After each test block, the average accuracy
independent raters provided visual distinctiveness ratings for a set of 30 was calculated across the test trials for that block. If the test accuracy
famous faces and places. The four most visually distinctive female and was greater than or equal to 85%, participants moved onto the delayed
male faces and four most visually distinctive natural and man-made match to memory task phase. Otherwise, participants repeated the
places were selected for the experiment. study and test blocks with different randomized trial orders.
 M.L. Mack, A.R. Preston / NeuroImage 127 (2016) 144–157 147

Delayed match to memory (DMTM) task transition zone between PHC and PRC was excluded from both regions.
The test phase consisted of trials of a delayed match to memory task. Hippocampal, PHC, and PRC ROIs were reverse-normalized to each
In contrast to the widely-used delayed match to sample task, in which a individual's functional space using ANTS. Specifically, a nonlinear
stimulus is presented and held in working memory through a delay transformation was calculated from the MNI template brain to each
period before a match decision is made to a probe stimulus, the delayed participant's T1-weighted MPRAGE volume. This warp was then
match to memory task requires cued retrieval from long-term memory concatenated with the MPRAGE to functional space transformation
during the delay period. On each trial of the DMTM phase, an object calculated using ANTS. Finally, the concatenated transformation was
from one of the eighty pairs encoded during the associative learning applied to the anatomical MTL ROIs to move each ROI into each
phase was presented as a memory cue for 1 s followed by a 9 s blank participant's functional space. See the Supplemental Materials for
screen. After the 9 s delay, a memory probe was presented that was methods and findings from an exploratory analyses of hippocampal
either the correct face/place paired associate (a match trial) or a differ- subfields DG/CA2,3 and CA1.
ent face/place associated with a different object (a mismatch trial). The A region of interest including occipito-temporal cortex (OTC) was
probe stimulus was presented for 1 s before presentation of a 3 s fixation constructed from an automated cortical segmentation performed with
(Fig. 1C). Participants were instructed to respond whether or not the Freesurfer, version 5.3 (Fischl et al., 2002). Specifically, the Freesurfer
probe stimulus was an associative match or mismatch to the object cortical segmentation was performed on each participant's T1-
cue by pressing one of two buttons on a MRI compatible button box. weighted MPRAGE volume and all cortical regions contained within
Participants were also instructed to respond within 1 s of the probe occipito-temporal cortex, but excluding medial temporal lobe cortex,
onset. A strict response deadline was used to encourage memory were added to an OTC ROI that was specific to each participant. The
retrieval of the associated face/place during the delay period. Each run OTC ROIs was transformed to each participant's functional space using
of the DMTM phase consisted of sixteen trials. Across these sixteen ANTS.
trials, each of the eight famous faces and places was a retrieval target,
half of which were match trials and the other half mismatch trials. Pattern classification analysis
Across five runs all objects from the eighty learned pairs were presented
as a cue. Null fixation events with durations of 2-8 s sampled from a The goal of the classification analysis was to assess the extent to
truncated exponential distribution for a total of 72 s were randomly which category-level information (face vs. place) was reinstated in
intermixed between trials throughout each run to maximize the different MTL subregions during memory retrieval (Fig. 3). Pattern
efficiency of the trial order. Each run lasted 294 s with the entire classification analyses were implemented using PyMVPA (Hanke et al.,
memory test phase lasting approximately thirty minutes. 2009) and custom Python routines. All classification analyses were
conducted on preprocessed and spatially smoothed functional data.
MRI data acquisition The decision to perform multivariate analyses on spatially smoothed
data is consistent with recent studies employing MVPA in the MTL
Whole-brain imaging data were acquired on a 3.0 T Siemens Skyra (Kuhl and Chun, 2014; e.g., Schapiro et al., 2012) and demonstrations
system at the University of Texas at Austin Imaging Research Center. A that smoothing does not result in information loss (Kamitani and
high-resolution T1-weighted MPRAGE structural volume (TR = 1.9 s, Sawahata, 2010; Op de Beeck, 2010). A linear support vector machine
TE = 2.43 ms, flip angle = 9°, FOV = 256 mm, matrix = 256 × 256, (SVM) classifier was trained to discriminate neural activation patterns
voxel dimensions = 1 mm isotropic) was acquired for coregistration of faces, places, objects, and fixation acquired during the pre-exposure
and parcellation. Functional images were acquired using a T2*-weight- phase. Each trial in the pre-exposure phase was labeled according to
ed multiband accelerated EPI pulse sequence (TR = 1 s, TE = 30 ms, the stimulus category that was presented during the trial. To account
flip angle =62°, FOV = 220 mm, matrix =110 × 110, slice thickness = for the hemodynamic lag, the fMRI data was time shifted by 4 s.
2 mm, multiband factor = 4) allowing for whole brain coverage with First, the classifier was evaluated for each of the MTL ROIs (PRC, PHC,
2 mm isotropic voxels. and hippocampus) with a leave-one-out 4-fold cross-validation using
the pre-exposure phase's four functional runs. Classifiers were trained
MRI data preprocessing and statistical analysis to predict face, scene, object, and fixation blocks during the pre-
exposure phase. All four categories were included in the classifier to
MRI data were preprocessed and analyzed using FSL 5.0 (Smith et al., isolate the four sources of potential information in neural patterns
2004) and custom Python routines. Functional images were realigned to during the DMTM phase. By including both objects and fixation in the
the first volume of the fourth run from the preprocessing phase to classifier, the delay period information associated with perceptual
correct for motion, spatially smoothed using a 4 mm full-width-half- processing of the cue object and the fixation point was accounted for
maximum Gaussian kernel, high-pass filtered (128 s), and detrended in the classifier evidences for those two categories. With this experi-
to remove linear trends within each run. Functional images were mentally irrelevant variance accounted for, the classifier evidences for
registered to the MPRAGE structural volume using Advanced Normali- face and scene categories during the DMTM delay period was less biased
zation Tools (ANTS, version 1.9; Avants et al., 2011). All analyses were and more accurately reflected category information reinstated through
performed in the native space of each participant. memory retrieval. Classifier performance during the pre-exposure
phase was assessed by taking the average accuracy across the cross-
Regions of interest validation folds within each participant, then evaluating whether or
not across the participant's classification accuracy was significantly
The hippocampus and subregions of MTL cortex (PRC and PHC) were greater than chance levels (25% chance accuracy with four classes)
delineated by hand on the 1 mm Montreal Neurological Institute (MNI) according to a Bayesian Estimation Supersedes the t-Test procedure
template brain (Fig. 1B) by an expert trained to segment MTL and (BEST; Kruschke, 2012).
hippocampus. Given the established functional differences in lateral BEST is a hierarchical Bayesian version of the traditional t-test that
versus medial subregions of entorhinal cortex (ERC; e.g., Knierim estimates the t-test statistic through a Markov-Chain Monte Carlo
et al., 2014) and the limited resolution of fMRI, we excluded ERC from (MCMC) sampling method. The BEST procedure is a generalization of
our analyses. The segmentations were validated by two other expert the Student t-test such that the specific form of the t distribution
raters and have been used in previous published studies (Schlichting (defined by the mean, standard deviation, and shape parameter) used
and Preston, 2014; Schlichting et al., 2015). PRC was defined as the by the test is estimated from the data through MCMC methods. Instead
lateral and medial wall of the collateral sulcus and a 3 mm wide of assuming normally-distributed data (i.e., setting the shape parameter
148 M.L. Mack, A.R. Preston / NeuroImage 127 (2016) 144–157

equal to ∞) as is the case with traditional t-tests, the MCMC method category reinstatement were assessed with an analysis of variance that
estimates all three parameters of the t distribution defined by the data compared factors of region, retrieval category, and classifier category
and provides a valid test between groups of data even when the data evidence.
is not normally distributed. As the name of the test implies, the BEST
procedure is a generalization of the traditional t-test and is a statistically Multivariate pattern similarity
superior method for evaluating means from group data (Kruschke,
2012). All BEST tests conducted in the current work consisted of The goal of the similarity analysis was to assess the extent that item-
10,000 MCMC samples with the first 1000 samples discarded for burn- level information was reinstated in different MTL subregions during
in. A BEST analysis provides a distribution of credible values of the test memory retrieval (Fig. 4). In contrast to classification techniques that
statistic summarized in the form of 95% highest density interval (HDI) are used to decode activation patterns associated with relatively small
that is akin to a 95% confidence interval from traditional t-test methods. number of stimulus classes or conditions, pattern similarity methods
Although the results of a BEST analysis are best summarized by the 95% allow one to evaluate activation patterns at the level of single events
HDI, we also calculated a P statistic that corresponds to the proportion of or stimuli. For example, instead of training a classifier to dissociate
the sampling posterior probability distribution that falls below each activation patterns for places and faces, as is the case with classification
test's chance level (i.e., P is an analog to a p value from traditional null methods, pattern similarity methods can measure the degree that an
hypothesis testing methods). Additionally, all reported P values are activation pattern associated with a single trial of retrieving the Taj
corrected to control for false discovery rate with α = 0.05. All three Mahal is similar to activation patterns associated with viewing the Taj
ROIs showed greater than chance accuracy (all P b 0.0001) at decoding Mahal versus other places. Pattern similarity methods offer a greater
the four stimulus classes during the pre-exposure phase (PHC: mean = sensitivity in detecting between individual experimental events than
32.9%, HDI = [31.5%, 34.3%]; PRC: mean = 28.7%, HDI = [27.2%, 29.7%]; is afforded by multivariate classification.
hippocampus: mean = 27.3%, HDI = [26.2%, 28.3%]). Pattern similarity analyses were implemented using PyMVPA
To further test the reliability of the pre-exposure MVPA classifiers, (Hanke et al., 2009) and custom Python routines. First, activation
we ran randomization tests on all of the classifiers to empirically define patterns for visual and memory-related responses in each of the ROIs
chance levels. To perform these tests, we reran the classification analy- for each of the eight famous faces and places were estimated using
sis, but shuffled labels on the test data in each of the cross validation an event-specific univariate general linear model (GLM) approach
folds. This analysis was repeated 1000 times for each participant to employing the least squares single (LSS) method (Mumford et al.,
create a null distribution of classifier accuracies. For each participant 2012; Xue et al., 2010). In contrast to the classification approach that
and each ROI, a p value was calculated that corresponded to the propor- leverages the variance in neural patterns to learn voxel weights that
tion of the null distribution accuracies that were greater than the best discriminate conditions, pattern similarity analyses require stable
observed accuracy. These p values were converted to z values with the estimates of neural representations for the conditions of interest. In
inverse cumulative normal distribution and evaluated across partici- the current study, the condition of interest was at the level of specific
pants with a one-sample BEST test. Both PHC and PRC classifiers showed items. Thus, we took a GLM approach to model stable estimates of visual
significantly greater accuracies than a baseline z value of 1.65 and memory-related neural patterns for each of the eight famous faces
(corresponding to a p value of 0.05; PHC: mean = 3.475, HDI = and places. For each pre-exposure run, separate GLMs were conducted
[2.562, 4.498], P b 0.0001; PRC: mean = 2.945, HDI = [1.954, 3.944], for each of the eight faces and places, such that each GLM included
P = 0.0074). In contrast, accuracy of the hippocampus classifier across one regressor for the face or place modeled as 1 s boxcar convolved
participants was not significantly greater than the randomization test with a canonical hemodynamic response function (HRF), separate
null distributions (mean = 2.166, HDI = [1.188, 2.962], P = 0.159). regressors for the remaining events collapsed according to the event
Thus, whereas classifiers trained on pre-exposure data in PHC and PRC categories (e.g., face, place, object, and fixation) that were also
accurately classified above empirically defined chance levels, hippo- convolved with a HRF, and nuisance regressors that included six head
campal classifiers did not. As such, subsequent classification analyses motion parameters. These individual GLMs resulted in voxelwise
were restricted to PHC and PRC. parameter estimates to each of the stimuli. The resulting beta values
After assessing classifier performance during the pre-exposure consisted of four visually-evoked activation patterns across the runs
phase, a SVM classifier was trained on all data from the pre-exposure for each of the eight faces and places. A similar GLM approach was con-
phase for each ROI and used to predict evidence for face and place ducted on the DMTM functional data to extract memory retrieval-based
information during the delay period of the DMTM trials. This analysis activation patterns for the faces and places. Specifically, for each DMTM
was limited to correct trials. To account for the hemodynamic lag, the functional run, separate GLMs were conducted for the individual trials
trained classifier was applied to data acquired at time points 4-7 s such that each GLM included a single regressor for the current trial
during the delay period of the DMTM trials (i.e., the dataset was time modeled as a 9 s boxcar convolved with a canonical HRF, separate re-
shifted by 4 s). This range was selected a priori to capture the peak of gressors that combined the remaining events according to category
the hemodynamic response to the delay period. Probability estimates and match condition (face match, face mismatch, place match, and
for the face and place labels were extracted from the SVM classifier place mismatch), and nuisance regressors that included six head motion
and used to index the amount of classifier evidence for the two catego- parameters. These GLMs resulted in voxelwise parameter estimates of
ries at each time point during the tested range. Each evidence estimate activation related to retrieving each of the faces and places. The
was averaged across the time points within a trial, relabeled as correct resulting beta values consisted of five memory-retrieval activation
versus incorrect category according to the retrieval category of the patterns across the runs for each of the eight faces and places. In total,
trial, and averaged across memory test trials for each participant (Fig. we estimated four visually-evoked and five retrieval-based activation
3). Finally, the difference in classifier evidence for correct versus incor- activation patterns for each of the eight faces and places.
rect category was evaluated across participants with a paired t-test The pre-exposure and DMTM test activation patterns were used to
based on the BEST procedure (Kruschke, 2012), with separate tests for characterize the degree of item-specific information reinstated within
memory test trials with face and place retrieval targets. For these the ROIs during memory retrieval (Fig. 4). Pearson correlations
particular tests, a mean paired difference was considered significant if were computed to assess the similarity between DMTM test patterns
the 95% HDI did not include zero. We have included a P statistic in the of retrieving an item and pre-exposure patterns of viewing that same
Results section that corresponds to the proportion of the sampling pos- item (e.g., retrieving Taj Mahal and viewing Taj Mahal, self similarity)
terior probability distribution of mean differences that fell below zero. or viewing other items within the same category (retrieving Taj Mahal
Finally, regional interactions between regions demonstrating significant and viewing Eiffel Tower, category similarity). Self similarity and
 M.L. Mack, A.R. Preston / NeuroImage 127 (2016) 144–157 149

category similarity values were calculated for every DMTM test trial for Functional connectivity analysis
each participant in the hippocampus, PRC, and PHC. For the hippocam-
pal regions demonstrating an item reinstatement effect, we performed We assessed the functional coupling (Rissman et al., 2004) between
follow up exploratory analyses in CA subfields (see Supplemental MTL regions showing item-level reinstatement and item-reinstatement
Materials). Correlation coefficients were Fisher transformed to more regions in OTC identified in the pattern similarity searchlight analysis
closely conform to the assumptions of normality underlying standard described in the previous section (Fig. 5). For each trial, average delay-
statistical tests. period activation within these face and place reinstatement regions, as
To assess the degree of item-level reinstatement within each ROI, we well as the MTL ROIs exhibiting item reinstatement (right hippocampus
compared average self versus category similarity across participants for and left PRC, see Supplemental Materials for right CA1), were extracted
face and scene retrieval trials separately. Only correct trials with from the event-specific univariate GLM described in the pattern similar-
response times faster than or equal to 1000 ms were included in the ity analysis section (Fig. 5A). For each participant, we estimated the
participant averages. Given the visual nature of the materials and the degree of connectivity between the MTL ROIs and OTC face and place
implementation of the analysis, which emphasizes visual similarity, reinstatement regions separately for face and place retrieval trials
we expected that right hemisphere might be more likely to demonstrate using robust regression (Fig. 5B). Connectivity differences across differ-
item reinstatement effects (Glosser et al., 1995; Golby et al., 2001). We ent retrieval conditions (see Fig. 5C) were assessed with a repeated
therefore first assessed differences in the similarity measures between measure analysis of variance of retrieval type (face vs. place) and OTC
left and right hemispheres for each ROI by conducting an analysis of reinstatement region (face vs. place). In particular, we looked for a signif-
variance that included factors of hemisphere (left vs. right), retrieval icant interaction as evidence of connectivity modulated by retrieval type.
target (face vs. place), and similarity (self vs. category). If an ROI showed We also performed paired comparisons within each MTL ROI for connec-
a significant interaction that included hemisphere, subsequent analyses tivity with OTC face and place reinstatement regions for each retrieval
analyzed the ROI separated by hemisphere; otherwise, similarity condition separately using the BEST methods described previously.
measures were collapsed across hemisphere. Pairwise comparisons
between self and category similarity were conducted with BEST t- Diffusion modeling analysis
tests. Reported BEST P values are corrected to control for false discovery
rate with an alpha = 0.05. Regional interactions between MTL regions One key theoretical claim of pattern completion is that the mnemonic
showing item reinstatement were evaluated with an analysis of content reinstated in the hippocampus serves as evidence for memory
variance that compared factors of region, retrieval category, and based decisions. We sought to evaluate this prediction by linking our
similarity level. Any item reinstatement effects found in the hippocam- pattern similarity measure of item reinstatement to participants' match/
pus were followed up with exploratory pattern similarity analyses in mismatch decisions with a computational model of decision making.
subfield CA1 and the combined subfields DG/CA2,3 (see Supplemental Specifically, we fit participants' choice behavior in the DMTM task with
Materials). variants of the drift diffusion model (Ratcliff, 1978) that were differently
Additionally, we identified regions in occipito-temporal cortices informed by the neural item reinstatement measure. By doing so, we eval-
(OTC) that showed item-level reinstatement with a searchlight analysis uated whether or not the fidelity of internally generated item reinstate-
extension of the pattern similarity approach. This analysis was ment predicts how quickly responses to the retrieved memory are made
performed as part of the connectivity analysis described in the next on a trial-by-trial basis. This analysis extends beyond correlation methods
section. Specifically, we targeted OTC regions that selectively coded for (e.g., Gordon et al., 2014) by assessing the role of item reinstatement
items from one category (face or place) more than the other to assess within a computational framework of memory-based decision making.
the functional connectivity between stimulus representation regions The DDM posits that decisions are made by sequentially sampling
and MTL regions associated with item reinstatement. We used a search- noisy evidence. Over time, this evidence accumulates towards a
light approach due to the relatively large volume of OTC. The spatial threshold. Once the threshold is reached, the response associated with
sensitivity of a searchlight approach allowed us to identify only those the threshold is made. With this conceptualization, the DDM is able to
regions across all of OTC that reinstated item information during account for both response probabilities and response time distributions.
memory retrieval. The same pattern similarity analysis performed The DDM is based on three core parameters: drift rate v – the rate of
within the anatomically-defined MTL ROIs described previously evidence accumulation, threshold a – the distance between the starting
was employed for the searchlight analysis with the following excep- point of the evidence and the decision thresholds, and non-decision
tions. For each searchlight sphere (radius = 3 voxels), an item rein- time t – the time required for perceptual encoding and motor response
statement index was calculated by subtracting the category execution. These parameters uniquely affect predictions for response
similarity from the item similarity. This item reinstatement index probabilities and response time distributions. Typical applications of
was compared to a null distribution of reinstatement indices calcu- the DDM seek to decompose effects of experimental conditions on choice
lated from randomly shuffled trial orderings. This permutation test behavior onto one or more of the DDM parameters to gain insight into
was conducted for each searchlight sphere resulting in a probability the latent mechanisms of decision making (Ratcliff and Rouder, 1998).
map. Here, we used the DDM to investigate if and how the neural measure
The searchlight analysis was conducted separately for face and place of item reinstatement predicts, for every trial, response choice and speed.
item reinstatement measures and was restricted to occipito-temporal The DDM analysis was implemented with the Hierarchical Drift
cortex as defined by Freesurfer anatomical parcellations. The Diffusion Model toolbox (HDDM version 0.5.3; Wiecki et al., 2013).
resulting probability maps were converted to z-scores. For group HDDM performs hierarchical Bayesian parameter estimation for
analysis, the z-scored maps were normalized to MNI space and the DDM using Markov-Chain Monte Carlo (MCMC) sampling methods.
voxelwise nonparametric permutation testing (5000 permutations) By doing so, the HDDM allows for simultaneous estimation of participant
was performed using FSL Randomise (Winkler et al., 2014) to compare and group parameters such that participant parameters are sampled
face and place item reinstatement. The resulting statistical maps were from group distributions, multiple parameters to be estimated for differ-
voxelwise thresholded at p = 0.01 and cluster corrected at p = 0.05 ent conditions, and the estimation of trial-by-trial effects from external
which corresponded to a cluster extent threshold of greater than 29 variables (e.g., neural measures) on different DDM parameters.
voxels as determined by AFNI 3dClustSim (estimated smoothness of We fit a family of DDM variants to participant behavior, with different
FWHM =6.27). Significant face and place item reinstatement clusters model variants representing different hypotheses about how neural item
are reported in Table 1 and were used to create ROI masks for the reinstatement influences decision making (Fig. 6). First, we calculated a
connectivity analysis described next (Fig. 5A). trial-by-trial item reinstatement score by subtracting the category from
150 M.L. Mack, A.R. Preston / NeuroImage 127 (2016) 144–157

the item similarity values as computed in the neural pattern similarity effect of retrieval category (F1,22 = 4.34, p = 0.049, η2p = 0.165), but
analyses described previously. To remove the potential confound of no effect of probe match (F1,22 = 0.576, p = 0.456, η2p = 0.026) nor an
trial-by-trial BOLD activation on choice behavior within each ROI, overall interaction (F1,22 = 0.167, p = 0.687, η2p = 0.008). A similar analysis
BOLD response within each ROI was computed for each trial and of variance conducted on response times revealed significant main
regressed out of the trial-by-trial item reinstatement score. We then effects of probe match (F1,22 = 15.793, p = 0.0006, η2p = 0.418) and
used this modified item reinstatement score in the DDM analyses. Specif- retrieval category (F1,22 = 78.661, p = 1 × 10−8, η2p = 0.781), but no
ically, in separate models, we estimated the relationship between trial- interaction (F1,22 = 0.197, p = 0.661, η2p = 0.009). Thus, participants
by-trial item reinstatement in MTL regions showing evidence for item re- were overall more accurate and faster in responding to associative
instatement, as well as OTC regions identified in the searchlight analysis, pairs that included faces relative to places, as well as faster to respond
and the three core DDM parameters. This relationship was modeled as a when the probe item matched relative to mismatched the face or
linear effect between item reinstatement and a DDM parameter. Thus, place paired with the cue object.
the result of one model fit was two regression coefficients that describe
the linear relationship between the neural measure of item reinstate- Category reinstatement in the MTL
ment and the DDM parameter for match and mismatch trials.
For example, fitting a model with item reinstatement linked to drift We first targeted evidence of category reinstatement with a
rate (Fig. 6A) results in linear regression coefficients that characterize commonly-used decoding approach (e.g., Kuhl and Chun, 2014;
the effect of item reinstatement on drift rate separately for match and Zeithamova et al., 2012). Separate classifiers were trained on fMRI
mismatch trials. Positive valued coefficients would indicate that a data from the pre-exposure phase for each bilateral MTL ROI (PHC,
higher item reinstatement is associated with a higher drift rate. PRC, and hippocampus) to dissociate neural patterns for the four
Negative valued coefficients would indicate that higher item reinstate- stimulus categories: objects, faces, places, and fixation. Only classifiers
ment is associated with a lower drift rate. Because parameter estimation trained on PHC and PRC pre-exposure data showed above chance
in the HDDM is implemented in a hierarchical Bayesian framework with classification. Hippocampus did not differentiate category information
MCMC sampling, a distribution of regression coefficients is estimated during pre-exposure, thus this region was excluded from subsequent
for any one model fit (Fig. 6B). To assess the linear effect of item classification analyses for the DMTM test phase. The trained PHC and
reinstatement on the model parameters, we calculated a P statistic PRC classifiers were then applied to data recorded during the DMTM
that quantified the proportion of the regression coefficient distribution test phase to predict evidence for face and place information during
that was less than zero. P values less than 0.05 were considered the delay period of the DMTM trials (Fig. 3A). This analysis was
statistically significant. Separate models were fit to behavior associated restricted to correct DMTM trials. There was a difference in category re-
with face and place retrieval trials, and all three DDM parameters (v, a, instatement across MTL cortex as revealed by an analysis of variance
t) were free to vary across match and mismatch conditions. that showed a significant interaction of region × stimulus category ×
All models were estimated with 30,000 MCMC samples with the retrieval category (F1,22 = 29.18, p = 0.00002, η2p = 0.57). Within PRC
first 10,000 samples discarded for burn-in. We assessed model fits (Fig. 3B), we found category-specific evidence of face reinstatement
with two analyses. First, model convergence was evaluated with the (μdiff = 0.0162, 95% HDI = [0.0051, 0.0281], P = 0.002), but not place
Gelman-Rubin R-hat statistic computed over three independently esti- reinstatement (μdiff = 0.00158, 95% HDI = [− 0.012, 0.015], P =
mated chains of each model. All parameters in all models had a R-hat 0.41). Within PHC, we found evidence of place reinstatement (μdiff =
less than 1.1, suggesting the chains of MCMC sampling reached 0.0414, 95% HDI = [0.0197, 0.0653], P b 0.0001), and a trending effect
convergence. Second, we performed model comparison by calculating for face reinstatement (μdiff = 0.011, 95% HDI = [− 0.0014, 0.024],
a deviance information criterion (DIC) measure for each model. Critical- P = 0.076).
ly, we compared the item reinstatement models to baseline DDM
models that did not include effects of item reinstatement. An item Item reinstatement in the MTL
reinstatement model was considered as representing a meaningful rela-
tionship between behavior and item reinstatement if the model provid- We next targeted the specific item-level contents of memory by
ed a better fit (smaller DIC value) than the corresponding baseline comparing memory-based activation patterns within the hippocampus
model (Table 2). If more than one item reinstatement model provided and surrounding cortex to visually-evoked activation patterns with
a better fit than the baseline model, the item reinstatement model neural pattern similarity analyses. Specifically, the degree of item-
with the smaller DIC value was selected as the winning model. specific reinstatement was measured by correlating activation patterns
present during retrieval of a specific face or place in DMTM trials with
Results the corresponding activation patterns present when viewing that face
or place prior to associative learning (Fig. 4A). This self similarity
Behavioral performance measure was compared to a category similarity measure defined as the

For the pair encoding phase, learning performance was defined as match mismatch
the average number of correct responses during each test block. A 1.0 700
Correct RT (ms)

criterion of 85% accuracy was used to determine when to move on to

the DMTM phase. Of the 23 participants, 10 reached the learning 0.9 650
P(correct)

criterion after one study-test repetition, 11 after two study-test 0.8 600
repetitions, and 2 after three study-test repetitions.
For the DMTM phase, memory performance in each of the 0.7 550
four conditions (face match, face mismatch, place match, and place 0.6 500
mismatch) was measured with the mean probability of a correct
response and the median response times for correct responses (Fig. 2). 0.5 450
face place face place
Trials were excluded from all analyses if no response was provided
(1.1% of all trials) or if the trial response time was greater than
Fig. 2. Behavioral performance in the delayed match-to-memory (DMTM) test. Average
1000 ms (5.1% of all trials). An analysis of variance of probability of probability correct (left plot) and median correct response times (right plot) are depicted
correct responses that included the factors of probe match (match vs. for face and place retrieval trials where the probe was either a match (dark) or mismatch
mismatch) and retrieval category (face vs. place) revealed a significant (light). Error bars represent 95% confidence intervals of the mean.
 M.L. Mack, A.R. Preston / NeuroImage 127 (2016) 144–157 151

correlation between memory retrieval patterns for the faces and places faces (μdiff = 0.0044, 95% HDI = [−0.006, 0.0143], P = 0.184). Left hip-
and pre-exposure patterns for other items within the same stimulus pocampus showed no significant item reinstatement effects (p N 0.16).
class. In MTL cortex, only left PRC showed evidence of item-specific reinstate-
First, we assessed differences in the similarity measures between left ment as revealed by an interaction of similarity and retrieval target
and right hemispheres for each ROI by conducting an analysis of (F1,22 = 5.41, p = 0.029, η2p = 0.184). Paired BEST comparisons indicated
variance that included factors of hemisphere (left vs. right), retrieval significant reinstatement for faces (μdiff = 0.0099, 95% HDI = [0.0002,
target (face vs. place), and similarity (self vs. category). Given the visual 0.02], P = 0.022), but not places (μdiff = −0.001, 95% HDI = [−0.0112,
nature of the experimental materials and the established laterality 0.0092], P = 0.86). An analysis of variance assessing regional differences
effects for stimulus modality in MTL (Glosser et al., 1995; Golby et al., in reinstatement between right hippocampus and left PRC showed no
2001), we expected differences across hemispheres. If an ROI showed significant interactions with region (all Fs b 1). Right PRC and bilateral
a significant interaction that included hemisphere, subsequent analyses PHC did not show evidence of item reinstatement (all Fs b 1). See the
analyzed the ROI separated by hemisphere; otherwise, similarity Supplemental Materials for an exploratory item reinstatement analysis
measures were collapsed across hemisphere. Both PRC and hippocam- of CA subfields in right hippocampus.
pus showed significant interactions with hemisphere (PRC: hemisphere
× target × similarity - F1,22 = 9.87, p = 0.0044, η2p = 0.291; hippocam- MTL connectivity with item reinstatement regions in occipito-temporal
pus: hemisphere × target - F1,22 = 7.42, p = 0.0012, η2p = 0.25), thus cortex
corresponding ROIs in left and right hemispheres for these regions
were analyzed separately. We next examined the key theoretical component of pattern com-
Within right hippocampus (Fig. 4C), we found evidence of item- pletion that the hippocampus guides reinstatement in behaviorally-
specific reinstatement as indicated by a main effect of item versus relevant cortical regions (Marr, 1971). We used a functional connectiv-
category similarity (F1,22 = 6.99, p = 0.015, η2p = 0.241). Paired BEST ity approach (Rissman et al., 2004) to examine the coupling between
comparisons revealed this main effect was driven by item reinstatement hippocampal processes and task-related cortical regions during retriev-
for places (μdiff = 0.0085, 95% HDI = [0.0002, 0.017], P = 0.024), but not al. We identified regions throughout occipito-temporal cortex (OTC)

A
DMTM test trial

classifier
pattern evidence
+
classifier correct

incorrect

prediction

B face retrieval
place retrieval
category reinstatement

0.30 correct category

*
classifier evidence

incorrect category
0.28
*
0.26

0.24

0.22

0.20
PHC PRC
Fig. 3. Schematic of category-level reinstatement decoding analysis and results in PHC and PRC. (A) For each ROI, a linear SVM pattern classifier was first trained on activation patterns
acquired during the pre-exposure phase. The classifier was trained to discriminate patterns associated with faces and places and then applied to the data acquired during the delay period
of the DMTM trials. Probability estimates for the face and place labels were extracted from the SVM classifier and used to index the amount of classifier evidence for the two categories.
Finally, the evidence was relabeled as correct versus incorrect category according to the retrieval category of the trial (in the depicted example, scene would be the correct category).
(B) Category-specific reinstatement was observed in MTL cortex, with PRC showing significantly more face than place evidence during face retrieval and PHC showing significantly
more place than face evidence during place retrieval. The hippocampus showed no evidence for category reinstatement (see Methods). Error bars represent 95% high density intervals
of the paired BEST comparisons between correct and incorrect category classifier evidence. Significant paired differences (p b 0.05) are noted with asterisks (*).
152 M.L. Mack, A.R. Preston / NeuroImage 127 (2016) 144–157

A B pre-exposure DMTM test trial

...
rcategory
+

rself

face retrieval
C place retrieval
0.03
* * self similarity
category similarity
reinstatement (r)

0.02

0.01

0.00

-0.01 left right bilateral left right

hippocampus PHC PRC

Fig. 4. Schematic of item reinstatement neural pattern similarity analyses and results for each of the ROIs. (A) Within each ROI (perirhinal cortex is depicted in the figure and inset), the
pattern of neural activation was extracted for every trial, here represented as a matrix grid. (B) Item-level reinstatement was measured by calculating the similarity between activation
patterns during DMTM trials for specific faces and places and activation patterns of viewing the same face or place during the pre-exposure phase (rself) relative to activation patterns
of viewing other within-category items (rcategory). (C) Item reinstatement similarity analysis results. For each ROI, self (darker bars) versus category (lighter bars) similarity is plotted
for face (blue) and place (green) retrieval trials. Both hippocampus and PRC showed significant differences across hemispheres, thus left and right ROIs within these regions were
considered separately. Right hippocampus and left PRC showed item-level reinstatement for place and face retrieval, respectively. Error bars represent 95% confidence intervals of the
means. Asterisks (*) note significant paired comparisons of self greater than category similarity corrected for false discovery rate (α = 0.05).

that exhibited item-specific reinstatement exclusively for faces or places Modeling the influence of item reinstatement on decision making
with a searchlight analysis extension of the item reinstatement analysis
(Table 1 and Fig. 5A). Average delay-period activation within these face We next sought to assess the behavioral significance of reinstated
and place reinstatement regions, as well as right hippocampus and left hippocampal and PRC patterns, specifically testing the hypothesis that
PRC, were extracted for each correct trial. We estimated the degree of reinstated episodic information in hippocampal and PRC patterns serves
correlation between the MTL ROIs and face/place reinstatement regions as the critical evidence for memory-based decision making. We fit
separately for correct face and place retrieval trials (Fig. 5B). We found participants' choice behavior to the memory probe with the drift diffu-
that the connectivity pattern between right hippocampus and face sion model (DDM). The DDM simultaneously accounts for response
and place reinstatement regions (Fig. 5C) was modulated by the type choices and times across different experimental conditions by opera-
of retrieval such that there was greater coupling with face regions tionally defining decision making as an evidence accumulation process
during face retrieval and place regions during place retrieval (F1,22 = (Ratcliff, 1978). We linked the neural signatures of item reinstatement
11.56, p = 0.0026, η2p = 0.344; see Supplemental Materials for explor- in right hippocampus and left PRC to the computational mechanisms
atory analysis with right CA1). In contrast, left PRC connectivity was of decision making in the DDM (Fig. 6A and Table 2).
not modulated by retrieval (F1,22 = 0.038, p = 0.85, η2p = 0.002).

Table 2
Drift diffusion model (DDM) results. Separate models that included trial-by-trial effects of
Table 1 item reinstatement from the three regions that showed significant item reinstatement in
Item reinstatement regions in occipito-temporal cortices. Regions were identified from the pattern similarity analysis (right hippocampus and left PRC) on the three core DDM
the paired comparison of face versus place reinstatement with a voxelwise threshold of parameters (drift rate v, threshold a, and nondecision time t) were fit and compared to
p = 0.01 and cluster extent threshold of p = 0.05 (extent greater than 29 voxels). In the baseline models with no item reinstatement effects (top row). In the table, models are
table, clusters that survived correction are described by their corresponding region labeled according to their reinstatement-parameter relationship (e.g., the v ~ rhipp model
(region), the number of voxels in the cluster (voxels), the t value of the peak voxel in included an effect of hippocampal item reinstatement on drift rate v). Behavior from face
the cluster (t value), and the location of the peak voxel in MNI coordinates (location). (left two columns) and place (right two columns) retrieval conditions were fit in separate
models. Rows in bold text note item reinstatement models with a significantly better
region voxels t value location (x, y, z) account of the behavior data than the corresponding baseline models.
face regions
Face retrieval trials DIC Place retrieval trials DIC
left lingual gyrus 87 3.49 −24, −54, −2
right middle temporal 49 2.84 62, −8, −16 Baseline model −843.09 Baseline model −400.12
left temporal pole 49 3.24 −50, 6, −40 v ~ rhipp −842.47 v ~ rhipp −406.32
scene regions a ~ rhipp −844.74 a ~ rhipp −387.15
left lateral occipital 727 5.44 −16, −94, 8 t ~ rhipp −830.79 t ~ rhipp −399.13
right lateral occipital 566 3.77 36, −76, 6 v ~ rPRC −853.02 v ~ rPRC −393.14
89 3.6 44, −80, −10 a ~ rPRC −844.25 a ~ rPRC −400.79
left ventral temporal 143 3.31 −44, −22, −18 t ~ rPRC −848.36 t ~ rPRC −399.93
 M.L. Mack, A.R. Preston / NeuroImage 127 (2016) 144–157 153

A B
bold(place regions) ~ β × bold(hipp)? place retrieval β = 0.34

bold signal
hipp
face retrieval β = -0.12

trials
C
*
connectivity (β)

0.3 face retrieval

* place retrieval
face regions
0.2
place regions

0.1

0.0
right hipp left PRC
Fig. 5. Schematic of the functional connectivity analysis and the results for right hippocampus and left PRC. (A) Trial-by-trial average activation was extracted from the three MTL ROIs
(hippocampus is shown in red on the right) and regions within occipito-temporal cortex (OTC) that exhibited item-reinstatement signatures exclusively for faces or places (face regions
shown in dark gray, place regions in white; see Table 1 for a list of all OTC item reinstatement regions). (B) Functional connectivity between the three MTL ROIs and OTC reinstatement
regions was assessed with robust regression for face and place retrieval trials separately. In the depicted example, average activation within the right hippocampus (red line) correlates
more with average activation in OTC place reinstatement regions (dotted line) during place (top plot) than face retrieval trials (bottom plot). (C) Results for each MTL ROI are plotted in
separate graphs that depict average connectivity to OTC regions exhibiting face (darker colors) and place (lighter colors) reinstatement during face (blue) and place (green) retrieval trials.
Error bars represent the 95% confidence intervals of the interaction between retrieval type and OTC region. Significant interactions are noted with a tensor symbol, and significant pairwise
BEST comparisons with an asterisk.

Specifically, on a trial-by-trial basis, we estimated in separate models Although accuracy was near ceiling in the DMTM task, thereby limiting
the degree that item-specific neural reinstatement predicted evidence the DDM models to accounting mostly for differences in response time
accumulation, decision threshold, or non-decision time for match and distributions, these results suggest a trial-by-trial link between the
mismatch probe decisions in the DMTM task using Bayesian parameter fidelity of reinstated memories in the MTL and decisions about the
estimation (Wiecki et al., 2013). We found that item-specific reinstate- specific contents of those memories.
ment in right hippocampus significantly predicted the rate of evidence
accumulation to matching place probes on a trial-by-trial basis (baseline Discussion
model: DIC = −400.12; hippocampus model: DIC = −406.32, Pmatch =
0.021; Fig. 6B). The relationship between hippocampal reinstatement Taken together, our results suggest that the specific contents of
and evidence accumulation was such that reinstating more place- episodic memories are reinstated in the activation patterns of hippo-
specific information led to faster evidence accumulation, resulting in campal subfields and subregions of MTL cortex. In particular, our
faster response times. This relationship was not present for findings that reinstated neural patterns in the hippocampus contain
mismatching place probes (Pmismatchs N 0.4). Item-specific reinstatement item-level information are consistent with computational theories
in PRC also showed a relationship to evidence accumulation, with that point specifically to CA3 and CA1 as critical for reinstating complete
greater reinstatement of face-specific information leading to faster memory representations from partial sensory input through pattern
accumulation as a general effect across both match and mismatch completion (Marr, 1971; Norman and O'Reilly, 2003; Rolls, 2013).
probes (baseline model: DIC = −843.09; PRC model: DIC = −852.02, Importantly, our study extends beyond the existing evidence for
P = 0.024). See the Supplemental Materials for additional analyses of hippocampal pattern completion from rodent and human studies that
the relationship between hippocampal reinstatement and response rely on representations of spatial locations (Kyle et al., 2015; Leutgeb
times (Figure S1), the generative nature of the neurally-informed et al., 2007; Miller et al., 2013; Neunuebel and Knierim, 2014; Stokes
DDM results (Figure S2), and an exploratory analysis with right CA1. et al., 2015). In particular, such work has been limited in demonstrating
We also tested item reinstatement in the OTC regions identified in the link between hippocampal-based spatial codes and choice behavior.
the searchlight analysis (see Table 1) using the DDM framework. How- It has been shown that single unit place cell responses can predict
ever, none of models that included trial-by-trial reinstatement from direction choices made by rodents in maze tasks (Pastalkova et al.,
OTC place reinstatement regions fit better than the baseline scene 2008; Singer et al., 2013); however, these findings are not based on pat-
model (DICs N −395). One model including reinstatement from a face tern completion processes per se. One recent rodent study measured
region in left lingual gyrus as a factor on drift rate did fit slightly better both CA3 input from DG and CA3 output, finding direct evidence for pat-
than the face baseline model (DIC = − 845.44); however, the tern completion processes in CA3, but without a link to behavior
regression coefficient between item reinstatement and drift rate was (Neunuebel and Knierim, 2014). Finally, none of this research has
not significant (Ps N 0.2). All other models including OTC face reinstate- indexed specific memory content beyond spatial locations (i.e., place
ment regions did not fit better than the baseline model (DICs N −843). fields). By linking a continuous neural measure of memory to a
154 M.L. Mack, A.R. Preston / NeuroImage 127 (2016) 144–157

A pre-exposure DMTM test trials

v ~ (rs-rc)
+

rs “match”
rs
pair learning
+ v ~ (r -rc)

response time “match”

B right hippocampus left PRC

posterior probability

p = 0.021 1.2 p=0.024

0.6
match mismatch
density

0.4 0.8 face

place
0.2 0.4

0.0 0.0
0 2 4 0 1 2
item reinstatement regression coefficient for drift rate
Fig. 6. Schematic of the drift diffusion modeling (DDM) analysis and the results for right hippocampus, right CA1, and left PRC. (A) An item-level reinstatement index was calculated on a
trial-by-trial basis by taking the difference between the self and category similarity on each trial (rs–rc). This trial-by-trial item reinstatement index was then entered in a DDM analysis of
response choices and times to the probe stimulus. Specifically, the item reinstatement index was modeled as a linear effect on the drift rate (v) parameter of the DDM, thereby influencing
the rate of evidence accumulation. For example, the top trial depicts a situation where reinstatement of the place associated with the boot, Taj Mahal, is high. This results in a faster ac-
cumulation of evidence and a shorter response time. When item reinstatement is low, as depicted in the bottom trial, evidence accumulates gradually and the response time is longer.
(B) The plotted distributions represent the Bayesian posterior densities of the interaction of drift rate and item reinstatement for right hippocampus (left), right CA1 (middle) and left
PRC (right). Higher item reinstatement in right hippocampus and CA1 increased drift rates on place match trials (solid green), but had no effect on place mismatch trials (dotted
green). Item reinstatement in left PRC showed a main effect of facilitation on drift rates for both match and mismatch face trials (blue lines).

prominent mathematical model of decision making, our results provide In contrast to electrophysiological studies, fMRI-based findings in
support for theories of hippocampal function that link retrieval of spe- support of memory reinstatement in the hippocampus are considerably
cific memory elements to reinstatement of hippocampal representa- lacking. One recent fMRI study found evidence for reinstatement of spe-
tions that themselves are critical for subsequent decisions (Lisman cific memories in hippocampus (Wimber et al., 2015). However, that
and Grace, 2005; Marr, 1971; Norman and O'Reilly, 2003; O'Reilly and finding does not address the critical questions set forth in the current
Rudy, 2001; Rolls, 2013). study: reinstatement of specific memory contents and their impact on
Evidence of memory reinstatement in human hippocampus has decisions. In particular, the fact that reinstatement in that study was
been limited to electrophysiological studies of epilepsy patients measured by comparing retrieval patterns to episode-specific templates
(e.g., Gelbard-Sagiv et al., 2008; Paz et al., 2010; Rutishauser et al., acquired after both encoding and retrieval necessarily reflects retrieval
2015). By recording single cells within the hippocampus, it has been of entire experiences rather than individual event elements. Thus, sim-
demonstrated that a subset of cells show item-specific selectivity during ilar to other studies (Chadwick et al., 2010, 2014), the Wimber et al.
visual presentation (Quiroga et al., 2005, 2007), and that selective firing (2015) study measured the distinctiveness of individual composite
for specific episodes during memory encoding reoccurred during free events during retrieval, but not retrieval of the individual memory con-
recall of the same episodes (Gelbard-Sagiv et al., 2008). Furthermore, tents themselves, which is the critical theoretical prediction of pattern
the strength of episodic encoding in single hippocampal cells, measured completion. More importantly, they failed to link episode-specific hip-
as a correlation in temporal patterns of encoding-related firing rates, pocampal patterns to behavior, which is the second key theoretical
has been shown to predict subsequent recall performance (Paz et al., test. In the current study, by comparing reinstated memory representa-
2010). Single cell firing patterns in the hippocampus have also been tions to visual representations recorded before associative learning, we
tied to memory decisions during retrieval, with evidence that a subset provide fundamental evidence that hippocampus and perirhinal cortex
of hippocampal cells fire proportionally to graded responses of memory uniquely reinstate the specific contents of episodic memories. We not
strength and confidence (Rutishauser et al., 2015). This work offers only relate this reinstatement to behavior, but also use reinstatement
compelling demonstrations of hippocampal memory reinstatement to make trial-by-trial predictions about decisions in a generative
with links to behavior, yet such findings are limited to characterizing fashion.
the reinstatement signatures of single cells. Moreover, although this We observed behaviorally relevant content-specific reinstatement
work suggest some hippocampal cells represent memory information in the hippocampus and PRC. This finding is consistent with proposals
that is correlated with decision making, the existing findings are limited that the hippocampus, PRC, and PHC play unique roles in encoding
to a general level of memory information and cannot speak to item- and retrieval depending on the content of memories (e.g., Bird and
specific pattern completion. Burgess, 2008; Diana et al., 2007). For example, the observation that
 M.L. Mack, A.R. Preston / NeuroImage 127 (2016) 144–157 155

reinstatement of specific faces in PRC influenced memory decisions con- employing optogenetics to examine functional coupling between
verges with prior studies of lesion patients that suggest a selective role hippocampus and behaviorally-relevant cortical regions. In one such
for PRC in processing faces (Mundy et al., 2013). Similar research has study, cortical neurons active during fear conditioning failed to reacti-
also implicated the hippocampus in spatial processing with hippocam- vate when CA1 neurons were silenced during retrieval (Tanaka et al.,
pal lesions leading to deficits in place but not face processing (Bird 2014). Moreover, another study found that CA1 neurons exhibited
and Burgess, 2008). Such a functional role for hippocampus is consistent distinct projections to different target brain areas depending on the
with our findings of item-reinstatement in the hippocampus guiding type of task performed by the animal (Ciocchi et al., 2015). One recent
memory decisions for places. human fMRI study examined the connectivity between the hippocam-
PHC is also known to be content-selective, with a preference for pus and MTL cortex with dynamic causal modeling, finding that
processing places (Diana et al., 2007). A recent study with a pattern hippocampal activation mediates successful retrieval of associative
similarity approach similar to the current study found evidence of information from content-specific regions of MTL (Staresina et al.,
episode-specific reinstatement of place images in PHC, but not the 2013). The hippocampus has also been implicated in several fMRI stud-
hippocampus (Staresina et al., 2012). However, PHC is known to code ies examining cortical reinstatement. Hippocampal activation during
for scene subcategories (Walther et al., 2009) and the restricted set of both encoding and retrieval scales with task (Leiker and Johnson,
four place images each representative of disparate place categories 2015), category (Gordon et al., 2014; Horner et al., 2015) and item
(home, office, city, and nature) used in this work suggests these (Ritchey et al., 2012; Wing et al., 2015) reinstatement signatures in neo-
findings could be due to reinstatement of place subcategory rather cortex. In particular, Ritchey et al. (2012) found a relationship between
than episode-specific information. Such an interpretation is consis- hippocampal univariate activation and the degree of encoding-retrieval
tent with our findings of category, but not item reinstatement in pattern similarity in cortical regions, consistent with the notion that
PHC. Although the current study is limited to eight place images, hippocampus mediates memory reinstatement in cortex. However,
we selected famous landmarks that were not representative of because the same visual input was present at both encoding and retriev-
separate place subcategories to better target item rather that catego- al, it is unknown whether their findings are related to reinstatement
ry representations. processes or to perceptual processes acting on the same visual input
The relationship between the item-level reinstatement and during both encoding and retrieval.
category-level decoding findings within individual regions also has The current findings build upon and extend these previous results by
important implications for the nature of memory representations in demonstrating that during retrieval, hippocampus is preferentially
each region. Significant category-level reinstatement in PRC and PHC coupled with OTC regions that reinstate item-level representations of
suggests MTL cortical representations are categorically organized such memory content. It should be noted that the current connectivity find-
that items from the same category are represented in similar parts of ings are limited by their correlative nature; specifically, the task-
the network, resulting in functional memory modules (Diana et al., dependent correlation between hippocampal and OTC neural activity
2008; Huffman and Stark, 2014; LaRocque et al., 2013; Liang et al., cannot be used to infer the existence or directionality of a causal
2013). This finding stands in clear contrast to the lack of category relationship between the brain regions (Friston, 2011). However, the
reinstatement in the hippocampus. The failure of prior decoding current findings, together with the existing literature, are consistent
approaches to identify hippocampal reinstatement may be due to the with the theoretical proposal that the hippocampus plays an important
relatively category-agnostic nature of the hippocampus such that role in the re-experiencing of mnemonic content through cortical
items from the same category are uniquely represented (Huffman and reinstatement and may route behaviorally-relevant information to
Stark, 2014; Kuhl and Chun, 2014; LaRocque et al., 2013; Liang et al., task-specific cortical areas during memory retrieval.
2013). It is worth noting that although the current work has focused on
Hippocampal coding is generally thought to be sparse in nature, a reinstatement as arising from pattern completion, pattern separation
theoretical position with much empirical support (Quiroga et al., is also functionally tied to memory reinstatement. In particular, overlap-
2008; Waydo et al., 2006). Although several fMRI-based studies, in ping experiences that share content must be encoded in orthogonal
addition to the current study, have successfully measured episodic- representations in order to retrieve distinct memory traces of these
specific neural representations in the hippocampus (Bonnici et al., experiences (Marr, 1971; Norman and O'Reilly, 2003; O'Reilly and
2012; Chadwick et al., 2010, 2014; Hassabis et al., 2009; Wimber et al., Rudy, 2001). This separation process is driven by competition between
2015), the sparse representations of the hippocampus are a high meth- incoming sensory information from the current experience and
odological hurdle for BOLD-based empirical investigations and may reinstated information from related previously encoded memories
have been a limiting factor in previous attempts to uncover item- (Hulbert and Norman, 2014; Norman and O'Reilly, 2003; O'Reilly and
specific reinstatement signatures in the hippocampus. An additional Rudy, 2001). The current study was designed to target reinstatement
factor to consider is that our study included relatively more associative signatures arising from pattern completion processes. The object cues
pair training (average of 3.5 presentations of each pair during training during the DMTM task provided partial information for retrieving the
across participants) compared to previous studies potentially leading correctly paired face or place associate through pattern completion. It
to stronger representations. It may be that more training during is likely that pattern separation was involved both during encoding of
encoding is required to observe BOLD-based signatures of hippocampal the paired associates and potentially influenced what information was
item reinstatement. However, the extensive associative pair training available during retrieval (Kyle et al., 2015). However, the specificity
and the intentional nature of the encoding demands during training is of the item reinstatement signatures we observed during retrieval is
a limitation of the current study; such strong associative representa- consistent with a “filling in” of item-specific mnemonic information
tions may have been observable with fMRI methods, but may involve through pattern completion. Future studies could extend these findings
more learning related rather than episodic memory related processes. by parametrically varying the degradation or similarity between a cue
Nonetheless, the findings in the current study suggest that whereas a and the associative memory (e.g., Bakker et al., 2008; Kirwan and
hierarchy of category and item-specific information is represented in Stark, 2007) to characterize how degraded cues lead to reinstatement
PRC, hippocampal representations code for the specific contents of of item specific information through pattern completion.
memories in a non-modular fashion. The present findings provide a critical demonstration for the
By targeting MTL and OTC regions that exhibit item-specific theorized role of reinstated hippocampal representations in mnemonic
reinstatement, we were able to uncover a hippocampal-OTC network decision making (Lisman and Grace, 2005; Norman and O'Reilly, 2003),
that demonstrated connectivity patterns that are modulated by current while further revealing important information about the organization of
retrieval demands. This finding is consistent with recent rodent studies the underlying mnemonic codes. Theoretical accounts posit that
156 M.L. Mack, A.R. Preston / NeuroImage 127 (2016) 144–157

reinstated representations in CA3 and CA1 act as predictions about Bird, C.M., Burgess, N., 2008. The hippocampus and memory: insights from spatial
processing. Nat. Rev. Neurosci. 9, 182–194. http://dx.doi.org/10.1038/nrn2335.
present events derived from past knowledge. Limited support for this Bogacz, R., Wagenmakers, E.J., Forstmann, B.U., Nieuwenhuis, S., 2010. The neural basis of
proposal is found in existing neuroimaging work that suggests the the speed-accuracy tradeoff. Trends Neurosci. 33, 10–16. http://dx.doi.org/10.1016/j.
magnitude of hippocampal activation during retrieval is correlated tins.2009.09.002.
Bollimunta, A., Totten, D., Ditterich, J., 2012. Neural dynamics of choice: single-trial anal-
with the speed of memory decisions (Gordon et al., 2014) and ysis of decision-related activity in parietal cortex. J. Neurosci. 32, 12684–12701.
confidence judgments (Leiker and Johnson, 2015; Thakral et al., 2015). http://dx.doi.org/10.1523/JNEUROSCI.5752-11.2012.
In the current study, we provided a direct test of the role of the hippo- Bonnici, H.M., Chadwick, M.J., Kumaran, D., Hassabis, D., Weiskopf, N., Maguire, E.A, 2012.
Multi-voxel pattern analysis in human hippocampal subfields. Front. Hum. Neurosci.
campus in memory retrieval by linking trial-by-trial item reinstatement 6, 1–13. http://dx.doi.org/10.3389/fnhum.2012.00290.
signatures to the accumulation of evidence in a computational model of Bosch, S.E., Jehee, J.F.M., Fernández, G., Doeller, C.F., 2014. Reinstatement of associative
memory-based decisions. Indeed, the computational modeling analysis memories in early visual cortex is signaled by the hippocampus. J. Neurosci. 34,
7493–7500. http://dx.doi.org/10.1523/JNEUROSCI.0805-14.2014.
showed that the fidelity of reinstated item-level representations is
Chadwick, M.J., Hassabis, D., Weiskopf, N., Maguire, E.A., 2010. Decoding individual epi-
directly predictive of the speed of memory decisions. The relationship sodic memory traces in the human hippocampus. Curr. Biol. 1–4.
between reinstatement and behavior was specific to MTL-based item Chadwick, M.J., Hassabis, D., Maguire, E.A., 2011. Decoding overlapping memories in the
reinstatement; reinstatement in occipito-temporal regions was not medial temporal lobes using high-resolution fMRI. Learn. Mem. 8, 742–746. http://
dx.doi.org/10.1101/lm.023671.111.
predictive of mnemonic decisions. Mechanistically, these findings are Chadwick, M.J., Bonnici, H.M., Maguire, E.A., 2014. CA3 size predicts the precision of
consistent with a memory-based decision process whereby the memory memory recall. Proc. Natl. Acad. Sci. 111, 10720–10725. http://dx.doi.org/10.1073/
evidence internally generated during retrieval in the hippocampus and pnas.1319641111.
Chen, J., Olsen, R.K., Preston, A.R., Glover, G.H., Wagner, A.D., 2011. Associative retrieval pro-
perirhinal cortex is compared to the sensory input of the probe stimulus. cesses in the human medial temporal lobe: hippocampal retrieval success and CA1 mis-
The fidelity of the memory evidence influences this comparison such match detection. Learn. Mem. 18, 523–528. http://dx.doi.org/10.1101/lm.2135211.
that more evidence leads to fast accumulation of decision evidence Ciocchi, S., Passecker, J., Malagon-Vina, H., Mikus, N., Klausberger, T., 2015. Selective
information routing by ventral hippocampal CA1 projection neurons. Science 348,
towards a match/mismatch response. 560–563.
It is interesting to note that although the DDM has been leveraged Diana, R.A., Yonelinas, A.P., Ranganath, C., 2007. Imaging recollection and familiarity in the
more recently as a model of perceptual decision making (Ratcliff and medial temporal lobe: a three-component model. Trends Cogn. Sci. 11, 379–386.
http://dx.doi.org/10.1016/j.tics.2007.08.001.
Rouder, 1998), its original conception was an account of memory
Diana, R.A., Yonelinas, A.P., Ranganath, C., 2008. High-resolution multi-voxel pattern
retrieval that linked internally-generated information to the accumula- analysis of category selectivity in the medial temporal lobes. Hippocampus 18,
tion of evidence for memory-based decisions (Ratcliff, 1978). Our 536–541. http://dx.doi.org/10.1002/hipo.20433.
Eichenbaum, H., 2004. Hippocampus: cognitive processes and neural representations that
results bridge brain measures and behavior to ground this classic
underlie declarative memory. Neuron 44, 109–120. http://dx.doi.org/10.1016/j.
theory's mechanisms in neural evidence and show that internally neuron.2004.08.028.
generated content from past experiences guides decision making. Fischl, B., Salat, D.H., Busa, E., Albert, M., Dieterich, M., Haselgrove, C., Van Der Kouwe, A.,
Moreover, our approach opens the door to answering long-standing Killiany, R., Kennedy, D., Klaveness, S., Montillo, A., Makris, N., Rosen, B., Dale, A.M.,
2002. Whole brain segmentation: automated labeling of neuroanatomical structures
questions about memory function at the resolution of individual in the human brain. Neuron 33, 341–355. http://dx.doi.org/10.1016/S0896-
memories in healthy individuals. For instance, how do specific memo- 6273(02)00569-X.
ries emerge over learning experiences? When are the details of the Friston, K.J., 2011. Functional and effective connectivity: a review. Brain Connect. 1,
13–36. http://dx.doi.org/10.1089/brain.2011.0008.
past forgotten versus successfully consolidated into robust memory Gelbard-Sagiv, H., Mukamel, R., Harel, M., Malach, R., Fried, I., 2008. Internally generated
networks? How are the specific elements of memories flexibly integrat- reactivation of single neurons in human hippocampus during free recall. Science 322,
ed to create new knowledge? The current study also paves the way 96–101. http://dx.doi.org/10.1126/science.1164685.
Glosser, G., Saykin, A.J., Deutsch, G.K., O'Connor, M.J., 1995. Neural organization of
towards a unified computational approach in which the neural repre- material-specific memory functions in temporal lobe epilepsy patients as assessed
sentations supporting our memories are quantitatively linked to the by the intracarotid amobarbital test. Neuropsychology 9, 449–456. http://dx.doi.
formal mechanisms that underlie decision making. org/10.1037/0894-4105.9.4.449.
Golby, A.J., Poldrack, R.A., Brewer, J.B., Spencer, D., Desmond, J.E., Aron, A.P., Gabrieli, J.D.,
2001. Material-specific lateralization in the medial temporal lobe and prefrontal
Acknowledgements cortex during memory encoding. Brain 124, 1841–1854. http://dx.doi.org/10.1093/
brain/124.9.1841.
Gold, J.I., Shadlen, M.N., 2007. The neural basis of decision making. Annu. Rev. Neurosci.
The authors declare no competing financial interests. This study was
30, 535–574. http://dx.doi.org/10.1146/annurev.neuro.29.051605.113038.
supported by NIH (F32-MH100904), NSF CAREER Award (1056019), Gordon, A.M., Rissman, J., Kiani, R., Wagner, A.D., 2014. Cortical reinstatement mediates
and NIH (R01-MH100121). We wish to thank A. Huk for helpful the relationship between content-specific encoding activity and subsequent
recollection decisions. Cereb. Cortex 24, 3350–3364. http://dx.doi.org/10.1093/
comments on an earlier version of this report; J. Mumford for advice
cercor/bht194.
on optimizing the experimental design for RSA; B. Love for helpful Hanke, M., Halchenko, Y.O., Sederberg, P.B., Hanson, S.J., Haxby, J.V., Pollmann, S., 2009.
comments on the DDM analysis; and M. Schlichting, E. Stein, B. Gelman, PyMVPA: a python toolbox for multivariate pattern analysis of fMRI data.
and K. Nguyen for their help with data collection. Thank you also to the Neuroinformatics 7, 37–53. http://dx.doi.org/10.1007/s12021-008-9041-y.
Hanks, T.D., Kiani, R., Shadlen, M.N., 2014. A neural mechanism of speed-accuracy
Texas Advanced Computing Center (TACC; http://www.tacc.utexas. tradeoff in macaque area LIP. Elife 2014, 1–17. http://dx.doi.org/10.7554/eLife.
edu) at The University of Texas at Austin for providing high- 02260.
performance computing resources critical for the work presented Hassabis, D., Chu, C., Rees, G., Weiskopf, N., Molyneux, P.D., Maguire, E.A., 2009. Decoding
neuronal ensembles in the human hippocampus. Curr. Biol. 19, 546–554. http://dx.
within this report. doi.org/10.1016/j.cub.2009.02.033.
Heekeren, H.R., Marrett, S., Ungerleider, L.G., 2008. The neural systems that mediate
Appendix A. Supplementary data human perceptual decision making. Nat. Rev. Neurosci. 9, 467–479. http://dx.doi.
org/10.1038/nrn2374.
Heit, G., Smith, M.E., Halgren, E., 1988. Neural encoding of individual words and faces by
Supplementary data to this article can be found online at http://dx. the human hippocampus and amygdala. Nature 333, 773–775. http://dx.doi.org/10.
doi.org/10.1016/j.neuroimage.2015.12.015. 1038/333773a0.
Horner, A.J., Bisby, J.A., Bush, D., Lin, W.-J., Burgess, N., 2015. Evidence for holistic episodic
recollection via hippocampal pattern completion. Nat. Commun. 6, 7462. http://dx.
References doi.org/10.1038/ncomms8462.
Huffman, D.J., Stark, C.E.L., 2014. Multivariate pattern analysis of the human medial
Avants, B.B., Tustison, N.J., Song, G., Cook, P.A., Klein, A., Gee, J.C., 2011. A reproducible temporal lobe revealed representationally categorical cortex and representationally
evaluation of ANTs similarity metric performance in brain image registration. agnostic hippocampus. Hippocampus 10, 1–10. http://dx.doi.org/10.1002/hipo.
NeuroImage 54, 2033–2044. http://dx.doi.org/10.1016/j.neuroimage.2010.09.025. 22321.
Bakker, A., Kirwan, C.B., Miller, M., Stark, C.E.L., 2008. Pattern separation in the human Hulbert, J.C., Norman, K.A., 2014. Neural differentiation tracks improved recall of compet-
hippocampal CA3 and dentate gyrus. Science 319, 1640–1642. http://dx.doi.org/10. ing memories following interleaved study and retrieval practice. Cereb. Cortex 1–15.
1126/science.1152882. http://dx.doi.org/10.1093/cercor/bhu284.
 M.L. Mack, A.R. Preston / NeuroImage 127 (2016) 144–157 157

Johnson, J.D., McDuff, S.G.R., Rugg, M.D., Norman, K.A., 2009. Recollection, familiarity, and Polyn, S.M., Natu, V.S., Cohen, J.D., Norman, K.A., 2005. Category-specific cortical activity
cortical reinstatement: a multivoxel pattern analysis. Neuron 63, 697–708. http://dx. precedes retrieval during memory search. Science 310, 1963–1966. http://dx.doi.
doi.org/10.1016/j.neuron.2009.08.011. org/10.1126/science.1117645.
Kamitani, Y., Sawahata, Y., 2010. Spatial smoothing hurts localization but not information: Quiroga, R.Q., Reddy, L., Kreiman, G., Koch, C., Fried, I., 2005. Invariant visual representa-
pitfalls for brain mappers. NeuroImage 49, 1949–1952. http://dx.doi.org/10.1016/j. tion by single neurons in the human brain. Nature 435, 1102–1107. http://dx.doi.
neuroimage.2009.06.040. org/10.1038/nature03687.
Kirwan, C.B., Stark, C.E.L., 2007. Overcoming interference: an fMRI investigation of pattern Quiroga, R.Q., Reddy, L., Koch, C., Fried, I., 2007. Decoding visual inputs from multiple neu-
separation in the medial temporal lobe. Learn. Mem. 14, 625–633. http://dx.doi.org/ rons in the human temporal lobe. J. Neurophysiol. 98, 1997–2007. http://dx.doi.org/
10.1101/lm.663507. 10.1152/jn.00125.2007.
Knierim, J.J., Neunuebel, J.P., Deshmukh, S.S., B, P.T.R.S., 2014. Cortex : Objects, Path Quiroga, R.Q., Kreiman, G., Koch, C., Fried, I., 2008. Sparse but not “grandmother-cell” cod-
Integration And Local−Global Reference Frames Functional Correlates Of The Lateral ing in the medial temporal lobe. Trends Cogn. Sci. 12, 87–91. http://dx.doi.org/10.
And Medial Entorhinal Cortex : Objects, Path Integration And Local–Global Reference 1016/j.tics.2007.12.003.
Frames. Ratcliff, R., 1978. A theory of memory retrieval. Psychol. Rev. 85, 59–108. http://dx.doi.
Kriegeskorte, N., Mur, M., Bandettini, P., 2008. Representational similarity analysis — org/10.1037/0033-295X.85.2.59.
connecting the branches of systems neuroscience. Front. Syst. Neurosci. 2, 4. http:// Ratcliff, R., Rouder, J.N., 1998. Modeling response times for two-choice decisions. Psychol.
dx.doi.org/10.3389/neuro.06.004.2008. Sci. 9, 347–356.
Kruschke, J.K., 2012. Bayesian estimation supersedes the t test. J. Exp. Psychol. Gen. 142, Rissman, J., Gazzaley, A., D'Esposito, M., 2004. Measuring functional connectivity during
573–603. http://dx.doi.org/10.1037/a0029146. distinct stages of a cognitive task. NeuroImage 23, 752–763.
Kuhl, B.A., Chun, M.M., 2014. Successful remembering elicits event-specific activity pat- Ritchey, M., Wing, E.A., LaBar, K.S., Cabeza, R., 2012. Neural similarity between encoding
terns in lateral parietal cortex. J. Neurosci. 34, 8051–8060. http://dx.doi.org/10. and retrieval is related to memory via hippocampal interactions. Cereb. Cortex 23,
1523/JNEUROSCI.4328-13.2014. 2818–2828. http://dx.doi.org/10.1093/cercor/bhs258.
Kyle, C.T., Stokes, J.D., Lieberman, J.S., Hassan, A.S., Ekstrom, A.D., 2015. Successful retrieval Rolls, E.T., 2013. The mechanisms for pattern completion and pattern separation in the
of competing spatial environments in humans involves hippocampal pattern separa- hippocampus. Front. Syst. Neurosci. 7, 74. http://dx.doi.org/10.3389/fnsys.2013.00074.
tion mechanisms. Elife 4, 160. http://dx.doi.org/10.7554/eLife.10499. Rutishauser, U., Ye, S., Koroma, M., Tudusciuc, O., Ross, I.B., Chung, J.M., Mamelak, A.N.,
Lacy, J.W., Yassa, M.A., Stark, S.M., Muftuler, L.T., Stark, C.E.L., 2011. Distinct pattern sepa- 2015. Representation of retrieval confidence by single neurons in the human medial
ration related transfer functions in human CA3/dentate and CA1 revealed using high- temporal lobe. Nat. Neurosci. 1–12. http://dx.doi.org/10.1038/nn.4041.
resolution fMRI and variable mnemonic similarity. Learn. Mem. 18, 15–18. http://dx. Schapiro, A.C., Kustner, L.V., Turk-Browne, N.B., 2012. Shaping of object representations in
doi.org/10.1101/lm.1971111. the human medial temporal lobe based on temporal regularities. Curr. Biol. 22,
LaRocque, K.F., Smith, M.E., Carr, V.a., Witthoft, N., Grill-Spector, K., Wagner, A.D., 2013. 1622–1627. http://dx.doi.org/10.1016/j.cub.2012.06.056.
Global similarity and pattern separation in the human medial temporal lobe predict Schlichting, M.L., Preston, A.R., 2014. Memory reactivation during rest supports upcoming
subsequent memory. J. Neurosci. 33, 5466–5474. http://dx.doi.org/10.1523/ learning of related content. Proc. Natl. Acad. Sci. 111, 15845–15850. http://dx.doi.org/
JNEUROSCI.4293-12.2013. 10.1073/pnas.1404396111.
Leiker, E.K., Johnson, J.D., 2015. Pattern reactivation CO-varies with activity in the core Schlichting, M.L., Mumford, J.A., Preston, A.R., 2015. Learning-related representational
recollection network during source memory. Neuropsychologia 75, 88–98. http:// changes reveal dissociable integration and separation signatures in the hippocampus
dx.doi.org/10.1016/j.neuropsychologia.2015.05.021. and prefrontal cortex. Nat. Commun. 6, 8151. http://dx.doi.org/10.1038/ncomms9151.
Leutgeb, J.K., Leutgeb, S., Moser, M.-B., Moser, E.I., 2007. Pattern separation in the dentate Singer, A.C., Carr, M.F., Karlsson, M.P., Frank, L.M., 2013. Hippocampal SWR activity pre-
gyrus and CA3 of the hippocampus. Science 315, 961–966. http://dx.doi.org/10.1126/ dicts correct decisions during the initial learning of an alternation task. Neuron 77,
science.1135801. 1163–1173. http://dx.doi.org/10.1016/j.neuron.2013.01.027.
Liang, J.C., Wagner, A.D., Preston, A.R., 2013. Content representation in the human Smith, S.M., Jenkinson, M., Woolrich, M.W., Beckmann, C.F., Behrens, T.E.J., Johansen-
medial temporal lobe. Cereb. Cortex 23, 80–96. http://dx.doi.org/10.1093/ Berg, H., Bannister, P.R., De Luca, M., Drobnjak, I., Flitney, D.E., Niazy, R.K.,
cercor/bhr379. Saunders, J., Vickers, J., Zhang, Y., De Stefano, N., Brady, J.M., Matthews, P.M.,
Lisman, J.E., Grace, A.A., 2005. The hippocampal-VTA loop: controlling the entry of 2004. Advances in functional and structural MR image analysis and implementa-
information into long-term memory. Neuron 46, 703–713. http://dx.doi.org/10. tion as FSL. NeuroImage 23 (Suppl. 1), S208–S219. http://dx.doi.org/10.1016/j.
1016/j.neuron.2005.05.002. neuroimage.2004.07.051.
Marr, D., 1971. Simple memory: a theory for archicortex. Philos. Trans. R. Soc. Lond. Ser. B Squire, L.R., 2004. Memory systems of the brain: a brief history and current perspective.
Biol. Sci. 262, 23–81. http://dx.doi.org/10.1098/rstb.1971.0078. Neurobiol. Learn. Mem. 82, 171–177. http://dx.doi.org/10.1016/j.nlm.2004.06.005.
McClelland, J.L., McNaughton, B.L., O'Reilly, R.C., 1995. Why there are complementary Staresina, B.P., Henson, R.N.A., Kriegeskorte, N., Alink, A., 2012. Episodic reinstatement in
learning systems in the hippocampus and neocortex: insights from the successes the medial temporal lobe. J. Neurosci. 32, 18150–18156. http://dx.doi.org/10.1523/
and failures of connectionist models of learning and memory. Psychol. Rev. 102, JNEUROSCI.4156-12.2012.
419–457. http://dx.doi.org/10.1037/0033-295X.102.3.419. Staresina, B.P., Cooper, E., Henson, R.N., 2013. Reversible information flow across the me-
Miller, J.F., Neufang, M., Solway, A., Brandt, A., Trippel, M., Mader, I., Hefft, S., dial temporal lobe: the hippocampus links cortical modules during memory retrieval.
Merkow, M., Polyn, S.M., Jacobs, J., Kahana, M.J., Schulze-Bonhage, A., 2013. Neu- J. Neurosci. 33, 14184–14192. http://dx.doi.org/10.1523/JNEUROSCI.1987-13.2013.
ral activity in human hippocampal formation reveals the spatial context of Stokes, J., Kyle, C., Ekstrom, A.D., 2015. Complementary roles of human hippocampal sub-
retrieved memories. Science 342, 1111–1114. http://dx.doi.org/10.1126/ fields in differentiation and integration of spatial context. J. Cogn. Neurosci. 27,
science.1244056. 546–559. http://dx.doi.org/10.1162/jocn_a_00736.
Mumford, J.A., Turner, B.O., Ashby, F.G., Poldrack, R.A., 2012. Deconvolving BOLD Tanaka, K.Z., Pevzner, A., Hamidi, A.B., Nakazawa, Y., Graham, J., Wiltgen, B.J., 2014. Corti-
activation in event-related designs for multivoxel pattern classification analyses. cal representations are reinstated by the hippocampus during memory retrieval.
NeuroImage 59, 2636–2643. http://dx.doi.org/10.1016/j.neuroimage.2011.08. Neuron 84, 347–354. http://dx.doi.org/10.1016/j.neuron.2014.09.037.
076. Thakral, P.P., Wang, T.H., Rugg, M.D., 2015. Cortical reinstatement and the confidence and
Mundy, M.E., Downing, P.E., Dwyer, D.M., Honey, R.C., Graham, K.S., 2013. A critical role accuracy of source memory. NeuroImage 109, 118–129. http://dx.doi.org/10.1016/j.
for the hippocampus and perirhinal cortex in perceptual learning of scenes neuroimage.2015.01.003.
and faces: complementary findings from amnesia and FMRI. J. Neurosci. 33, Walther, D.B., Caddigan, E., Fei-Fei, L., Beck, D.M., 2009. Natural scene categories revealed
10490–10502. http://dx.doi.org/10.1523/JNEUROSCI.2958-12.2013. in distributed patterns of activity in the human brain. J. Neurosci. 29, 10573–10581.
Neunuebel, J.P., Knierim, J.J., 2014. CA3 retrieves coherent representations from http://dx.doi.org/10.1523/JNEUROSCI.0559-09.2009.
degraded input: direct evidence for CA3 pattern completion and dentate gyrus pat- Waydo, S., Kraskov, A., Quian Quiroga, R., Fried, I., Koch, C., 2006. Sparse representation in
tern separation. Neuron 81, 416–427. http://dx.doi.org/10.1016/j.neuron.2013.11. the human medial temporal lobe. J. Neurosci. 26, 10232–10234. http://dx.doi.org/10.
017. 1523/JNEUROSCI.2101-06.2006.
Ng, A.Y., Jordan, M.I., 2002. On discriminative vs. Generative classifiers: a comparison of Wiecki, T.V., Sofer, I., Frank, M.J., 2013. HDDM: hierarchical bayesian estimation of the
logistic regression and naive bayes, neural information processing systems. Adv. Neu- drift-diffusion model in python. Front. Neuroinform. 7, 14. http://dx.doi.org/10.
ral Inf. Process. Syst. 14, 841–849. 3389/fninf.2013.00014.
Nienborg, H., Cumming, B.G., 2009. Decision-related activity in sensory neurons reflects Wimber, M., Alink, A., Charest, I., Kriegeskorte, N., Anderson, M.C., 2015. Retrieval induces
more than a neuron's causal effect. Nature 459, 89–92. adaptive forgetting of competing memories via cortical pattern suppression. Nat.
Norman, K., O'Reilly, R., 2003. Modeling hippocampal and neocortical contributions to Publ. Group 18, 582–589. http://dx.doi.org/10.1038/nn.3973.
recognition memory: a complementary-learning-systems approach. Psychol. Rev. Wing, E.A., Ritchey, M., Cabeza, R., 2015. Reinstatement of individual past events revealed
110, 611–646. by the similarity of distributed activation patterns during encoding and retrieval.
Op de Beeck, H.P., 2010. Against hyperacuity in brain reading: spatial smoothing does not J. Cogn. Neurosci. 27, 679–691. http://dx.doi.org/10.1162/jocn_a_00740.
hurt multivariate fMRI analyses? NeuroImage 49, 1943–1948. http://dx.doi.org/10. Winkler, A.M., Ridgway, G.R., Webster, M.a., Smith, S.M., Nichols, T.E., 2014. Permutation
1016/j.neuroimage.2009.02.047. inference for the general linear model. NeuroImage 92, 381–397. http://dx.doi.org/
O'Reilly, R.C., Rudy, J.W., 2001. Conjunctive representations in learning and memory: 10.1016/j.neuroimage.2014.01.060.
principles of cortical and hippocampal function. Psychol. Rev. 108, 311–345. Xue, G., Dong, Q., Chen, C., Lu, Z., Mumford, J.A., Poldrack, R.A., 2010. Greater neural pat-
Pastalkova, E., Itskov, V., Amarasingham, A., Buzsáki, G., 2008. Internally generated cell tern similarity across repetitions is associated with better memory. Science 330,
assembly sequences in the rat hippocampus. Science 321, 1322–1327. http://dx.doi. 97–101. http://dx.doi.org/10.1126/science.1193125.
org/10.1126/science.1159775. Zeithamova, D., Dominick, A.L., Preston, A.R., 2012. Hippocampal and ventral medial pre-
Paz, R., Gelbard-Sagiv, H., Mukamel, R., Harel, M., Malach, R., Fried, I., 2010. A neural sub- frontal activation during retrieval-mediated learning supports novel inference. Neu-
strate in the human hippocampus for linking successive events. Proc. Natl. Acad. Sci. ron 75, 168–179. http://dx.doi.org/10.1016/j.neuron.2012.05.010.
U. S. A. 107, 6046–6051. http://dx.doi.org/10.1073/pnas.0910834107.