940555

article-commentary2020
PSSXXX10.1177/0956797620940555Witt et al.Motor Simulation and Naming Tools

ASSOCIATION FOR
Commentary PSYCHOLOGICAL SCIENCE
Psychological Science

Reanalysis Suggests Evidence for Motor 1­–4

© The Author(s) 2020
Article reuse guidelines:
Simulation in Naming Tools Is Limited: sagepub.com/journals-permissions
DOI: 10.1177/0956797620940555
https://doi.org/10.1177/0956797620940555

A Commentary on Witt, Kemmerer, www.psychologicalscience.org/PS

Linkenauger, and Culham (2010)

Jessica K. Witt1 , David Kemmerer2, Sally A. Linkenauger3 ,

and Jody C. Culham4
1
Department of Psychology, Colorado State University; 2Department of Speech, Language, and Hearing
Sciences, Purdue University; 3Department of Psychology, Lancaster University; and 4Brain and Mind
Institute, Department of Psychology, Western University

Received 12/3/19; Revision accepted 5/15/20

Embodied cognition is based on the radical claim that ball) slowed down and reduced the accuracy of the
motor processes influence cognitive processes. Accord- naming of the tools.
ing to a strong account of embodied cognition, motor On learning of a failure to replicate our results
processes are necessary for cognitive processes such (Saccone, Thomas, & Nicholls, in press) and noticing
as identifying tools. This strong account has been substantial differences in overall RTs between the rep-
rejected given evidence showing that some people with lication failure and our original study, we reinspected
apraxia who are impaired at tool use have unimpaired our data. With a growing appreciation of multiverse
tool recognition. However, this leaves the possibility and Bayesian approaches to data analyses in psychol-
for a weak account of embodied cognition in which ogy, we no longer believe that the original article accu-
motor processes play a supportive role. Evidence for rately reflects the results. We were inspired by the
such an account comes not only from neuroscientific Loss-of-Confidence Project by Julia Rohrer and col-
studies (e.g., Arévalo et al., 2007; Kan, Kable, Van leagues (in press) and their call for science to have an
Scoyoc, Chatterjee, & Thompson-Schill, 2006; Pobric, outlet to be self-correcting. One option was to retract
Jefferies, & Lambon Ralph, 2010) but also from purely our original article. A retraction implies an end of a
behavioral ones (Bub & Masson, 2012; Bub, Masson, & research project, whereas a self-correction via a Com-
Bukach, 2003; Witt, Kemmerer, Linkenauger, & Culham, mentary could promote open, nuanced discussion of
2010). the data, its flaws, and potential hints for future research.
In our article (Witt et al., 2010), we examined the role
of motor processes in naming tools and animals in two
Experiment 1 Reanalysis and
experiments. In Experiment 1, participants squeezed a
ball in one hand and named pictures of tools and ani- Multiverse Analysis
mals as quickly as they could. We reported that partici- Our reanalysis of the data from Experiment 1 revealed
pants were slower to name tools with handles oriented that the statistical outcomes were dependent on the
toward the hand squeezing the ball than tools with choice of data-cleaning procedures. In the original
handles oriented toward the unoccupied hand. No dif- article, we failed to disclose that RTs slower than 4,000
ference in reaction time (RT) was found for the animals. ms (which comprised 2% of the trials) were excluded.
In Experiment 2, participants performed the same task, RT data frequently contain extreme values, so outliers
but we limited viewing time and analyzed tool- and are typically excluded. However, there were consider-
animal-naming accuracy. We found that accuracy was able differences in RTs across participants and between
lower when the tool handle was oriented toward the
ball-squeezing hand than the unoccupied hand. We
Corresponding Author:
used this evidence to conclude that motor simulation Jessica K. Witt, Colorado State University, Department of Psychology
played a supportive role in identifying tools, given that (1876 Campus Delivery), Fort Collins, CO 80523
interfering with motor simulation (via squeezing the E-mail: [email protected]
2 Witt et al.

the tool and animal categories. Rather than use a single 1.5, 3, 10, and 1,000, respectively. For the nontrans-
exclusion criterion of 4,000 ms, a more appropriate formed RTs, the percentage of trials excluded was 8%,
exclusion criterion would be to trim RTs as a function 4%, 1%, and 0%. For comparison, the results originally
of the interquartile range (IQR) for each participant for excluded 2% of the trials.
each condition. With the exclusion criterion of 1.5 times We also applied Bayesian statistics, which have grown
the IQR, the critical effect in Experiment 1 was no lon- in popularity in psychology and other fields since the
ger significant, F(1, 62) = 0.14, p = .71. time of our original publication (e.g., Wagenmakers
Furthermore, there were additional aspects of the et  al., 2018) and which allow a quantification of the
data and the analysis that we could have considered. evidence not only in favor of an experimental hypothesis
Our analyses would have benefited from additional (as do frequentist statistics) but also against it (i.e., in
explorations because it would have been clear that the favor of the null hypothesis). To calculate a Bayes factor,
effect met the criterion for significance (p < .05) only we used the BayesFactor package in R (Version 0.9.12-
under select conditions. In other words, the effect was 4.2; Morey & Rouder, 2015). We ran a paired-samples
so fragile that exploring the data would have made t test with mean RT as the dependent measure and
clear that the effect was not robust. Verifying the robust- tool–hand congruency as the independent factor. The
ness against different criteria for outliers and other Bayes factor was calculated using a Cauchy prior with
experimenter decisions could have been reported in a a scale of 0.3, which was chosen because these kinds
multiverse analysis (Steegen, Tuerlinckx, Gelman, & of embodiment effects tend to be small. Note that
Vanpaemel, 2016). We correct the record by reporting increasing the scale of the Cauchy prior would only
the results of a multiverse analysis here. increase the strength of the evidence in favor of the null
We conducted the multiverse analysis to examine the hypothesis. A Bayes factor between 1 and 3 is consid-
robustness of the outcomes. We varied the outlier cri- ered anecdotal evidence of the alternative hypothesis
terion (1.5, 3, 10, and 1,000 times the participant’s con- over the null hypothesis, whereas a Bayes factor greater
dition-specific IQR; the last value was added so no trials than 3 is considered substantial evidence ( Jeffreys, 1961;
would be excluded). Because RTs are positively skewed, Lakens, 2016; Wetzels et al., 2011). Of the eight Bayes
they are sometimes log-transformed, so we varied factors (4 levels of outlier criteria × 2 levels of data
whether we used raw RTs or log-transformed RTs. We transformations), only two reached the threshold of hav-
also varied the analysis. In the original study, we con- ing substantial evidence, and both showed substantial
ducted a repeated measures analysis of variance with evidence for the null hypothesis over the alternative
mean RT as the dependent measure and tool–hand con- hypothesis (see Fig. 1). Another possible criterion for
gruency (whether the handle was facing the same or exclusion relates to accuracy. We calculated the percent-
the opposite hand that was squeezing the ball) as the age of trials with correct responses for each participant.
within-subjects factor. As part of this multiverse analysis, Five participants had scores beyond 1.5 times the IQR
we also analyzed the data using a linear mixed model. (all were less than 82% correct). Excluding these partici-
The dependent measure was RT on each trial, the fixed pants did not change the pattern of results.
effect (independent measure) was tool–hand congru- The multiverse analysis shows that the effect was not
ency, and random effects were included for participant, robust to various outlier, transformation, and analysis
stimulus, and their interaction. The model was singular decisions. If there were to be an effect of squeezing a
when we included random slopes for tool–hand congru- ball on naming tools, the effect seems to be present
ency, so we ran the model with and without random only for longer RTs, given that when longer RTs were
slopes. When the random slopes were included, they excluded, there was substantial evidence for the null
were included for both participant and for stimulus. hypothesis.
Only 21% of the p values were less than .05. This
suggests that the effect is not robust to experimenter
Experiment 2 Reanalysis
decisions such as outlier exclusion, data transforma-
tions, and analysis type. The magnitude of the estimated Experiment 2 tested whether squeezing the ball would
effect when calculated using nontransformed (raw) RTs affect naming accuracy when the images were pre-
increased as the outlier threshold increased (see Fig. sented for short durations (17, 33, 50, and 167 ms). The
1). For the log-transformed RTs, the outcomes were aim of Experiment 2 was to provide converging evi-
similar except that the estimates for 3, 10, and 1,000 dence for Experiment 1. Because accuracy rather than
were nearly identical because fewer trials were identi- RT was analyzed for Experiment 2, the results did not
fied as outliers when the data were log transformed. depend as much on outlier removal. The critical effect
For the log-transformed RTs, the percentage of trials was reported as significant. Because we retained only
excluded was 4%, 1%, 0%, and 0% for outlier cutoffs of the means and not the raw data, we were unable to
Motor Simulation and Naming Tools 3

5 Evidence for Null ANOVA

Evidence for Alternative 100 LMM: No Random Slope
LMM: Random Slope

80
4

60

Effect Estimate (ms)

Bayes Factor

3
40

20
2

1 −20

1.5 3 10 None 1.5 3 10 None

Outlier Criterion Outlier Criterion
Fig. 1.  Results from the multiverse analysis of the data from Experiment 1. The
left panel shows Bayes factors (BFs) as a function of the outlier criterion. Squares
indicate analyses on untransformed reaction times (RTs), and circles indicate
analyses on log-transformed RTs. Each open symbol corresponds to a BF01, which
is the amount of evidence for the null hypothesis over the alternative hypothesis.
Each solid symbol corresponds to a BF10, which is the amount of evidence for the
alternative hypothesis over the null hypothesis. Values greater than 3 (horizon-
tal dotted line) are considered to show substantial evidence; values less than 3
show anecdotal evidence. The right panel shows the estimates from the various
models in untransformed units as a function of the outlier criterion. Error bars
are ±1 SEM calculated from each model. ANOVA = analysis of variance; LMM =
linear mixed model.

conduct a linear mixed model. However, we were able tools. The new analyses on data from Experiment 1
to calculate Bayes factors to assess the strength of the found substantial evidence for the null hypothesis over
evidence. Overall, the Bayes factor was only 1.60 in the alternative when RTs were trimmed using our most
favor of the alternative hypothesis over the null hypoth- liberal criterion, and the new Bayesian analyses on data
esis, which corresponds to anecdotal evidence. When from Experiment 2 found only anecdotal evidence. The
we analyzed the data at each of the four durations debate over whether motor processes can inform cogni-
separately, the Bayes factor was less than 1. This sug- tion is critical for understanding cognitive processes,
gested more support for the null hypothesis over the but the current data do not advance this debate as had
alternative, but the Bayes factor did not reach the been originally argued.
threshold for substantial evidence, except for the 50-ms
duration. Transparency
Action Editor: D. Stephen Lindsay
Conclusions Editor: D. Stephen Lindsay
Author Contribution
We conclude that our data did not provide evidence J. K. Witt ran the reanalysis and drafted the manuscript.
for a supportive role for motor simulation in identifying D. Kemmerer, S. A. Linkenauger, and J. C. Culham edited
4 Witt et al.

the manuscript. All of the authors approved the final ver- Kan, I. P., Kable, J. W., Van Scoyoc, A., Chatterjee, A., &
sion for submission. Thompson-Schill, S. L. (2006). Fractionating the left
Declaration of Conflicting Interests frontal response to tools: Dissociable effects of motor
The author(s) declared that there were no conflicts of experience and lexical competition. Journal of Cognitive
interest with respect to the authorship or the publication Neuroscience, 18, 267–277.
of this article. Lakens, D. (2016, January 14). Power analysis for default
Funding Bayesian t-tests [Blog post]. The 20% statistician. Retrieved
This work was funded by a grant from the National Science from http://daniellakens.blogspot.com/2016/01/power-
Foundation (BCS-1632222) to J. K. Witt. analysis-for-default-bayesian-t.html
Open Practices Morey, R. D., & Rouder, J. N. (2015). Package ‘bayesFactor.’
Data, analysis scripts, and stimulus materials have been Retrieved from https://cran.r-project.org/web/packages/
made publicly available via the Open Science Framework BayesFactor/BayesFactor.pdf
and can be accessed at https://osf.io/rv8pe/. The design Pobric, G., Jefferies, E., & Lambon Ralph, M. A. (2010).
and analysis plans for the experiments were not preregis- Category-specific versus category-general semantic
tered. The complete Open Practices Disclosure for this impairment induced by transcranial magnetic stimulation.
article can be found at http://journals.sagepub.com/doi/ Current Biology, 20, 964–968.
suppl/10.1177/0956797620940555. This article has received Rohrer, J., Tierney, W., Uhlmann, E. L., DeBruine, L. M.,
the badges for Open Data and Open Materials. More infor- Heyman, T., Jones, B., . . . Yarkoni, T. (in press). Putting
mation about the Open Practices badges can be found at the self in self-correction: Findings from the loss-of-
http://www.psychologicalscience.org/publications/badges. confidence project. Perspectives on Psychological Science.
Saccone, E. J., Thomas, N. A., & Nicholls, M. E. R. (in press).
  One-handed motor activity does not interfere with nam-
ing lateralised pictures of tools. Journal of Experimental
ORCID iDs Psychology: Human Perception and Performance.
Steegen, S., Tuerlinckx, F., Gelman, A., & Vanpaemel, W.
Jessica K. Witt https://orcid.org/0000-0003-1139-1599
(2016). Increasing transparency through a multiverse
Sally A. Linkenauger https://orcid.org/0000-0002-6056-0187
analysis. Perspectives on Psychological Science, 11, 702–
712. doi:10.1177/1745691616658637
References Wagenmakers, E.-J., Marsman, M., Jamil, T., Ly, A., Verhagen,
Arévalo, A., Perani, D., Cappa, S. F., Butler, A., Bates, E., J., Love, J., . . . Morey, R. D. (2018). Bayesian inference
& Dronkers, N. (2007). Action and object processing in for psychology. Part I: Theoretical advantages and prac-
aphasia: From nouns and verbs to the effect of manipu- tical ramifications. Psychonomic Bulletin & Review, 25,
lability. Brain & Language, 100, 79–94. 35–57.
Bub, D. N., & Masson, M. E. J. (2012). On the dynamics of Wetzels, R., Matzke, D., Lee, M. D., Rouder, J. N., Iverson,
action representations evoked by names of manipulable G. J., & Wagenmakers, E.-J. (2011). Statistical evidence
objects. Journal of Experimental Psychology: General, 141, in experimental psychology: An empirical comparison
502–517. using 855 t tests. Perspectives on Psychological Science,
Bub, D. N., Masson, M. E. J., & Bukach, C. M. (2003). Gesturing 6, 291–298. doi:10.1177/1745691611406923
and naming: The use of functional knowledge in object Witt, J. K., Kemmerer, D., Linkenauger, S. A., & Culham,
identification. Psychological Science, 14, 467–472. J. (2010). A functional role for motor simulation in
Jeffreys, H. (1961). Theory of probability. Oxford, England; identifying tools. Psychological Science, 21, 1215–1219.
Oxford University Press. doi:10.1177/0956797610378307