THE BATTLE

Feature

AGAINST
PAPER MILLS
Some journals have admitted to a problem
with fake research papers. Now editors
are trying to combat it. By Holly Else and
Richard Van Noorden

W
hen Laura Fisher noticed sleuths have repeatedly warned that some guest speaker was Elisabeth Bik, a research-in-
striking similarities between scientists buy papers from third-party firms tegrity analyst in California known for her skill
research papers submitted to to help their careers. Rather, it was extraordi- in spotting duplicated images in papers, and
RSC Advances, she grew suspi- nary that a publisher had publicly announced one of the sleuths who posts their concerns
cious. None of the papers had something that journals generally keep quiet about paper mills online.
authors or institutions in com- about. “We believe that it is a paper mill, so we Bik thinks there are thousands more of these
mon, but their charts and titles want to be open and transparent,” Fisher says. papers in the literature. The RSC’s announce-
looked alarmingly similar, says The RSC wasn’t alone, its statement added: ment is significant for its openness, she says.
Fisher, the executive editor at the journal. “I “We are one of a number of publishers to have “It is pretty embarrassing that so many papers
was determined to try to get to the bottom of been affected by such activity.” Since last Janu- are fake. Kudos to them to admit that they have
what was going on.” ary, journals have retracted at least 370 papers been fooled.”
A year later, in January 2021, Fisher retracted that have been publicly linked to paper mills, an At some journals that have had a spate of
68 papers from the journal, and editors at two analysis by Nature has found, and many more apparent paper-mill submissions, editors have
other Royal Society of Chemistry (RSC) titles retractions are expected to follow. now revamped their review processes, aiming
retracted one each over similar suspicions; Much of this literature cleaning has come not to be fooled again. Combating industrial-
15 are still under investigation. Fisher had about because, last year, outside sleuths pub- ized cheating requires stricter review: telling
found what seemed to be the products of licly flagged papers that they think came from editors to ask for raw data, for instance, and
paper mills: companies that churn out fake paper mills owing to their suspiciously simi- hiring people specifically to check images.
scientific manuscripts to order. All the papers lar features. Collectively, the lists of flagged Science publishing needs a “concerted, coor-
came from authors at Chinese hospitals. papers total more than 1,000 studies, the dinated effort to stamp out falsified research”,
The journals’ publisher, the RSC in London, analysis shows. Editors are so concerned by the RSC said.
announced in a statement that it had been the the issue that last September, the Committee
victim of what it believed to be “the systemic on Publication Ethics (COPE), a publisher-ad- Paper-mill detectives
production of falsified research”. visory body in London, held a forum dedicated In January 2020, Bik and other image detec-
What was surprising about this was not the to discussing “systematic manipulation of tives who work under pseudonyms — Smut
paper-mill activity itself: research-integrity the publishing process via paper mills”. Their Clyde, Morty and Tiger BB8 — posted, on a

516 | Nature | Vol 591 | 25 March 2021

©
2
0
2
1
S
p
r
i
n
g
e
r
N
a
t
u
r
e
L
i
m
i
t
e
d
.
A
l
l
r
i
g
h
t
s
r
e
s
e
r
v
e
d
.
 mentions a paper mill — despite the RSC’s
announcement that it thinks the articles
did come from one. But Nature has tallied
370 articles retracted since January 2020,
all from authors at Chinese hospitals, that
either publishers or independent sleuths have
alleged to come from paper mills (see ‘Fraud
allegations’). Most were published in the past
three years (see ‘Chinese hospital papers on
the rise’). Publishers have added expressions
of concern to another 45 such articles.
Nature has identified a further 197 retrac-
tions of papers from authors at Chinese hos-
pitals since the start of last year. These are not
ones that have made it onto lists of potential
publication-mill products, although some
were flagged by sleuths for image concerns,
often on the post-publication peer-review
website PubPeer.

Industrialized cheating
The problem of organized fraud in publishing
is not new, and not confined to China, notes
Catriona Fennell, who heads publishing ser-
vices at the world’s largest scientific publisher,
Elsevier. “We’ve seen evidence of industrial-
ized cheating from several other countries,
including Iran and Russia,” she told Nature last
year. Others have also reported on Iranian and
Russian paper-mill activities.
In a statement this year to Nature, Elsevier
said that its journal editors detect and prevent
the publication of thousands of probable
paper-mill submissions each year, although
some do get through.
China has long been known to have a prob-
lem with firms selling papers to researchers,
says Xiaotian Chen, a librarian at Bradley Uni-
versity in Peoria, Illinois. As far back as 2010, a
blog run by science journalist Leonid Schnei- team led by Shen Yang, a management-studies
ILLUSTRATION BY ANA KOVA

der, a list of more than 400 published papers researcher then at Wuhan University in China,
they said probably came from a paper mill. Bik warned of websites offering to ghostwrite

WE’RE WONDERING
dubbed it the ‘tadpole’ paper mill, because papers on fictional research, or to bypass
of the shapes that appeared in the papers’ peer-review systems for payment. In 2013,

HOW WE PROTECT
western blot analyses, a type of test used to Science reported on a market for authorships
detect proteins in biological samples. A spate on research papers in China. In 2017, China’s
of media headlines followed. Throughout the
year, the sleuths (not always working together) OURSELVES FROM Ministry of Science and Technology (MOST)
said it would crack down on misconduct after
posted spreadsheets of other suspect papers —
picking up on similar features across multiple PUBLISHING THIS a scandal in which 107 papers were retracted at
the journal Tumor Biology; their peer reviews
studies. By March 2021, they had collectively
listed more than 1,300 articles, by Nature’s
tally, as possibly coming from paper mills.
STUFF.” had been fabricated and a MOST investigation
concluded that some had been produced by
third-party companies.
Journals started to look at the papers. Graf, director of research integrity at Wiley, Physicians in China are a particular target
According to Nature’s analysis, around 26% which publishes JCB, said in January that the market because they typically need to publish
of the articles that the sleuths alleged came publisher had completed investigations into research articles to gain promotions, but are so
from paper mills have so far been retracted 73 papers identified by Bik and others, and had busy at hospitals that they might not have time
or labelled with expressions of concern. Many found no reason to act on 11 of them. Seven to do the science, says Chen. Last August, the
others are still under investigation. The Jour- others required corrections and 55 have been Beijing municipal health authority published
nal of Cellular Biochemistry (JCB), for instance, retracted or will be retracted. a policy stipulating that an attending physi-
announced in February1 that, last year, editors Publishers almost never explicitly declare cian wanting to be promoted to deputy chief
investigated and retracted 23 of 137 papers on retraction notices that a particular study physician must have at least two first-author
alleged to contain image manipulation. is fraudulent or was created by a company to papers published in professional journals;
Journals did not identify problems with all order, because it is difficult to prove. None three first-author papers are required to
of the papers that had been flagged. Chris of the RSC’s retraction notices, for instance, become a chief physician. These titles affect

Nature | Vol 591 | 25 March 2021 | 517

©
2
0
2
1
S
p
r
i
n
g
e
r
N
a
t
u
r
e
L
i
m
i
t
e
d
.
A
l
l
r
i
g
h
t
s
r
e
s
e
r
v
e
d
.
Feature
a physician’s salary and authority, as well as FRAUD ALLEGATIONS

SCIENCEINTEGRITYDIGEST.COM AND NATURE ANALYSIS

the surgeries they are allowed to perform, Since January 2020, independent sleuths and journals have flagged more than 1,000 published
papers potentially linked to paper mills — companies that produce fraudulent articles. Hundreds
says Changqing Li, a former senior physician
of these papers have been retracted, corrected or labelled with an expression of concern (EOC).
and gastroenterology researcher at a Chinese
hospital who now lives in the United States. Retracted EOC Corrected Paper-mill alleged; article so far unchanged

“The effect is devastating,” says Li, about the

SOURCES: ORBETTERSCIENCE.COM,
impacts on Chinese science. “The literature
environment published in Chinese is already
ruined, since hardly anyone believes them or
references studies from them.” 0 200 400 600 800 1,000 1,200 1,400 1,600
“Now this plague has eroded into the inter- Number of papers*
national medical journals,” he adds. The fact *All papers include authors from Chinese hospitals. Another 197 articles (at least) with authors at Chinese hospitals have been retracted since
January 2020. These are not ones that have made it onto lists of potential publication-mill products. Data collated by Nature up to 18 March.
that people use paper mills also affects Chi-
na’s reputation globally, says Futao Huang, the publisher FEBS Press in Heidelberg, Ger- don’t seem to be linked to any of the author
a Chinese researcher working at Hiroshima many, who screens incoming manuscripts for names. “Individually, these factors may not
University in Japan. a number of journals, and helped the RSC with be problematic, but taken together they raise
The prevalence of problem papers is leading its investigation. concerns and could be part of a pattern,” she
some journal editors to doubt the submissions Potential signs of trouble include papers said. Editors at the forum also noted that a
they get from Chinese hospital researchers. from different authors at different institu- manuscript-processing system, ScholarOne,
“The increasing volume of this ‘junk science’ tions sharing similar features: western blots can flag up unusual activity when it picks up
is wreaking havoc on the credibility of the with identical-looking backgrounds and sus- on submissions from the same computer. A
research emanating out of China and increas- piciously smooth outlines, titles that seem ScholarOne alert was also instrumental in the
ingly casting doubt upon legitimate science to be variations on a theme, bar charts with RSC’s investigation.
from the region,” said a February 2021 edito- identical layouts that supposedly represent In February, Naunyn-Schmiedeberg’s Archives
rial2 in the journal Molecular Therapy. different experiments, or identical plots of of Pharmacology said it had been affected by
Several other editors echo these concerns flow cytometry analyses, which are used in paper mills. The journal published an editorial3
about the impact of paper mills. “They are studying cells. It seems that these manuscripts listing important features of paper-mill articles.
undermining our confidence in the other man- are produced from common templates, with These included non-academic e-mail addresses
uscripts received from Chinese groups,” says words and images slightly tweaked to make (which happen to be common with Chinese sci-
Frank Redegeld, editor in chief of the European the papers look a little different. entists), authors’ inability to supply raw data
Journal of Pharmacology, published by Elsevier. A particular problem is biomedical arti- when asked, and poor English. The journal is
China’s science and education ministries cles that claim to investigate understudied retracting 10 studies, and it reports that around
have taken steps to curb problematic publi- 5% of all its submissions are from paper mills.
cation incentives. They published a notice Publishers and others battling paper mills
last February telling research institutions suspect they are only seeing the tip of the ice-
— including hospitals — not to promote or berg in the published literature. In part, that’s

WE HAVE VERY LITTLE

recruit researchers solely on the basis of the because similarities between images across
numbers of papers they publish, and also studies might become obvious only when

INFORMATION ABOUT
told them to stop paying cash bonuses for many papers are compared. Sleuths also know
papers. And in August, China announced the that features such as similar western blots

THE PEOPLE OR
introduction of measures to crack down on and flawed nucleotide sequences might be
research misconduct, including attempts to the most obvious signs of paper-mill activity,
curb independent contractors who fabricate
data on others’ behalf. (MOST didn’t respond COMPANIES DOING says Bik. “There may be tonnes of other paper
mills that have done a better job of hiding it,”
to Nature’s request for comment on the scale
of the problem or the impact of its measures.)
Some Chinese researchers think these
THIS.” she says. Editors at the COPE forum said they’d
seen paper mills in areas such as computer
sciences, engineering, humanities and social
measures are beginning to work. Li Tang, who genetic regions that might be involved in sciences, for instance.
researches science policy at Fudan University cancers. Jennifer Byrne, a molecular oncology The overall size of the paper-mill problem
in Shanghai, China, is hopeful that submis- researcher at the University of Sydney, Aus- probably runs to thousands or tens of thou-
sions from paper mills in China will fall in the tralia, specializes in exposing flawed papers sands of papers, Bik, Byrne and others think4.
future — although she notes that the issue isn’t of this type, by spotting that their experimen- Graf, at Wiley, says it’s hard to estimate. “I don’t
confined to Chinese research. tal details sometimes list incorrect nucleotide think it should be understated, I can’t say how
Redegeld says he hasn’t yet seen a decrease sequences or reagents, so that the experi- big it is,” he says. “We have very little informa-
in the number of suspected paper-mill manu- ments described cannot have taken place. tion about the people or companies doing this.
scripts his journal receives, which he estimates Many of these papers are probably doctored I am exasperated by the situation, and that is
to be around 15 a month. simply by switching around the type of cancer being polite.”
or the genes involved in the study, says Byrne, “It’s detrimental to science as a whole
Problem signs although it’s hard to prove they’re from paper because it makes science and scientists look
Image-integrity sleuths and journal editors mills. “This problem of incorrect nucleotide unreliable,” says Christopher. Byrne has iden-
have identified a range of features in manu- sequences in the literature is rampant,” she says. tified a different concern: she worries that by
scripts that could be fingerprints of a paper At last September’s COPE forum, Bik rattled simply appearing in journals, fake studies that
mill. “We’re wondering how we protect our- off other red flags for editors to watch out link genes to particular cancers can give the
selves from publishing this stuff,” says Jana for, including papers from Chinese hospitals perception of activity in an area where there is
Christopher, an image-integrity analyst at and manuscripts with e-mail addresses that none, and might be included in meta-analyses.

518 | Nature | Vol 591 | 25 March 2021

©
2
0
2
1
S
p
r
i
n
g
e
r
N
a
t
u
r
e
L
i
m
i
t
e
d
.
A
l
l
r
i
g
h
t
s
r
e
s
e
r
v
e
d
.
 “People die from cancer — it is not a game. It have changed their scope to avoid subject publish there, which could be why some jour-
is important that the literature describes the areas that seem to be a particular focus of nals seem to be more affected than others.
work that takes place,” she adds. paper mills, the publisher says. And several One journal, the European Review for Medical
publishers say many of their journals have and Pharmacological Sciences, has retracted
Zombie papers updated their policies to require that authors 186 articles since January 2020, most of them
Journal editors know that if they reject man- present the raw data behind their western flagged by Bik and Smut Clyde. “We were
uscripts they suspect to be fabricated, that blots at the time of submission. Asking for raw shocked by these investigations,” says one of
might not kill the paper forever. Fraudulent data is one of the main ways that publishers its editors in chief, Antonio Gasbarrini.
manuscripts can be submitted to multiple tell editors to follow up when they think there Many journals are starting to employ ana-
journals at the same time: so even if an editor might be something wrong with a manuscript. lysts to try to spot problems in manuscripts
rejects it during peer review, they might see it But editors are aware that even raw data can be as they come in. Graf, for instance, says that
published elsewhere. faked, especially if paper-mill firms catch on last year Wiley employed and trained 11 peo-
This has happened to Christopher, who 3 that such requests are being made. ple to try to spot manipulated images across
years ago saw alarming similarities in a clus- “Asking for raw data is not an absolute guar- 24 journals — focusing on the papers most
ter of 13 research manuscripts submitted to 2 antee, as you can fake the data. It is a deterrent,” likely to be published. It hopes to expand the
journals published by FEBS Press, where she says Sabina Alam, director of publishing integ- programme to more titles.
worked. Their western blots seemed to be not rity and ethics at Taylor and Francis. One of its Publishers would like to automate some of
only fabricated, but also similar, as if they’d journals, Artificial Cells, Nanomedicine, and this screening process. Many have teamed up
been created by tweaking a template. The jour- Biotechnology, is investigating almost 100 pub- with research groups to develop software that
nals rejected the manuscripts on her advice. lished papers alleged to be from paper mills. could detect duplicated images across pub-
Christopher published a 2018 paper5 warning Alam also says that once they started lished papers, and, last May, an industry group
of “systematic fabrication of scientific images” investigations, some authors quickly asked formed to try to set standards for these checks.
and urged journals to invest in pre-publica- to withdraw their papers. Some sent raw data Software is improving but isn’t yet capable of
tion image screening. She also noted that she’d in unreadable formats or without labels. In all looking through many papers on a massive
seen some papers appear in other journals. these cases, journal editors say they’re not sure scale, says IJsbrand Jan Aalbersberg, the head
Christopher told Nature that she tried to whether it’s correct to withdraw such articles, of research integrity at Elsevier, who chairs the
privately raise the alarm about the papers. or to do something else — and are hoping for group. To do this would also require a giant
In 2018, for instance, she and FEBS Letters’ guidance on this from COPE. Bik has pointed shared database of images that publishers
managing editor advised the journal Cellular out that some journals have already allowed could check for duplication between papers.
Physiology and Biochemistry that a paper it had authors to withdraw papers without stating That will come when software can handle it,
published that year was probably fabricated; the reason for retraction. Aalbersberg predicts.
it had been simultaneously submitted to FEBS COPE says it will update its existing guid- Suzanne Farley, Springer Nature’s
Letters, which had rejected it. But the journal’s ance on how journals should deal with system- research-integrity director, based in London,
publisher at the time, Karger in Basel, Switzer- atic manipulation of the publication process, says she thinks that there will be a fall in the pro-
land, did not hear of any problem until 2020, and is also creating a task force of editors from portion of paper-mill submissions. “The paper
when the paper was flagged up again in Bik and its membership to determine how the organi- mills are aware that publishers are getting bet-
others’ ‘tadpole paper mill’ collection, along zation can provide better support on the issue. ter at detecting their submissions, and poten-
with other papers in the journal. Karger is now tial paper-mill customers are aware that there
investigating all these papers together with Path forwards are now more serious consequences of using
the journal’s current publisher, says Christna Publishers say that they are limited in what the services,” she says. (Nature’s news team is
Chap, Karger’s head of editorial development. they can do to share information between jour- editorially independent of its publisher.) In
This year, Christopher again looked into the nals because even titles within the same stable the meantime, Farley says, there will be more
13 manuscripts that had been submitted to her are editorially independent of one another. retractions and expressions of concern. “We are
journals. She found they had all been published They’re wary of sharing information between committed to cleaning house,” she says.
in other journals; so far, only three have been titles or publishers about an author that could But Christopher worries that an arms race
retracted and one has an expression of concern. be defamatory, and data-protection rules could develop if fraudsters get better at avoid-
Many journals have changed their editori- hinder the sharing of authors’ personal data. ing obvious mistakes. One preprint posted to
al-review processes to try to combat organized Once fraudsters know they can get a paper bioRxiv last year6, for instance, suggested that
fraud. Some Elsevier journals, for instance, into a particular title, they might continue to artificial intelligence techniques could gener-
ate fake western blots that were indistinguish-
CHINESE HOSPITAL PAPERS ON THE RISE able from real ones. “I’m really worried about
The number of English-language journal articles with authors from the sophistication going up,” she says.
Chinese hospitals has risen roughly fifty-fold over two decades.
7% of all papers with
Only Chinese authors International co-authors authors from China Holly Else is a reporter and Richard Van
40
Noorden is a features editor, both at Nature in
Number of articles (thousands)

London.
30
1. Behl, C. J. Cell. Biochem. https://doi.org/10.1002/
jcb.29906 (2021).
2. Frederickson, R. M. & Herzog, R. W. Mol. Ther. https://doi.
20
org/10.1016/j.ymthe.2021.02.011 (2021).
3. Seifert, R. Naunyn Schmiedebergs Arch. Pharmacol. 394,
431–436 (2021).
SOURCE: LENS.ORG

10 3% of all papers with 4. Byrne, J. A. & Christopher, J. FEBS Lett. 594, 583–589
authors from China (2020).
5. Christopher, J. FEBS Lett. 592, 3027–3029 (2018).
0 6. Qi, C., Zhang, J. & Luo, P. Preprint at bioRxiv https://doi.
1990 1992 1994 1996 1998 2000 2002 2004 2006 2008 2010 2012 2014 2016 2018 2020 org/10.1101/2020.11.24.395319 (2020).

Nature | Vol 591 | 25 March 2021 | 519

©
2
0
2
1
S
p
r
i
n
g
e
r
N
a
t
u
r
e
L
i
m
i
t
e
d
.
A
l
l
r
i
g
h
t
s
r
e
s
e
r
v
e
d
.