Scientific knowledge is shared through the publication of articles in journals. Sometimes, such an article later turns out to be flawed and is retracted. Though the act of retraction aims to purge the scientific literature of bad science, retracted articles continue to be cited in other articles. These citations are thus aptly called zombie citations.

We calculated a metric that measures the extent to which journals include articles that cite retracted work. We used the open-source database provided by OpenAlex and curated this data to reflect our focus on peer-reviewed journal articles. Our metric is dynamically and retrospectively updated, i.e., a new retraction influences previously calculated indices.

Furthermore, we will improve the calculation methods that as the quality of the available data and our analytical capacity is upgraded.

The act of retracting inadequate research is both laudable and necessary, but worth nothing without accountability. A publication in a prestigious journal implies outstanding quality of the published work. Consequently, a retraction comes with a similar responsibility towards the scientific community. Through this metric, we provide more transparency on the second order effect of retractions in scientific publishing. Consequently, we strive to improve the quality of science itself, albeit in tiny increments.

Introduction

They say you die twice. Once when you stop breathing and a second time, a bit later on, when somebody mentions your name for the last time. This quote is attributed to Banksy, and as always, the there is a thin line between art and science.

Scientific knowledge is shared mostly through the publication of articles in peer reviewed journals. Though many an author would describe the peer review process as cumbersome and tedious, that same author appreciates the process when reading the work of colleagues. However, sometimes an article that contains statistical errors (honest mistake) or fabricated data (deliberate misconduct) slips through the supposedly well-guarded gates of the review process. In that case, the scientific journals issues an editorial ‘expression of concern’ until investigation is concluded, with the retraction of said article as ultimate sanction. This equivalent of the death penalty should be sufficient to solve the problem, right?

Not so much. It is through citation in other articles that retracted works stay alive and continue to spread misinformation. Hence the very appropriate terms ‘zombie literature’ or ‘zombie papers’ [1]. Not to be confused with zombie trials (i.e., false patient data) currently and coincidentally circulating the domain of anaesthesia [2].

The first study on this phenomenon dates back to 1990, performing citation analysis to quantify the effect of retraction on subsequent citations (retraction leads to 35% reduction in citations) and to identify responsible factors (inconsistent formatting and indexing) [3]. A more recent study found a decline in citations of 69% for articles retracted since 2000 [4].

It seems, however, that a lot of journals rather not talk about their mistakes. Most of the 12.000 journals recorded in Clarivate's widely used Web of Science database of scientific articles have not reported a single retraction since 2003 [5]. Even though every stakeholder in the scientific publishing process agrees that the system of retractions is an important mode of governance – or at least says so. It is through the act of retraction that the scientific community is alerted to false knowledge, reducing the spread of misinformation, and helping researchers avoid building on studies predicated on false science [4].

In order to increase the transparency regarding the retraction process, Ivan Oransky and Adam Marcus launched the blog ‘Retraction Watch’. Since more than 80 scientific papers have cited their work (by means of a blog post, scientific article, or database), there is no doubt their non-profit initiative has outgrown the status of a blog, and should receive more credit and support. They provide a much needed ‘window into the self-correcting nature of science’ [6].

It is our ambition to increase the transparency on how our scientific journals cope with values such as integrity, inclusivity, and accountability. To do so, we will calculate a wide variety of metrics that concern all aspects of scientific publishing. Starting with the current metric, on zombie citations.

Methods

Database

As for all of the Peerbound metrics, we used the OpenAlex API as fundamental database. We only included those journals that were indexed in PubMed on January 1st, 2022. Any journals that allow the submission of manuscripts by invitation only, were excluded.

We limited the type or genre of the work to type=5, which means id=“journal-article”. This excludes all content published in books, and content from conferences published as proceedings. Furthermore, we excluded any article type that isn’t subjected to peer review (e.g., editorials).

A detailed whitepaper (such as the current document) on our methods of curation of the underlying database is in the works, for clarification and future reference.

Selection criteria

Those articles that cite articles flagged as retracted articles (is_retracted=true) are the primary interest of the current metric. We are aware that there are many more retractions indexed in the Retraction Watch database than in the current dataset and hope to include those in the nearby future (see also Discussion).

This metric is primarily oriented towards scientific journals. Of course, the authors are responsible for the content of their work, including citations. However, as the raison d’être of a journal has become the signaling of the quality of the published work therein, we believe the final responsibility lies with the journal.

Calculation methods

For each article that meets our inclusion criteria (see Database), we calculated the following value:

Article zombie citation quotient (azcqn) = x citations of retracted articles / y total comparable citations

To subsequently create the metric on a journal level, we use the root-mean-square (RMS) deviation method. We feel that a journal that publishes works that cite multiple retracted articles (i.e., larger errors) should have a disproportionately larger index value. Therefore, for each journal that fulfills our inclusion criteria (see Database), we calculated the following value:

Journal zombie citation index = sqrt ((azcq1^2 + azcq2^2 + azcqn^2) / n articles)

This index will be dynamically updated every month, and will be calculated for each year since 2000. Recent retractions of older work will introduce new zombie citations, thus continuously influencing the quotients and indices. It is important to recognize that progressive insight is fundamental for this metric. Therefore, the value of the zombie citation index regarding a particular journal becomes more precise as time passes on. Just like science itself.

Results

As an example, let’s say that journal A published 100 relevant articles in a certain year. Most articles won’t cite zombie papers (hopefully). But in this case, 5 articles each cite 1 retracted article, and 1 article cites 4 retracted articles. This journal has a fictious reference limit of 50 for each article. On average, 45 of the references concern comparable citations (i.e., not citing book chapters, editorials, proceedings, or articles from predatory journals, etc.). The corresponding zombie citation index would be:

Example = sqrt ((94 x 0 + 5 x (1/45)^2 + 1 x (4/45)^2) / 100) = 0,00916

The dashboard visualizing the index values could look similar to Figure 1. For the purpose of clarity, up to five different journals can be selected for comparison. Benchmarks on best and worst performing journals within the domain will be included.

Clicking on the datapoint that depicts a certain year for a particular journal will lead to an overview page, displaying the information on every article that cites a zombie paper. In-depth information on the retracted article itself will not be presented, instead a link to the editorial note concerning the retraction – if available – will be shown.

Discussion

Our ‘zombie citation index’ will provide the much-needed transparency on the second order effect of retractions. More specifically, which journals share the responsibility for the fact these zombie papers just.refuse.to.die and consequently should be held accountable. There are, however, several issues that need to be considered when analyzing the indices. We hope to improve our calculation methods as the quality of the available data and our analytical capacity is upgraded.

Incomplete database

First and foremost, we know that corrections and retractions are far from being registered in a comprehensive manner, illustrating the lack of commitment to the ongoing stewardship of published content by the various publishers [7]. The manually curated database provided by Retraction Watch represents an underestimation of the volume of retracted works. Therefore, the quest for a comprehensive dataset of retractions needs to be continued.

Note that this is a related, but different problem than the general tendency of editorial indifference and negligence regarding the handling of expressions of concern. We hope to capture this problem in a future metric.

Timing of citation

Secondly, the date of retraction plays an important role. Citing an article that ends up being retracted is worrisome but not culpable. Citing an article that has been retracted before the submission/publication of a manuscript, is culpable for both authors and journal. As the OpenAlex API does not seem to provide the date of retraction, we will need to find a work-around or organize access to the Retraction Watch database.

Once we are able to make the distinction between benign and malignant citations, the current metric can be considered a resultant metric of both. However, because of the RMS deviation methods, keep in mind that it is essential to first add up benign and malignant citations, before calculating the resultant index.

Context of citation

Another issue is that perhaps a zombie paper is cited exactly because it has been retracted. For instance, the infamous Wakefield paper linking autism and vaccines [8], which according to Google Scholar has been cited almost 4000 times (according to our inclusion criteria (see Database) the number of citations is [XXX]). Citations of retracted work as an illustration of ‘how not to conduct science’ should not be regarded as malignant. In order to correct for the context and content of a citation further analysis using the scite API (a platform that provides the context of a citation) might give sufficient details for adequate interpretation.

Author metrics

When considering metrics on author level, the discussion of bad science and citation of some examples will simply increase the citational impact of the authors found guilty of misconduct. Thus, retracted papers will continue to boost the bibliometric indices of their authors even when they are simply used to illustrate fraud [1]. All the more reason to correctly cite retracted work, and to create author metrics immune for this paradoxical effect.

Responsibilities

The International Committee of Medical Journal Editors (ICMJE) recommendations for manuscript preparations suggest that authors are responsible for ensuring that reference lists are accurate, which includes checking for retractions. Whilst understandable considering editorial workload, it is the journal that accepts (or rejects) a manuscript and gets paid handsomely for doing so [9]. Therefore, it should be the responsibility of the publishing journal to perform a final check on the accepted manuscript for citations of previously retracted work [10].

Consequently, in the case of a new retraction, it should be the responsibility of the retracting journal to inform the hitherto citing authors/journals about this decision and supporting arguments.

Current initiatives

The Committee on Publication Ethics (COPE) states that “Journals are responsible for ensuring that retractions are labelled in such a way that they are identified by bibliographic databases and should also include a link to the retracted article” [11]. As not all journals adhere to these guidelines – some just remove the article and pretend it never existed – an agenda was recently developed to reduce the inadvertent spread of retracted science [12].

The previously mentioned Retraction Watch provides the most comprehensive source of information on this key issue in scientific publishing. It is therefore quite unbelievable and undesirable that this database is built on charitable contributions from the community, whilst the academic publishers are swimming in >40% profits paid for with taxpayers money [13]. Our suggestion would be that a journal should donate the collected APC (article processing charge, a publication fee to be paid by the authors to have an article published with ‘open access’ availability) to Retraction Watch for every article that ends up being retracted. We will start a lobby for this proposal as soon as time permits.

But before an article ends up being retracted, expressions of concern are usually discussed on a post publication peer review platform, such as PubPeer. What’s more, as a free service, PubPeer offers a browser plugin that flags articles once these have raised comments by its users. Though an elegant solution, most hospitals don’t assign their doctors administrative rights that allow them to install software on their desktop or virtual machine. This means that many doctors do their research on a computer that is unable to flag these controversial articles. Additionally, an obvious selection bias is at play here: the readers that know of PubPeer aren’t the ones benefitting the most from some guidance on the identification of bad science and research misconduct.

Future applications

Though the retraction of flawed articles purges the scientific literature of bad science, the continuous citation of these works presents a challenge. Perhaps the integration of the retraction database into a trustworthy search engine would be a first step. Currently, a substantial portion of the retraction notices on the publishers’ platforms are synchronously unavailable on PubMed [14]. Therefore, it is possible for a researcher to read and cite an article, not knowing it has been retracted.

Finally, though unimaginable, some authors have succeeded in the resurrection of withdrawn literature through the self-citation of their retracted papers. It seems we also need to work on a metric to identify these Frankenstein scientists [1].

Conclusion

The existing system of retractions being the key bulwark against false science is surrounded by ambiguity and obscurity. Publishers often lack commitment to the ongoing stewardship of published content and should be held accountable. Through the ‘zombie citation index’ we provide transparency on the second order effect of retractions. We will continue to improve this metric and expand towards a wide variety of metrics that concern all aspects of scientific publishing.

Acknowledgments

We would like to thank Maarten van Dun for his constructive feedback.

References

1.         Bucci EM. On zombie papers. Cell Death Dis. 2019;10(3):189. doi:10.1038/s41419-019-1450-3

2.         Ioannidis JPA. Hundreds of thousands of zombie randomised trials circulate among us. Anaesthesia. 2021;76(4):444-447. doi:10.1111/anae.15297

3.         Pfeifer MP, Snodgrass GL. The continued use of retracted, invalid scientific literature. JAMA. 1990;263(10):1420-1423.

4.         Furman JL, Jensen K, Murray F. Governing knowledge in the scientific community: Exploring the role of retractions in biomedicine. Res Policy. 2012;41(2):276-290. doi:10.1016/j.respol.2011.11.001

5.         Brainard J, You J. What a massive database of retracted papers reveals about science publishing’s ‘death penalty’. Science. Published online October 25, 2018. Accessed January 13, 2022. https://www.science.org/content/article/what-massive-database-retracted-papers-reveals-about-science-publishing-s-death-penalty

6.         Marcus A, Oransky I. What studies of retractions tell us. J Microbiol Biol Educ. 2014;15(2):151-154. doi:10.1128/jmbe.v15i2.855

7.         Meddings K. Encouraging even greater reporting of corrections and retractions. Published March 30, 2020. Accessed January 12, 2022. https://www.crossref.org/blog/encouraging-even-greater-reporting-of-corrections-and-retractions/

8.         Suelzer EM, Deal J, Hanus KL, Ruggeri B, Sieracki R, Witkowski E. Assessment of Citations of the Retracted Article by Wakefield et al With Fraudulent Claims of an Association Between Vaccination and Autism. JAMA Netw Open. 2019;2(11):e1915552. doi:10.1001/jamanetworkopen.2019.15552

9.         Buranyi S. Is the staggeringly profitable business of scientific publishing bad for science? The Guardian. https://www.theguardian.com/science/2017/jun/27/profitable-business-scientific-publishing-bad-for-science. Published June 27, 2017.

10.       Berenbaum MR. On zombies, struldbrugs, and other horrors of the scientific literature. Proc Natl Acad Sci U S A. 2021;118(32). doi:10.1073/pnas.2111924118

11.       COPE Council. COPE Retraction Guidelines - English. Committee on Publication Ethics; 2019. Accessed January 20, 2022. https://doi.org/10.24318/cope.2019.1.4

12.       Schneider J, Woods ND, Proescholdt R, Fu Y, The RISRS Team. Recommendations from the Reducing the Inadvertent Spread of Retracted Science: Shaping a Research and Implementation Agenda Project. MetaArXiv. Published online July 29, 2021. doi:https://doi.org/10.31222/osf.io/ms579

13.       Time to break academic publishing’s stranglehold on research. NewScientist. Published November 21, 2018. https://www.newscientist.com/article/mg24032052-900-time-to-break-academic-publishings-stranglehold-on-research/

14.       Bar-Ilan J, Halevi G. Post retraction citations in context: a case study. Scientometrics. 2017;113(1):547-565. doi:10.1007/s11192-017-2242-0