The use of AI to produce phony scientific papers is emerging as a big problem that threatens critical medical research and hurts the credibility of academic publishing, researchers warn.
Computing noted last year that tools like ChatGPT were helping do a very large number of research papers, often without proper acknowledgment.
A striking increase in the use of words such as commendable, innovative, meticulous, lucidly, and impressively appeared in recent scientific papers, Gray from UCL commented. This trend, he noted, is a hallmark of output generated by LLMs. Lack of clarity concerning the use of these tools, Gray also indicated, will ultimately result in trust erosion about academic publications.
A new study by researchers at Retraction Watch, Université Grenoble Alpes, and Institut de Recherche en Informatique de Toulouse shows that what we see is just the tip of a big, bad problem. Phony research papers from so-called paper mills have become a fast-growing industry that poses an ever-greater risk to the honesty of academic work.
The academic publishing industry has been penetrated by fraudsters who put profit above scholarly integrity, says The Conversation. Those technological skills, plus an ability to be flexible and a network of undishonest researchers, are helping them create papers across an entire spectrum from obscure genes to artificially intelligent applications in medicine.
"Much like highly partisan websites masquerading as impartial news are undermining evidence-based journalism and jeopardizing elections, phony science is eroding the foundation of knowledge that underpins modern society."
Scammers typically use GenAI to create seemingly plausible content. Scholars used a tool called Problematic Paper Screener to scan the work for “tortured phrases,” like “creepy crawlies” instead of just saying insects and “preparation information” instead of training data. Those tortured phrases are used strategically to escape plain-vanilla plagiarism, which is easier to detect.
Another sign of GenAI trickery exists.
An organization called Retraction Watch, which tracks the retraction of academic journal articles, noticed in several papers the strange phrase ‘vegetative electron microscopy’, including in some published by big-name journals like Springer Nature. It also appears in another paper that is still up and accessible on Elsevier’s ScienceDirect, co-written by Vijay Kumar Thakur, who works as a special content editor for Elsevier.
"It is a unique trait found in certain publications which can possibly indicate finger-print paper mill involvement," said Frederik Joelving, contributing editor at Retraction Watch.
Where does the term 'vegetative electron microscopy' come from, seeing that it is not widely acknowledged?
A Russian chemist named "Paralabrax clathratus" and a software engineer from Kazakhstan, Alexander Magazinov, both think they know the meaning of the phrase that first piqued their interest.
This was attributed to the possibly continued failure of an AI model in recognizing the two-column format, which is still widely used in academic publications. A paper from 1959 was found that has "vegetative" in the left-hand column and "electron microscopy" in the right. Perhaps the model saw the column divider as just a part of regular text.
The burgeoning fraudulent enterprise modernized by contemporary instruments, alongside the escalating pressure on academics to produce publishable work, compromises the sanctity of scholarship. The tsunami of submissions makes it nigh impossible for editors and reviewers to keep on top of things. Publishers claim that no more than 2% of submitted papers are fraudulent. In reality, it is probably much higher, especially in sectors such as biology, medicine, and computer science.
Papers are being retracted at unprecedented levels, and in some cases, entire journals have been closed because of the high volume of fraudulent content from paper mills.
This matters a lot. It’s a fake paper, and one that really holds up progress in medical research — it’s specifically hurting cancer studies.
It’s a problem that’s only going to get worse as the pressures to publish increase and the ways to create fraud become more sophisticated.
