Scientific investigators discover dishonest use of ChatGPT in research

On August 9, the magazine Written Physics He published a paper aimed at revealing new solutions to a complex mathematical equation¹. It seemed real, but scientific investigator Guillaume Cabanac discovered a strange phrase on page three of the manuscript: “regeneration response”.

That phrase was the title of a button on ChatGPT, the free AI-powered chatbot that generates seamless text when users ask it a question. Cabanac, a computer scientist at the University of Toulouse in France, posted a screenshot of the page in question on PubPeer, a website where scientists discuss published research.

The authors have since confirmed to the journal that they used ChatGPT to help draft their manuscript, says Kim Eagleton, head of peer review and research integrity at IOP Publishing, Written PhysicsPublisher in Bristol, UK. This anomaly was not detected during two months of peer review (paper submitted in May, revised version sent in July) or during typesetting. The publisher has now decided to withdraw the paper, because the authors did not declare their use of the tool when it was submitted. “This is a violation of our ethics policies,” says Eagleton. He did not respond to the reporter, Abdullah Youssef, who is affiliated with Al-Biruni University in Istanbul and the Lebanese American University in Beirut. natureComment request.

“The tip of the iceberg”

This is not the only case in which a ChatGPT-assisted manuscript slips into a peer-reviewed journal unannounced. Since April, Cabanac has flagged more than a dozen journal articles with ChatGPT hashtags “response refresh” or “As an AI language model, I…” and published them on PubPeer. Several publishers, including Elsevier and Springer Nature, have said that authors can use ChatGPT and other large language modeling (LLM) tools to help them produce their manuscripts, as long as they declare it. (natureThe Springer Nature news team is editorially independent of its publisher, Springer Nature.)

Searching for key phrases only captures silly, unstated uses of ChatGPT — in which the authors forgot to omit the telltale signs — so the number of undisclosed, peer-reviewed papers created with the undisclosed help of ChatGPT is likely to be greater. much. “It’s just the tip of the iceberg,” says Cabanac. (The telltale signs are changing, too: ChatGPT’s “Regenerate Response” button earlier this year was changed to “Refresh” in an update to the tool.)

Cabanac discovered typical ChatGPT phrases in a few papers published in the Elsevier journals. The most recent is a paper published on August 3 in resource policy that explored the impact of e-commerce on fossil fuel efficiency in developing countries². Cabanac notes that some of the equations in the paper don’t make sense, but the result is above table: “Please note that as an AI language model, I am unable to create specific tables or run tests…”

An Elsevier spokesperson said nature The publisher is “aware of the problem” and is investigating it. The paper’s authors, at Liaoning University in Shenyang, China, and the Chinese Academy of International Trade and Economic Cooperation in Beijing, did not respond to the request. natureComment request.

Frightening fluency

Research papers written wholly or partly by computer programs, but without the authors revealing this fact, are not new. However, they usually contain subtle but detectable traces — such as certain patterns of language, or mistranslated “torment phrases” — that set them apart from their human-written counterparts, says Matt Hodgkinson, director of research integrity at the Office of Research Integrity. UK, headquartered in London. But if the researchers omit the ChatGPT benchmark, the smooth text of the most advanced chatbot is “nearly impossible” to detect, Hodgkinson says. “It’s basically an arms race – the scammers versus the people trying to turn them away,” he says.

Cabanac et al. also discovered an undisclosed use of ChatGPT (through cues) in peer-reviewed conference papers, and in preprints – manuscripts that have not been peer-reviewed. When these issues were brought up to PubPeer, the authors sometimes admitted that they used ChatGPT, undisclosedly, to help create the work.

Elizabeth Beck, a microbiologist and independent research integrity consultant in San Francisco, California, says the rapid rise of ChatGPT and other generative AI tools will give power to paper mills, companies that make and sell fake manuscripts to researchers looking to boost their publication output. . “It will exacerbate the problem a hundredfold,” says Beck. “I’m very concerned that we really have an influx of these papers that we just don’t recognize anymore.”

stretched to the limit

The problem of undisclosed papers that MBAs produce in journals points to a deeper problem, which is that extended peer reviewers often don’t have enough time to scrutinize manuscripts for red flags, says David Pemmler, who discovered the papers. Forged under the alias Smoot Clyde. “The whole scientific ecosystem is either published or destroyed,” says Pemmler, a retired psychologist formerly of Massey University in Palmerston North, New Zealand. “The number of gatekeepers cannot keep up.”

Hodgkinson says that ChatGPT and other MBAs tend to publish false references, which could be a signal to peer reviewers looking to discover the use of these tools in manuscripts. “If the reference isn’t there, that’s a red flag,” he says. For example, the Retraction Watch website reported an early edition of millipedes written using ChatGPT; One of the researchers who cited the work spotted it and noticed that its references were fake.

Rune Stensold, a microbiologist at the State Serum Institute in Copenhagen, ran into the problem of fake references when a student asked him for a copy of a paper Stensold appeared to have co-authored with a colleague in 2006. The paper did not come out. The student asked the AI ​​chatbot to suggest research papers Blastocystis – a genus of intestinal parasites – and the chatbot has compiled a reference with Stensold’s name on it. “It just felt so real,” he says. “It taught me that when I get papers to review, maybe I should start looking at the references section.”

Additional reporting by Chris Stockl-Walker.