Research Uncovers AI Influence in Academic Papers
A graduate student at the University of Tennessee, Knoxville, investigating the performance of ChatGPT found evidence that scholars quickly embraced the technology when writing peer-reviewed articles on political science topics.
Shortly after the artificial intelligence program was introduced in November 2022, use of the top words it favors jumped higher than the word “terrorism” did in journal articles following September 11, 2001, according to R. Paul Keener.
Between 2001 and 2002, use of the word “terrorism” rose nearly 125%, while between 2022 and 2023 use of the top five words ChatGPT favors in generating responses rose more than 820%.
“The findings are absurdly high, and it is undeniable that a large number of academics are using ChatGPT in peer-reviewed articles,” said Keener. His research, “More A than I: Testing for Large Language Model Plagiarism in Political Science,” was published in PS: Political Science & Politics.
Keener calls for professional organizations to develop specific guidance for when the use of AI crosses the line to plagiarism. Even if scholars disclose that they used programs such as ChatGPT, he noted that large language models (LLMs) don’t usually cite their sources, which can lead to the programs unintentionally plagiarizing human work.
“Although many academic institutions have different definitions of plagiarism, when one defines it as simply ‘to steal and pass off (the ideas or words of another) as one’s own: use (another’s production) without crediting the source’ (Merriam-Webster 2019) the line between plagiarism and acceptable use of LLMs is clear,” he wrote.
“ChatGPT is fed large amounts of data that it replicates, doing little work on its own, and giving no/incorrect citations, making it essentially a plagiarism machine,” Keener said. “Plagiarism cheapens our entire profession and, left unchecked, will lead to a credibility gap among all academics that threatens all of our careers.”
Exploring AI
Keener is pursuing a PhD in political science with a concentration on international relations. He was preparing to study coding for work on his dissertation when ChatGPT was released.
“I became an early adopter of ChatGPT, spending days stress-testing it to see what it was capable of,” he said. “I felt it important to seize the utility of ChatGPT and other LLMs while it is still an emerging technology.”
Knowing that ChatGPT strongly prefers certain words such as delve, vibrant, and realm, he went to the OpenAlex website, which catalogs academic articles, and used its search tools to check for how often those words appeared in political science papers.
Studying International Affairs
Keener is writing his doctoral dissertation on international formal alliances and recognizes LLMs as a great tool for uses such as identifying cases that meet certain criteria and assisting in brainstorming.
His interest in international affairs started early. “My aunt circumnavigated the world on her sailboat and sent me letters from several countries,” he said. “Having that kind of international perspective at that young age made me generally curious about other cultures.”
Keener earned his master’s degree in China studies while living in China and teaching English as a second language. When he returned to the US and decided to pursue a PhD, he applied only to UT. “Both my parents went here, and I knew the Department of Political Science had a strong international relations program,” he said.
By Amy Beth Miller