New Study Reveals Key Differences Between Human and AI-Generated Text

Carnegie Mellon University researchers set out to test how well large language models (LLMs) can replicate human writing styles. Their findings, recently published in the Proceedings of the National Academy of Sciences, highlight key differences between AI-generated and human-written text.

“Humans adapt their writing style based on context—sometimes formal, sometimes informal,” explained Alex Reinhart, lead author and associate teaching professor in the Department of Statistics & Data Science. “What we found is that LLMs, like ChatGPT and Llama, tend to write in a fixed style rather than adjusting to different contexts. Their writing remains distinct from human-written text, and we were able to quantify this in a way that hadn’t been done before.”

To analyze these differences, the researchers prompted LLMs with excerpts from various genres, including TV scripts and academic articles. Using a custom-coded analysis tool developed by co-author David West Brown, an associate teaching professor in the Department of English, they identified significant grammatical, lexical, and stylistic variations between AI and human writing.

Instruction-tuned models, such as ChatGPT, showed the most pronounced differences. These models, which undergo additional training to improve question-answering and instruction-following abilities, used present participle clauses two to five times more frequently than human writers. For example, GPT-4o produced sentences like, “Bryan, leaning on his agility, dances around the ring, evading Show’s heavy blows.”

AI Text Structure: Denser, Noun-Heavy, and Less Adaptable

Additionally, LLMs relied on nominalizations 1.5 to two times more than humans and used agentless passive voice half as often. This suggests that AI-generated text tends to be denser, more noun-heavy, and less adaptable across writing styles.

Instruction-tuned LLMs overused certain words at striking rates. ChatGPT favored “camaraderie” and “tapestry” 150 times more than human writers, while Llama models frequently used “unease.” Both leaned on words like “palpable” and “intricate.”

Brown highlighted concerns for writing education, noting that writers revise constantly, whereas LLMs generate one-off responses. He cautioned that AI is fine for tasks like doctor’s notes but may not be suitable for personal writing, such as job applications.

Reinhart warned about students using LLMs for assignments, arguing they differ from calculators, which perform the same math humans would. LLMs generate writing that isn’t just automated but fundamentally different.

Ph.D. student Ben Markey is studying whether LLMs provide consistent essay scores. The team aims to expand its research to better understand AI-generated text and its role in writing and education.

Read Original Article: TechXplore

Read More: AI Breakthrough Uncovers Hidden Signals in the Universe