Who watches the (AI) watchmen?

Long before ChatGPT arrived, algorithms were already reading what people write and passing judgment. Now, these AI-backed tools are everywhere we look online: They’re deciding which emails land in a spam folder, which product reviews look fake, which posts get flagged for moderation and which pages rise to the top of a search.

These tools can shape lives in unexpected ways. For example, testing organizations have long used machine learning models to help score essays on exams such as the GRE and TOEFL. These models are  thought to sidestep many human biases, like guessing where a test-taker is from and marking them down for it. As with spam filters and fake-review detectors, these scoring models learned their trade from a single source: human writing.

Then the machines started writing, too.

“The question is, what happens when GenAI crashes the party?” asked Ahmed Abbasi, the Joe and Jane Giovanini Professor of IT, Analytics, and Operations at the University of Notre Dame’s Mendoza College of Business.

New research from Mendoza offers an unsettling answer: The judges may prefer the GenAI newcomer. In a paper published in ACM Transactions on Information Systems, Abbasi and Mendoza doctoral students Kezia Oketch, Marialena Bevilacqua, Will Stamey, Xinyuan Zhang, and their co-authors found that the most advanced essay-scoring models gave higher marks to essays written by GPT, even though they’d been trained only on human essays. The researchers trace that preference to what they call a “familiarity bias,” and they say its implications reach well beyond the classroom.

“The question is, what happens in the future, when the AI we’ve been using to protect us against illicit actors is now susceptible to this familiarity bias?” Abbasi said. “In other words, who watches the watchmen?”

A Ph.D. Seminar Meets ChatGPT

ChatGPT was released in 2020, which was the year Mendoza’s Ph.D. in Analytics program was approved and about three months before Abbasi was set to teach a doctoral seminar on designing human-centered AI. After more than 20 years working in natural language processing, he found himself unsure of where the technology was headed next or what impact it would have.

Rather than having each student pursue a separate project, the cohort decided to write one paper together. Oketch stepped up as a lead author and kept the work moving through rounds of revision long after the semester ended.

The students homed in on a shift that was easy to miss amid the excitement over chatbots: AI had long been used to judge text, and now it could also produce it. What were the implications of that duality?

“These AI tools are now playing two roles,” Oketch said. “The question that we wanted to pursue is, what happens when those two roles collide?”

Essay scoring offered a controlled place to find out, with set prompts, rubrics and scores assigned by expert graders. The team built an analysis framework with three parts: The first was a lineup of scoring models spanning several generations of technology, including older feature-based models, an earlier wave of deep learning models, and models called BERT and RoBERTa, which are the state of the art for scoring text and close cousins of the technology behind ChatGPT, as well as large language model-based assessment methods. All were trained only on human essays.

The second part of the analysis framework was the essays themselves: 15,437 written by U.S. students in grades 7 through 10 and by English language learners, plus 3,023 written by GPT-3.5 and GPT-4 in response to the same 68 prompts, with no coaching on how to write them. The third aspect of the framework was a statistical model that compared scores according to who wrote an essay, which model judged it and which of six genres it fell into.

Preferring What’s Familiar

The older models behaved as one might expect. Feature-based models scored the human essays 10% to 15% higher than GPTs, and the earlier deep learning models favored human writing by 26% to 32%. Because GPTs prose was unlike anything they had been trained on, they marked it down.

But the transformer models (that is, BERT and RoBERTA) went the other way.

Transformers, introduced in 2017, are a newer kind of AI model. Where older models tally up features of a text or read it a few words at a time, a transformer takes in a whole passage at once and weighs how each word relates to the others. It also arrives having already read vast amounts of text. This is the design that powers ChatGPT; in fact, GPT stands for Generative Pre-trained Transformer.

In the study, the newer models scored essays written by GPT-3.5 roughly 10% to 20% higher than essays by people. (As a robustness check, the researchers re-ran the analysis with essays written by GPT-4 and found the same pattern, though the gap was narrower.)

“Despite never being trained on these GenAI essays, they tend to like them,” Abbasi said.

The result caught the team off guard. The transformer scorers had been built on vast amounts of human writing and then fine-tuned on human essays specifically.

“Our expectation was that, because they were trained on human data and were even fine-tuned on it, they’re probably going to prefer that,” Oketch said. “And then we run these experiments and we find that they actually prefer GPT-generated content!”

While it’s worth noting that the human essays came from writers still developing their skills, and Oketch acknowledged that more experienced writers might well have earned higher scores overall, she doesn’t expect the gap in how the two types of models treated human and machine writing would have looked any different.

A Bias Toward the Familiar

Why would a model schooled on human essays prefer a machine’s? The paper weighs a few possibilities, and the explanation the researchers find most compelling is familiarity.

Before a transformer model (like BERT) is ever shown a student essay, it is “pre-trained” on enormous amounts of text, including Wikipedia, books and web pages. GPT models learned from much of the same material. So when GPT writes an essay, it draws on words and phrasing that the scoring model has, in a sense, met before.

“Even though there are phrases or terms they’ve never seen before, they like them because they technically have seen them in their pre-training,” Abbasi said.

To test the idea, the team looked under the hood. Transformer models work by assigning “attention” to the words that matter most to their judgment. When the researchers tracked where BERT’s attention went, they found it gravitated toward certain words that appeared only in GPT’s essays, such as the names of authors and characters. Those words never came up in the human essays BERT was trained to score, but they were likely part of its pre-training.

Oketch pointed to a related factor: The scorers that favored GPT writing share its underlying transformer design, and the ones that favored humans do not.

“In a nutshell, I would say both matter: the architecture of a system that is used to judge content, as well as the data it is pre-trained on,” she said.

The preference wasn’t uniform across genres. The transformer scorers favored GPT most on narrative, argumentative and short-response prompts, while humans roughly held their own on letters, suggestions and commentary. 

Testing the Watchmen

For Abbasi, the finding lands on familiar ground. He has spent much of his career studying adversarial settings, from fraud detection on Wall Street to illicit activity on the dark web. 

Until recently, these “cat-and-mouse games,” as he called them, were played between people, and the defenders generally held the better tools. But generative AI makes convincing content cheap to produce, and a familiarity bias adds a new wrinkle: Machine-written text might get past a filter because the filter finds it familiar.

“It’s a brave new world,” Abbasi said. “It’s unclear what’s going to happen, and it’s a bit scary.”

The researchers are clear about the study’s limits. It examined one setting — essay scoring — using GPT-3.5 and GPT-4, and generative models have advanced considerably since. It doesn’t prove that machine-written content is already getting a boost in search results, recommendation feeds or fake-news detection.

Still, Oketch said, the study sends up an important flag about the mechanism through which these barrier breaches could happen and provides a framework for testing for them.

Oketch said organizations that build systems to judge content in similar settings could use the researchers’ framework to check how those systems treat human and machine writing before releasing them.

The reason to make the effort, she added, is that someone is on the receiving end of every one of these judgments.

“The final consumer of this content is not AI,” Oketch said. “The final consumer is a human being.”