Study Says AI Data Contaminates Vital Human Input

In the early 2000s, Jeff Bezos introduced the concept of mechanical turks, which involved hiring remote workers for menial tasks that computers found challenging. These workers were often paid low wages and collaborated with numerous others to complete small portions of larger computer projects.

Bezos coined the term “artificial intelligence” to describe this fusion of human and digital intelligence, aimed at incorporating a human perspective into tasks that computers struggled with, primarily those of a simpler nature.

Approximately 250,000 individuals are employed through Amazon’s Mechanical Turk marketplace, which is just one of several platforms offering similar services.

The Increasing Reliance of Human Workers on AI-Generated Content

Recently, researchers from the Swiss-based EPFL University highlighted a concerning trend: workers who had previously provided valuable human input were now relying on AI-generated content to complete their tasks. They coined the term “artificial intelligence” to describe this phenomenon.

While the term may elicit amusement, the researchers express serious concerns about the implications. They argue that if workers increasingly rely on AI generators to fulfill their tasks, it would significantly diminish the reliability of crowdsourced data.

Large language models (LLMs) excel at processing training data, but there are certain tasks for which human input remains superior. Humans are more efficient at labeling data for model input, describing images, and responding to CAPTCHA screens compared to computers.

The Potential Pitfalls of Crowdsourcing with Large Language Models

The temptation to use crowdsourcing to validate large language model outputs or create human gold-standard data can lead to a problem: What if crowd workers themselves use LLMs to boost their productivity and income on crowdsourcing platforms?

This scenario would contaminate the data pool, potentially undermining the reliability of AI-based operations.

The term “Turk” originates from an 18th-century chess-playing “robot” that defeated chess players throughout Europe, fooling them into believing they were playing against a machine. Today, crowdsourcing with “Turks” has become a billion-dollar industry, though its reputation has been marred by low wages.

However, the industry now faces a threat due to the rapid adoption of large language models. A recent study found that a ChatGPT 3.5 turbo model performed significantly better than crowd workers at a fraction of the cost.

As workers face mounting pressure to produce more output at a faster pace, they may increasingly rely on AI resources. The EPFL researchers estimated that 33% to 46% of worker assignments on Amazon’s Mechanical Turk platform were completed with the aid of large language models, based on a limited study.

The researchers caution that as large language models gain popularity and multimodal models supporting text, image, and video input and output emerge, steps must be taken to ensure that human data remains distinctly human. Their findings serve as a warning signal for platforms, researchers, and crowd workers to find new approaches to preserve the human element in data.

Read The Original Article On TechXplore.

Read More: AI Framework Improves Communication Analysis in Team Training.