The first statistically significant findings are in: Large Language Model (LLM) AIs are not only capable of generating expert-level scientific research ideas, but their ideas are also deemed more innovative and exciting than those of human experts, according to expert evaluations.
Recent advancements in LLMs have sparked enthusiasm among researchers about their potential to transform scientific discovery. Models like ChatGPT and Anthropic’s Claude have demonstrated the ability to autonomously produce and validate new research concepts.
This was one of the many tasks people believed AI could never surpass humans in—generating new knowledge and making scientific discoveries, rather than simply combining existing information from their training data.
Multimodal AI: A New Frontier in Original Research
Yet, much like artistic creation, music composition, coding, and even understanding subtext or body language, today’s multimodal AIs seem capable of producing truly original research—often more innovative than human-generated ideas.
Until recently, no research had explored this potential. In a new study, over 100 natural language processing (NLP) experts (PhDs and postdocs from 36 respected institutions) competed against AI-driven “ideation agents.” Their research ideas were judged on originality, excitement, and feasibility by human experts.
Natural Language Processing (NLP) is a branch of AI focused on facilitating communication between humans and machines, allowing both to “understand” each other—not just in terms of basic syntax, but also with regard to nuance, tone, and emotional inflection.
In the study, 49 human experts developed ideas on seven NLP topics, while an AI model trained by researchers generated ideas on the same subjects. To motivate the participants, the study offered $300 per idea and a $1,000 bonus for the top five human-generated concepts, encouraging clear, actionable proposals.
Standardizing Submissions
After all entries were submitted, an LLM was employed to standardize the writing style of each submission while maintaining the original content. This approach aimed to create a more level playing field and ensure the study remained as blind as possible.
A panel of 79 human experts reviewed all the submissions in a blind assessment, providing a total of 298 evaluations, with each idea receiving between two and four independent reviews. The results showed that AI-generated ideas outperformed those of human researchers in terms of novelty and excitement. Although AIs ranked slightly lower in feasibility and slightly higher in effectiveness compared to humans, these differences were not statistically significant.
Chenglei Si
Identifying Limitations
The study revealed some weaknesses in LLMs, such as a lack of idea diversity and difficulties with self-assessment. Despite being instructed not to repeat itself, the LLM often did so. Additionally, LLMs struggled with consistency when reviewing and scoring ideas, showing low agreement with human judgments.
The study also notes that judging the “originality” of an idea is subjective, even among experts. To further explore whether LLMs are truly better suited for autonomous scientific discovery, the researchers plan to involve more expert participants in a follow-up study. This time, the ideas from both AI and humans will be fully developed into projects to assess their real-world impact.
The Unreliability of Advanced Language Models
These initial findings are certainly eye-opening. Humanity now faces an unusual challenge from highly capable language model AIs. While these models can accomplish remarkable feats, they remain unreliable and prone to what AI companies term “hallucinations“—or what others might call fabrications.
Though AIs can handle vast amounts of paperwork, the scientific method requires rigor, and there’s no place for “hallucinations.” It’s already concerning that estimates suggest AIs are co-authoring at least 10% of research papers.
On the flip side, we can’t ignore AI’s potential to accelerate progress, as seen with DeepMind’s GNoME system, which condensed 800 years’ worth of materials discovery into months, producing recipes for 380,000 new inorganic crystals with potential for revolutionary applications.
As the fastest-evolving technology, many of AI’s current flaws could be fixed in the coming years. Some researchers even believe we’re nearing general superintelligence, where AIs will surpass expert knowledge in most fields.
Watching AIs quickly master skills we once thought defined human uniqueness, including the generation of novel ideas, is a strange experience. Human ingenuity seems to be edging humans out, but for now, the best path forward is a partnership between organic and artificial intelligence, as long as we align our goals.
If this were a competition, it would be AI: 1, humans: 0 for this round.
Read the original article on: New Atlas
Read more: Who’d Have Thought Robotic Bee Swarms Could be so Captivating?
