According to a new study by researchers from Google DeepMind and University College London, large language models (LLMs) occasionally lack confidence when responding to questions and end up discarding correct answers.
Large language models (LLMs) are advanced AI tools designed to comprehend and produce human language. They’re widely used across sectors like finance, healthcare, and IT for tasks that involve reasoning and decision-making. Because of this, their performance must be both accurate and dependable, requiring consistent confidence in their responses. Yet, that confidence can falter.
To ensure safe deployment, LLMs provide confidence scores alongside their answers. However, how these scores influence their decision-making remains unclear.
In a study posted on the arXiv preprint server, researchers found that LLMs often start off overly confident in their answers but may lose confidence and revise their responses when confronted with a misleading counterargument.
Evaluating The Confidence Of Large Language Models (LLMs)
To explore this seeming contradiction, the researchers examined how large language models (LLMs) adjust their confidence and decide whether to revise their answers after receiving external input.
They began by giving a binary-choice question to an “answering LLM.” Once it responded, a second model an “advice LLM” provided guidance along with an accuracy score. This advice could support, contradict, or remain neutral toward the initial answer. The answering LLM was then asked to make a final decision. In some cases, the model could view its original response during this stage, while in others, it could not.
Research Results
The study revealed that LLMs are more likely to maintain their original answer when it’s visible to them, suggesting increased confidence. However, when the initial response is hidden, they are more prone to changing their answer. Additionally, the models tended to lose confidence and revise their decision more often when presented with opposing advice, compared to when the advice was supportive. This behavior was consistent across various models, including Gemma 3, GPT-4o, and o1-preview.
“Our findings show that LLMs diverge from ideal decision-making in notable ways,” the researchers explained. “First, they display a strong choice-supportive bias that reinforces their initial answers, even when faced with contradictory evidence. Second, although LLMs incorporate new information, they do so suboptimally placing too much weight on opposing advice, which leads to a significant drop in confidence in their original response.”
Improving AI Systems
This is important because a growing number of industries rely on LLMs. Yet, the research highlights that these models aren’t purely logical they exhibit their own biases and can be influenced. In extended interactions between a human and an AI, recent input may disproportionately shape the AI’s response.
Recognizing this behavior, along with other subtleties in how LLMs make decisions, is key to developing AI systems that are safer, more reliable, and better designed.
Read the original article on: Tech Xplore
Read more: Can AI Think Like Humans? New Research Reveals Behavior-Predicting Model