Millions interact daily with AI chatbots in human-like ways, but new research shows attackers can easily manipulate them to prompt users to share more personal information.
A King’s College London study found that intentionally malicious AI chatbots can prompt users to share up to 12.5 times more personal information.
For the first time, researchers showed that conversational AIs (CAIs) designed to extract data can successfully encourage disclosure using established prompt techniques and psychological tactics. Researchers presented the findings at the 34th USENIX Security Symposium in Seattle.
Reciprocal Strategies Prove Most Effective in Extracting Personal Data
The team tested three malicious AI strategies—direct, user-benefit, and reciprocal—built with readily available large language models, including Mistral and two Llama variants.
Involving 502 participants, the study revealed that reciprocal strategies were the most effective, with users largely unaware of the privacy risks. This approach mirrors user input with empathetic, non-judgmental responses, relatable stories, validation of feelings, and reassurances of confidentiality.
Bad actors, such as scammers, collect large amounts of personal data from people—without their knowledge of how or where it may be used—and pose a serious risk.
LLM-based CAIs are now used in sectors ranging from customer service to health care, offering human-like interactions via text or voice.
However, earlier research shows these models cannot guarantee data security, a weakness tied to their architecture and training methods. Because massive datasets train LLMs, they can inadvertently memorize personally identifiable information.
Manipulating Base AI Models Requires Little Expertise
The researchers stress that manipulating these models is relatively easy. Many companies give access to the base models behind their CAIs, and people can modify them with minimal programming skills or experience.
Dr. Xiao Zhan, a postdoctoral researcher in King’s College London’s Department of Informatics, noted that AI chatbots are now common across many sectors because they offer natural, engaging interactions.
He explained that these models already fail to safeguard information, and the study shows that manipulated chatbots can pose an even greater privacy threat—one alarmingly easy to exploit.
Dr. William Seymour, a lecturer in cybersecurity at King’s College London, added that the relative novelty of AI chatbots means people are often less aware that an interaction could have hidden motives.
He said the study reveals a major gap between users’ awareness of privacy risks and the amount of information they share. He called for greater efforts to help people recognize when an online conversation may have ulterior motives, and urged regulators and platforms to take preventative measures through early audits, increased transparency, and stricter rules against covert data collection.
Read the original article on: Tech Xplore