Are Chatbots Trustworthy? A New Tool Helps Simplify Their Evaluation

As AI tools such as ChatGPT become a regular part of our daily routines, we’re encountering AI chatbots online more often. But are we embracing their presence, or resisting it?

Researchers at Binghamton University are exploring these questions with the help of VizTrust, a new analytics tool designed to shed light on how trust develops and shifts in human-AI interactions.

VizTrust is being developed by Xin “Vision” Wang, a Ph.D. candidate at the School of Systems Science and Industrial Engineering within the Thomas J. Watson College of Engineering and Applied Science. Her work on the project is part of her doctoral dissertation.

New Tool Unveiled at CHI 2025 to Track Real-Time Shifts in Trust Toward AI

Wang presented her latest research and findings in April at the Association for Computing Machinery’s CHI 2025 conference in Yokohama, Japan. The paper appears in the Proceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems.

The creators developed the tool to address a key challenge: Trust in AI agents is fluid, heavily influenced by context, and difficult to measure using conventional approaches. “Most research depends on post-interaction surveys, which only reveal a user’s trust level before and after engaging with AI,” Wang explained. “They overlook the real-time cues that indicate how and why trust shifts throughout the conversation.”

To overcome this limitation, VizTrust assesses trust through four key dimensions rooted in social psychology: competence, benevolence, integrity, and predictability. It uses machine learning and NLP to analyze user messages for trust signals, such as emotional tone, engagement, and politeness. This allows the tool to visually track how trust evolves throughout the interaction.

“The capabilities of large language models and generative AI are expanding rapidly, but we still need to understand the user experience across different conversational platforms,” Wang noted. “If we don’t closely examine what causes a negative interaction, we can’t truly identify how to improve the AI model.”

Groundbreaking Tool Launched at CHI 2025 to Monitor Real-Time Shifts in AI Trust

The research paper demonstrates VizTrust in action through a scenario involving a software engineer feeling overwhelmed by work and a therapeutic chatbot aimed at supporting employees. As they talk through his job-related stress, the chatbot offers coping strategies.

By detecting subtle changes in language and behavior, VizTrust highlights key moments that either strengthen or diminish trust. For instance, it identifies a drop in trust when the chatbot repeats advice that the user finds unhelpful. These insights are crucial for advancing academia and improving the design and responsiveness of conversational AI systems.

“Trust isn’t just a user problem—it’s a system-level challenge,” Wang said. “VizTrust helps developers, researchers, and designers pinpoint trust breakdowns for targeted AI improvements.”

VizTrust has already received recognition as a late-breaking work at CHI 2025. Judges selected it from over 3,000 global submissions, with a competitive acceptance rate of just under 33%.

A Personalized Approach to AI Trust

The project is co-authored by SSIE Assistant Professors Sadamori Kojaku and Stephanie Tulk Jesso, Associate Professor David M. Neyens from Clemson University, and Professor Min Sun Kim from the University of Hawaii at Manoa.

Wang is now advancing VizTrust to be more adaptable to individual users.

“People approach AI with different mindsets,” she noted. “Understanding trust may require a personalized approach, factoring in traits, baseline trust, and past AI interactions.”

Looking ahead, Wang aims to make VizTrust publicly accessible as an online tool to support wider research and development.

“By opening access to VizTrust,” she said, “we can bridge the gap between technical efficiency and human experience, making AI more user-centered and ethical.”

Read the original article on: Tech Xplore

Read more: ChatGPT Found to Be More Persuasive Than Humans in Online Arguments