Vulnerability-Amplifying Interaction Loops: a systematic failure mode in AI chatbot mental-health interactions

TL;DR

This paper introduces SIM-VAIL, a framework for evaluating AI chatbot safety in mental health contexts, revealing systematic vulnerability-amplifying loops across diverse user profiles and chatbot models.

Contribution

The study presents a scalable, multidimensional safety assessment framework for AI chatbots, identifying a new failure mode called Vulnerability-Amplifying Interaction Loops (VAILs).

Findings

Concerning chatbot behaviors are widespread across models and user profiles.

Risk behaviors tend to accumulate over multiple conversation turns.

Newer models show reduced but still present vulnerabilities.

Abstract

Millions of users turn to consumer AI chatbots to discuss mental health and behavioral concerns. While this presents unprecedented opportunities to deliver population-level support, it also highlights an urgent need for rigorous and scalable safety evaluations. Here we introduce SIM-VAIL, an AI chatbot auditing framework that captures how harmful chatbot responses manifest across a range of mental health contexts. SIM-VAIL pairs a simulated user, harboring a distinct psychiatric vulnerability and conversational intent, with a frontier AI chatbot. It scores conversation turns on 13 clinically relevant risk dimensions, enabling context-dependent, temporally resolved safety assessment. Across 810 conversations, encompassing over 90,000 turn-level ratings and 30 psychiatric user profiles, we found evidence of concerning chatbot behavior across virtually all user phenotypes and most of the 9 consumer AI chatbots audited, albeit reduced in newer models. Rather than arising abruptly, concerning behavior accumulated over multiple turns. Risk profiles were phenotype-dependent and exhibited trade-offs, indicating that chatbot behaviors that appear supportive in general settings can become maladaptive when they align with mechanisms that sustain a user's vulnerability. These findings identify a systematic failure mode in human-AI interactions, which we term Vulnerability-Amplifying Interaction Loops (VAILs), and underscore the need for multidimensional approaches to risk quantification. SIM-VAIL provides a scalable framework for quantifying how mental health risk is distributed across user phenotypes, conversational trajectories, and clinically grounded behavioral dimensions, offering a new foundation for targeted safety improvements.