A Risk Ontology for Evaluating AI-Powered Psychotherapy Virtual Agents

TL;DR

This paper introduces a structured risk ontology for evaluating AI-powered psychotherapeutic virtual agents, aiming to improve safety and effectiveness in mental health applications by systematically identifying potential harms.

Contribution

It presents a novel risk ontology developed through literature review and expert input, tailored for assessing risks in conversational AI psychotherapists.

Findings

Developed a comprehensive risk ontology aligned with clinical criteria.

Outlined four practical use cases for the ontology in real-world settings.

Provides a foundation for safer AI-driven mental health interventions.

Abstract

The proliferation of Large Language Models (LLMs) and Intelligent Virtual Agents acting as psychotherapists presents significant opportunities for expanding mental healthcare access. However, their deployment has also been linked to serious adverse outcomes, including user harm and suicide, facilitated by a lack of standardized evaluation methodologies capable of capturing the nuanced risks of therapeutic interaction. Current evaluation techniques lack the sensitivity to detect subtle changes in patient cognition and behavior during therapy sessions that may lead to subsequent decompensation. We introduce a novel risk ontology specifically designed for the systematic evaluation of conversational AI psychotherapists. Developed through an iterative process including review of the psychotherapy risk literature, qualitative interviews with clinical and legal experts, and alignment with established clinical criteria (e.g., DSM-5) and existing assessment tools (e.g., NEQ, UE-ATR), the ontology aims to provide a structured approach to identifying and assessing user/patient harms. We provide a high-level overview of this ontology, detailing its grounding, and discuss potential use cases. We discuss four use cases in detail: monitoring real user interactions, evaluation with simulated patients, benchmarking and comparative analysis, and identifying unexpected outcomes. The proposed ontology offers a foundational step towards establishing safer and more responsible innovation in the domain of AI-driven mental health support.