Assessing LLM Response Quality in the Context of Technology-Facilitated Abuse

TL;DR

This study evaluates the effectiveness of four large language models in responding to technology-facilitated abuse questions from survivors, using expert assessments and user feedback to identify strengths and limitations.

Contribution

First comprehensive expert and user evaluation of LLM responses to TFA-related questions, providing insights for future model development and domain-specific fine-tuning.

Findings

LLMs show potential but have notable limitations in TFA contexts.

Expert assessments highlight areas for response improvement.

User feedback emphasizes the importance of perceived actionability.

Abstract

Technology-facilitated abuse (TFA) is a pervasive form of intimate partner violence (IPV) that leverages digital tools to control, surveil, or harm survivors. While tech clinics are one of the reliable sources of support for TFA survivors, they face limitations due to staffing constraints and logistical barriers. As a result, many survivors turn to online resources for assistance. With the growing accessibility and popularity of large language models (LLMs), and increasing interest from IPV organizations, survivors may begin to consult LLM-based chatbots before seeking help from tech clinics. In this work, we present the first expert-led manual evaluation of four LLMs - two widely used general-purpose non-reasoning models and two domain-specific models designed for IPV contexts - focused on their effectiveness in responding to TFA-related questions. Using real-world questions collected from literature and online forums, we assess the quality of zero-shot single-turn LLM responses generated with a survivor safety-centered prompt on criteria tailored to the TFA domain. Additionally, we conducted a user study to evaluate the perceived actionability of these responses from the perspective of individuals who have experienced TFA. Our findings, grounded in both expert assessment and user feedback, provide insights into the current capabilities and limitations of LLMs in the TFA context and may inform the design, development, and fine-tuning of future models for this domain. We conclude with concrete recommendations to improve LLM performance for survivor support.