Can You Break RLVER? Probing Adversarial Robustness of RL-Trained Empathetic Agents

TL;DR

This paper evaluates the robustness of RL-trained empathetic language models against adversarial emotional interactions, introducing a new benchmark and score to measure their resilience.

Contribution

It constructs the Adversarial Empathy Benchmark (AEB) and the Emotional Consistency Score (ECS) to assess empathetic robustness under adversarial conditions.

Findings

RLVER-PPO-Think outperforms baseline models in empathetic response quality.

Training improves emotional responsiveness but not internal state tracking.

ECS remains nearly flat, indicating a dissociation between behavior and internal understanding.

Abstract

Reinforcement learning from verifiable emotion rewards RLVER has produced language models with strong empathetic performance, evaluated on benchmarks that assume cooperative, honest users. Yet real emotional interactions systematically violate this assumption: users gaslight, escalate, and pressure AI systems for unconditional validation, dynamics that cooperative benchmarks cannot surface. We construct the Adversarial Empathy Benchmark AEB and introduce the Emotional Consistency Score ECS to evaluate empathetic robustness under adversarial conditions. AEB comprises six psychologically grounded adversarial trajectory types with discriminative reward structures that penalize formulaic responses; ECS formally disentangles a model's capacity to track user emotional states from its capacity to improve them. In a controlled experiment across eight scenario-matched conditions (think and no-think conditions on 2 RLVER models, and 2 base models (Qwen 1.5B and 7B) with 480 adversarial dialogues), RLVER-PPO-Think substantially outperforms the same-scale untuned baseline (0.963 vs. 0.761, \(p<0.001, r=0.688\)), with zero dialogue collapses and 47\% higher hidden-intention detection. However, ECS remains nearly flat and is not significantly different for RLVER-PPO-Think versus Base-7B-Think (\(p=0.650\)): RL training improves emotional responsiveness without measurable gains in observable state tracking. We interpret the ECS--FS (Final Score) gap as a behavioral/legibility dissociation inside this simulator family, not as evidence about internal understanding or clinical readiness.