Unifying Adversarial Robustness and Training Across Text Scoring Models

TL;DR

This paper unifies the study of adversarial robustness across various text scoring models, proposing new training methods that improve robustness and alignment in language models, especially in RLHF applications.

Contribution

It introduces a unified framework for adversarial robustness in text scoring models and develops new training methods that enhance robustness and task effectiveness.

Findings

Adversarial training methods improve robustness across models.

Combining training methods yields better generalization.

Adversarially trained reward models reduce reward hacking.

Abstract

Research on adversarial robustness in language models is currently fragmented across applications and attacks, obscuring shared vulnerabilities. In this work, we propose unifying the study of adversarial robustness in text scoring models spanning dense retrievers, rerankers, and reward models. This motivates adapting both attacks and adversarial training methods across model roles. Unlike open-ended generation, text scoring failures are directly testable: an attack succeeds when an irrelevant or rejected text outscores a relevant or chosen one. Using this principled lens of text scoring, we demonstrate that current adversarial training formulations for language models are often short-sighted, failing to effectively generalize across attacks. To address this, we introduce multiple adversarial training methods for text scoring models and show that combining complementary training methods can yield strong robustness while also improving task effectiveness. We also highlight the practical value of our approach for RLHF, showing that our adversarially trained reward models mitigate reward hacking and support the training of better-aligned LLMs. We provide our code and models for further study.