Improved Bounds for Private and Robust Alignment

TL;DR

This paper establishes new theoretical bounds for private and robust language model alignment, demonstrating near-optimal algorithms and improved guarantees in offline and online settings under privacy and adversarial corruption.

Contribution

It provides the first theoretical analysis of private and robust online alignment, along with improved offline bounds and new uniform convergence guarantees under privacy and corruption.

Findings

Log loss with MLE achieves near-optimal rates under privacy.

Existing offline algorithms offer stronger guarantees than previously known.

First results for private and robust online alignment.

Abstract

In this paper, we study the private and robust alignment of language models from a theoretical perspective by establishing upper bounds on the suboptimality gap in both offline and online settings. We consider preference labels subject to privacy constraints and/or adversarial corruption, and analyze two distinct interplays between them: privacy-first and corruption-first. For the privacy-only setting, we show that log loss with an MLE-style algorithm achieves near-optimal rates, in contrast to conventional wisdom. For the joint privacy-and-corruption setting, we first demonstrate that existing offline algorithms in fact provide stronger guarantees -- simultaneously in terms of corruption level and privacy parameters -- than previously known, which further yields improved bounds in the corruption-only regime. In addition, we also present the first set of results for private and robust online alignment. Our results are enabled by new uniform convergence guarantees for log loss and square loss under privacy and corruption, which we believe have broad applicability across learning theory and statistics.