Corruption-robust Offline Multi-agent Reinforcement Learning From Human Feedback

TL;DR

This paper develops robust algorithms for offline multi-agent reinforcement learning from human feedback that can withstand data corruption, providing theoretical guarantees under different coverage assumptions.

Contribution

It introduces the first systematic approach to handle adversarial data corruption in offline MARLHF with provable bounds and algorithms for various coverage settings.

Findings

Robust estimator guarantees an $O( ext{epsilon}^{1 - o(1)})$ bound under uniform coverage.

Proposed algorithms achieve an $O( ext{sqrt epsilon})$ bound under unilateral coverage.

A quasi-polynomial-time algorithm achieves an $O( ext{sqrt epsilon})$ CCE gap in the challenging setting.

Abstract

We consider robustness against data corruption in offline multi-agent reinforcement learning from human feedback (MARLHF) under a strong-contamination model: given a dataset $D$ of trajectory-preference tuples (each preference being an $n$-dimensional binary label vector representing each of the $n$ agents' preferences), an $\epsilon$-fraction of the samples may be arbitrarily corrupted. We model the problem using the framework of linear Markov games. First, under a uniform coverage assumption - where every policy of interest is sufficiently represented in the clean (prior to corruption) data - we introduce a robust estimator that guarantees an $O(\epsilon^{1 - o(1)})$ bound on the Nash equilibrium gap. Next, we move to the more challenging unilateral coverage setting, in which only a Nash equilibrium and its single-player deviations are covered. In this case, our proposed algorithm achieves an $O(\sqrt{\epsilon})$ bound on the Nash gap. Both of these procedures, however, suffer from intractable computation. To address this, we relax our solution concept to coarse correlated equilibria (CCE). Under the same unilateral coverage regime, we derive a quasi-polynomial-time algorithm whose CCE gap scales as $O(\sqrt{\epsilon})$. To the best of our knowledge, this is the first systematic treatment of adversarial data corruption in offline MARLHF.