Collapsing Sequence-Level Data-Policy Coverage via Poisoning Attack in Offline Reinforcement Learning

TL;DR

This paper introduces a sequence-level coverage measure in offline RL, reveals its exponential impact on errors, and demonstrates a poisoning attack that significantly degrades agent performance by targeting rare decision patterns.

Contribution

It proposes the sequence-level concentrability coefficient, analyzes its effect on errors, and develops a poisoning attack that exploits multi-step decision patterns to reduce data coverage.

Findings

Poisoning 1% of data can reduce performance by 90%.

Sequence-level coverage exponentially affects estimation errors.

The attack targets rare decision patterns to cause coverage collapse.

Abstract

Offline reinforcement learning (RL) heavily relies on the coverage of pre-collected data over the target policy's distribution. Existing studies aim to improve data-policy coverage to mitigate distributional shifts, but overlook security risks from insufficient coverage, and the single-step analysis is not consistent with the multi-step decision-making nature of offline RL. To address this, we introduce the sequence-level concentrability coefficient to quantify coverage, and reveal its exponential amplification on the upper bound of estimation errors through theoretical analysis. Building on this, we propose the Collapsing Sequence-Level Data-Policy Coverage (CSDPC) poisoning attack. Considering the continuous nature of offline RL data, we convert state-action pairs into decision units, and extract representative decision patterns that capture multi-step behavior. We identify rare patterns likely to cause insufficient coverage, and poison them to reduce coverage and exacerbate distributional shifts. Experiments show that poisoning just 1% of the dataset can degrade agent performance by 90%. This finding provides new perspectives for analyzing and safeguarding the security of offline RL.