Online Causal Kalman Filtering for Stable and Effective Policy Optimization

TL;DR

This paper introduces an online Kalman filtering approach to stabilize token-level importance sampling ratios in reinforcement learning for language models, improving policy optimization stability and effectiveness.

Contribution

It proposes a novel Online Causal Kalman Filtering method to model and update importance sampling ratios across tokens, addressing structural inconsistencies in previous approaches.

Findings

KPO outperforms state-of-the-art methods on math reasoning datasets.

The Kalman filter effectively smooths noise in importance ratios.

Token-wise structure-aware filtering improves training stability.

Abstract

Reinforcement learning for large language models suffers from high-variance token-level importance sampling (IS) ratios, which would destabilize policy optimization at scale. To improve stability, recent methods typically use a fixed sequence-level IS ratio for all tokens in a sequence or adjust each token's IS ratio separately, thereby neglecting temporal off-policy derivation across tokens in a sequence. In this paper, we first empirically identify that local off-policy deviation is structurally inconsistent at the token level, which may distort policy-gradient updates across adjacent tokens and lead to training collapse. To address the issue, we propose Online Causal Kalman Filtering for stable and effective Policy Optimization (KPO). Concretely, we model the desired IS ratio as a latent state that evolves across tokens and apply a Kalman filter to update this state online and autoregressively based on the states of past tokens, regardless of future tokens. The resulting filtered IS ratios preserve token-wise local structure-aware variation while strongly smoothing noise spikes, yielding more stable and effective policy updates. Experimentally, KPO achieves superior results on challenging math reasoning datasets compared with state-of-the-art counterparts.