Offline Reinforcement Learning of High-Quality Behaviors Under Robust Style Alignment

TL;DR

This paper introduces SCIQL, a novel offline reinforcement learning framework that effectively balances style alignment and task performance using a unified behavior style definition and innovative optimization techniques.

Contribution

It proposes a unified style definition and the SCIQL algorithm, combining goal-conditioned RL with a Gated Advantage Weighted Regression for improved style and task performance.

Findings

SCIQL outperforms prior offline methods in style and task metrics.

The framework effectively balances style preservation with high task performance.

Experimental results validate the superiority of SCIQL across various benchmarks.

Abstract

We study offline reinforcement learning of style-conditioned policies using explicit style supervision via subtrajectory labeling functions. In this setting, aligning style with high task performance is particularly challenging due to distribution shift and inherent conflicts between style and reward. Existing methods, despite introducing numerous definitions of style, often fail to reconcile these objectives effectively. To address these challenges, we propose a unified definition of behavior style and instantiate it into a practical framework. Building on this, we introduce Style-Conditioned Implicit Q-Learning (SCIQL), which leverages offline goal-conditioned RL techniques, such as hindsight relabeling and value learning, and combine it with a new Gated Advantage Weighted Regression mechanism to efficiently optimize task performance while preserving style alignment. Experiments demonstrate that SCIQL achieves superior performance on both objectives compared to prior offline methods. Code, datasets and visuals are available in: https://sciql-iclr-2026.github.io/.