Optimistic Transfer under Task Shift via Bellman Alignment

TL;DR

This paper introduces Bellman alignment and a re-weighted targeting method for transfer reinforcement learning, enabling effective reuse of source task data despite transition mismatches, with theoretical regret bounds and empirical validation.

Contribution

It proposes Bellman alignment as a new abstraction for transfer in online RL and develops RWT, a correction operator that improves transfer across tasks with transition differences.

Findings

RWT reduces task mismatch to a one-step correction.

Regret bounds scale with task shift complexity, not the target MDP.

Empirical results show consistent improvements over baseline methods.

Abstract

We study online transfer reinforcement learning (RL) in episodic Markov decision processes, where experience from related source tasks is available during learning on a target task. A fundamental difficulty is that task similarity is typically defined in terms of rewards or transitions, whereas online RL algorithms operate on Bellman regression targets. As a result, naively reusing source Bellman updates introduces systematic bias and invalidates regret guarantees. We identify one-step Bellman alignment as the correct abstraction for transfer in online RL and propose re-weighted targeting (RWT), an operator-level correction that retargets continuation values and compensates for transition mismatch via a change of measure. RWT reduces task mismatch to a fixed one-step correction and enables statistically sound reuse of source data. This alignment yields a two-stage RWT $Q$-learning framework that separates variance reduction from bias correction. Under RKHS function approximation, we establish regret bounds that scale with the complexity of the task shift rather than the target MDP. Empirical results in both tabular and neural network settings demonstrate consistent improvements over single-task learning and na\"{i}ve pooling, highlighting Bellman alignment as a model-agnostic transfer principle for online RL.