Generalisation in Multitask Fitted Q-Iteration and Offline Q-learning

TL;DR

This paper provides theoretical analysis of offline multitask reinforcement learning, showing how shared low-rank representations improve generalization and data efficiency in value function estimation.

Contribution

It introduces a multitask fitted Q-iteration method with finite-sample guarantees, highlighting the benefits of data pooling and shared representations for offline RL.

Findings

Pooling data across tasks improves estimation accuracy.

Shared representations reduce complexity in downstream tasks.

Finite-sample guarantees depend on total samples, horizon, and coverage.

Abstract

We study offline multitask reinforcement learning in settings where multiple tasks share a low-rank representation of their action-value functions. In this regime, a learner is provided with fixed datasets collected from several related tasks, without access to further online interaction, and seeks to exploit shared structure to improve statistical efficiency and generalization. We analyze a multitask variant of fitted Q-iteration that jointly learns a shared representation and task-specific value functions via Bellman error minimization on offline data. Under standard realizability and coverage assumptions commonly used in offline reinforcement learning, we establish finite-sample generalization guarantees for the learned value functions. Our analysis explicitly characterizes how pooling data across tasks improves estimation accuracy, yielding a $1/\sqrt{nT}$ dependence on the total number of samples across tasks, while retaining the usual dependence on the horizon and concentrability coefficients arising from distribution shift. In addition, we consider a downstream offline setting in which a new task shares the same underlying representation as the upstream tasks. We study how reusing the representation learned during the multitask phase affects value estimation for this new task, and show that it can reduce the effective complexity of downstream learning relative to learning from scratch. Together, our results clarify the role of shared representations in multitask offline Q-learning and provide theoretical insight into when and how multitask structure can improve generalization in model-free, value-based reinforcement learning.