TabQL: In-Context Q-Learning with Tabular Foundation Models

TL;DR

TabQL introduces a novel reinforcement learning framework that replaces traditional Q-networks with a tabular foundation model capable of in-context learning, enabling rapid adaptation with limited data.

Contribution

It formalizes TabQL, analyzes its convergence, and demonstrates its improved efficiency over DQN through extensive benchmarks.

Findings

TabQL achieves faster learning with fewer interactions.

It effectively interpolates between classical Q-learning and DQN.

Experimental results show superior performance on benchmarks.

Abstract

We propose Tabular Q-Learning (TabQL), a reinforcement learning framework that replaces the conventional parametric Q-network in Deep Q-Learning (DQN) with a tabular foundation model endowed with in-context learning capabilities. The key idea is to represent Q-values through a sequence-to-sequence foundation model operating over a tabularized representation of state-action-Q-value tuples, enabling rapid adaptation from limited online interaction by conditioning on recent experience. TabQL departs from classical DQN by leveraging (i) zero- or few-shot Q-value inference via in-context updates, and (ii) a warm-up phase using standard DQN to bootstrap high-quality context. Particularly, to enhance the context quality, new transitions are generated by executing actions output by TabQL with predicted Q values from DQN. We formalize TabQL, analyze its convergence and sample complexity under mild assumptions, and show that TabQL interpolates between vanilla Q-learning and DQN with in-context learning. Our analysis demonstrates that TabQL achieves improved efficiency compared to DQN by amortizing Bellman updates through in-context learning. Extensive numerical experiments with several benchmarks showcase the effectiveness and efficacy of the proposed TabQL.