Zero-Shot Context Generalization in Reinforcement Learning from Few Training Contexts

TL;DR

This paper introduces a novel approach called CEBE and CSE to improve the generalization of deep reinforcement learning policies across different contexts using minimal training data.

Contribution

The paper proposes the CEBE framework and CSE data augmentation method, enabling better generalization in DRL from limited context data.

Findings

CEBE provides a first-order approximation to multi-context Q-functions.

CSE enhances data efficiency in deterministic environments.

Numerical experiments validate improved generalization with CSE.

Abstract

Deep reinforcement learning (DRL) has achieved remarkable success across multiple domains, including competitive games, natural language processing, and robotics. Despite these advancements, policies trained via DRL often struggle to generalize to evaluation environments with different parameters. This challenge is typically addressed by training with multiple contexts and/or by leveraging additional structure in the problem. However, obtaining sufficient training data across diverse contexts can be impractical in real-world applications. In this work, we consider contextual Markov decision processes (CMDPs) with transition and reward functions that exhibit regularity in context parameters. We introduce the context-enhanced Bellman equation (CEBE) to improve generalization when training on a single context. We prove both analytically and empirically that the CEBE yields a first-order approximation to the Q-function trained across multiple contexts. We then derive context sample enhancement (CSE) as an efficient data augmentation method for approximating the CEBE in deterministic control environments. We numerically validate the performance of CSE in simulation environments, showcasing its potential to improve generalization in DRL.