Conceptual Reinforcement Learning for Language-Conditioned Tasks

TL;DR

This paper introduces a conceptual reinforcement learning framework that learns invariant, concept-like representations for language-conditioned policies, significantly enhancing transferability and efficiency in unseen environments.

Contribution

It proposes a novel CRL framework with multi-level attention and mutual information constraints to improve generalization in language-conditioned RL tasks.

Findings

Improves training efficiency by up to 70%.

Enhances generalization to new environments by up to 30%.

Effective in challenging environments RTFM and Messenger.

Abstract

Despite the broad application of deep reinforcement learning (RL), transferring and adapting the policy to unseen but similar environments is still a significant challenge. Recently, the language-conditioned policy is proposed to facilitate policy transfer through learning the joint representation of observation and text that catches the compact and invariant information across environments. Existing studies of language-conditioned RL methods often learn the joint representation as a simple latent layer for the given instances (episode-specific observation and text), which inevitably includes noisy or irrelevant information and cause spurious correlations that are dependent on instances, thus hurting generalization performance and training efficiency. To address this issue, we propose a conceptual reinforcement learning (CRL) framework to learn the concept-like joint representation for language-conditioned policy. The key insight is that concepts are compact and invariant representations in human cognition through extracting similarities from numerous instances in real-world. In CRL, we propose a multi-level attention encoder and two mutual information constraints for learning compact and invariant concepts. Verified in two challenging environments, RTFM and Messenger, CRL significantly improves the training efficiency (up to 70%) and generalization ability (up to 30%) to the new environment dynamics.