Learning Invariable Semantical Representation from Language for Extensible Policy Generalization

TL;DR

This paper introduces a method to extract environment-agnostic, semantically invariant representations from language instructions using randomization, enabling hierarchical policies to generalize across varied and unseen tasks in reinforcement learning.

Contribution

The paper proposes element randomization to learn invariant semantic representations and develops a hierarchical policy framework for improved generalization in RL tasks.

Findings

Low-level policy generalizes across environment changes

Hierarchical policy generalizes to new, decomposable tasks

Language trajectories enable one-shot task completion

Abstract

Recently, incorporating natural language instructions into reinforcement learning (RL) to learn semantically meaningful representations and foster generalization has caught many concerns. However, the semantical information in language instructions is usually entangled with task-specific state information, which hampers the learning of semantically invariant and reusable representations. In this paper, we propose a method to learn such representations called element randomization, which extracts task-relevant but environment-agnostic semantics from instructions using a set of environments with randomized elements, e.g., topological structures or textures, yet the same language instruction. We theoretically prove the feasibility of learning semantically invariant representations through randomization. In practice, we accordingly develop a hierarchy of policies, where a high-level policy is designed to modulate the behavior of a goal-conditioned low-level policy by proposing subgoals as semantically invariant representations. Experiments on challenging long-horizon tasks show that (1) our low-level policy reliably generalizes to tasks against environment changes; (2) our hierarchical policy exhibits extensible generalization in unseen new tasks that can be decomposed into several solvable sub-tasks; and (3) by storing and replaying language trajectories as succinct policy representations, the agent can complete tasks in a one-shot fashion, i.e., once one successful trajectory has been attained.