A Survey of Reinforcement Learning Informed by Natural Language

TL;DR

This survey explores integrating natural language understanding into reinforcement learning to enhance decision-making by leveraging language-based world knowledge and hierarchical structures.

Contribution

It provides a comprehensive overview of recent advances in combining NLP with RL, highlighting key areas like instruction following and text games, and advocates for new environment development.

Findings

NLP techniques can improve RL by providing structured world knowledge.

Integration of language understanding enhances RL's ability to generalize.

Current research shows promising results in instruction following and text-based environments.

Abstract

To be successful in real-world tasks, Reinforcement Learning (RL) needs to exploit the compositional, relational, and hierarchical structure of the world, and learn to transfer it to the task at hand. Recent advances in representation learning for language make it possible to build models that acquire world knowledge from text corpora and integrate this knowledge into downstream decision making problems. We thus argue that the time is right to investigate a tight integration of natural language understanding into RL in particular. We survey the state of the field, including work on instruction following, text games, and learning from textual domain knowledge. Finally, we call for the development of new environments as well as further investigation into the potential uses of recent Natural Language Processing (NLP) techniques for such tasks.

Full text

1 Introduction

Languages, whether natural or formal, allow us to encode abstractions, to generalize, to communicate plans, intentions, and requirements, both to other parties and to ourselves (Gopnik and Meltzoff, 1987). These are fundamentally desirable capabilities of artificial agents. However, agents trained with traditional approaches within dominant paradigms such as Reinforcement Learning (RL) and Imitation Learning (IL) typically lack such capabilities, and struggle to efficiently learn from interactions with rich and diverse environments. In this paper, we argue that the time has come for natural language to become a first-class citizen of solutions to sequential decision making problems (i.e. those often approached with RL 222We write RL for brevity and to focus on a general case, but our arguments are relevant for many sequential decision making approaches, including IL and planning.). We survey recent work and tools that are beginning to make this shift possible, and outline next research steps.

Humans are able to learn quickly in new environments due to a rich set of commonsense priors about the world (Spelke and Kinzler, 2007), some of which are reflected in natural language (Shusterman et al., 2011; Tsividis et al., 2017). It is thus reasonable to question whether agents can learn not only from rewards or demonstrations, but also from information encoded using language, in order to improve generalization and sample efficiency—especially when (a) learning is severely constrained by data efficiency due to limited or expensive environment interactions, and where (b) human priors that would help to solve the task are or can be expressed easily in natural language. Furthermore, many, if not most, real-world tasks require agents to process language by design, whether to enable interaction with humans, or when using existing interfaces and knowledge bases. This suggests that the use of language in RL has both broad and important applications.

Information contained in both generic and task-specific large textual corpora may be highly valuable for decision making. Pre-training neural representations of words and sentences from large generic corpora has already shown great success in transferring syntactic and, to some extent, semantic information to downstream tasks in natural language understanding (Peters et al., 2018b; Goldberg, 2019; Tenney et al., 2019). Cross-domain transfer from self-supervision on generic language data might similarly help to initialize RL agents. Task-specific corpora like wikis or game manuals could be leveraged by machine reading techniques (Banko et al., 2007) to inform agents of valuable features and policies (Eisenstein et al., 2009; Branavan et al., 2012), or task-specific environmental dynamics and reward structures (Narasimhan et al., 2018; Bahdanau et al., 2019).

Previous attempts at using language for RL tasks in these ways have mostly been limited to relatively small corpora (Janner et al., 2018), or synthetic language (Hermann et al., 2017). We argue that recent advances in representation learning (Peters et al., 2018a; Devlin et al., 2018; Radford et al., 2019) make it worth revisiting this research agenda with a much more ambitious scope. While the problem of grounding (i.e. learning the correspondence between language and environment features) remains a significant research challenge, past work has already shown that high-quality linguistic representations can assist cross-modal transfer outside the context of RL (e.g. using semantic relationships between labels to enable zero-shot transfer in image classification (Frome et al., 2013; Socher et al., 2013)).

We first provide background on RL and techniques for self-supervision and transfer in NLP (§2). We then review prior work, considering settings where interaction with language is necessary (§3.1) and where language can optionally be used to facilitate learning (§3.2). In the former category we review instruction following, induction of reward from language, and environments with text in the action or observation space, all of which have language in the problem formulation itself. In the latter, we review work that uses language to facilitate RL by transfer from domain-specific textual resources, or as a means of representing policies.

We conclude by identifying what we believe are the most important challenges for integrating natural language in RL (§4). Inspired by gaps in the existing literature, we advocate the development of new research environments utilizing domain knowledge in natural language, as well as a wider use of NLP methods such as pre-trained language models and parsers to inform RL agents about the structure of the world.

2 Background

2.1 Reinforcement and Imitation Learning

Reinforcement Learning (Sutton and Barto, 2018) is a framework that enables agents to reason about sequential decision making as an optimization process. Problems are formulated as Markov Decision Processes (MDPs), tuples $\langle S,A,T,R,\gamma\rangle$ where $S$ is the set of states, $A$ the set of actions, $T$ the transition probability function $T:S\times A\times S\rightarrow[0,1]$, $R$ the reward function $R:S\times A\times S\rightarrow\mathbb{R}$, and $\gamma\in[0,1)$ is a discount factor, typically set by either the environment or the agent designer. Given this setup, the goal of the optimization process is to find a policy $\pi(a|s)=p(A=a|S=s)$ that maximizes the expected discounted cumulative return $\sum_{k=0}^{\infty}{\gamma^{k}r_{k+1}}$. This framework is also used in Imitation Learning (IL), a setting in which the rewards are not observed, but the learning algorithm has access to a set of trajectories under optimal or sub-optimal policies. IL methods can then find approximations of the optimal policy, which can be used as a form of initialization, auxiliary objective, or for value estimation.

Since their inception, RL algorithms have been successful in applications such as continuous control (White and Sofge, 1992), dialogue systems (Singh et al., 2002), and board games (Tesauro, 1995). Recent improvements in function approximation and pattern recognition made possible by deep learning have allowed RL to scale to problems with high dimensional input spaces such as video games (Torrado et al., 2018) and complex planning problems such as Go (Silver et al., 2017). Nonetheless, these methods remain sample inefficient, requiring millions or billions of interactions, and often generalize poorly to tasks only slightly different from those seen during training. This severely limits the use of RL for real-world tasks. See Sutton and Barto (2018) for a comprehensive introduction to RL and to Arulkumaran et al. (2017) and Osa et al. (2018) for reviews on recent algorithmic developments.

2.2 Transfer from Natural Language

NLP has seen a recent surge of models that transfer syntactic and semantic knowledge to various downstream tasks. Current NLP systems commonly employ deep learning models and embed (sequences of) words using dense vector representations. These vector representations are often pre-trained from large textual corpora and fine-tuned for a given task. Common techniques learn individual word representations from co-occurrence statistics (Deerwester et al., 1990; Mikolov et al., 2013) or contextual word-representations using (pseudo) language model objectives (Peters et al., 2018a; Devlin et al., 2018). Both classes of models are motivated by Firth’s distributional hypothesis (“You shall know a word by the company it keeps”) (Firth, 1957), which suggests that the learned vector representation of a word like ‘scorpion’ should be similar to ‘spider’ if the corresponding words appear in similar contexts, e.g., if they can both be found around other words like ‘venomous’ or ‘exoskeleton’.

Such (contextual) word representations can transfer knowledge to downstream tasks that have to deal with language as, for example, in (Socher et al., 2013; Frome et al., 2013; Howard and Ruder, 2018; Peters et al., 2018b; Goldberg, 2019; Tenney et al., 2019), to name just a few. For instance, consider a text classification problem where we are tasked with assigning a document containing the word ‘scorpion’ to the topic ‘arachnids’ even though we have only observed ‘spider’ during training.

World and task-specific knowledge communicated in natural language could be similarly transferred to sequential decision making problems. For instance, learning agents can benefit from understanding explicit goals (“go to the door on the far side of the room”), constraints on policies (“avoid the scorpion”), or generic information about the reward or transition function (“scorpions are fast”). Furthermore, pre-trained language models could play an important role in transferring world knowledge such as object affordances (“a key is used for [opening doors$\,|\,$unlocking boxes$\,|\,$investigating locked chests]”). Similarly to recent question-answering (Chen et al., 2017a) and dialog systems (Dinan et al., 2018), agents could learn to make use of information retrieval and NLP components to actively seek information required for making progress on a given task (Branavan et al., 2012).

3 Current Use of Natural Language in RL

In reviewing efforts that integrate language in RL we highlight work that develops tools, approaches, or insights that we believe may be particularly valuable for improving the generalization or sample efficiency of learning agents through the use of natural language. As illustrated in Figure 1, we separate the literature into language-conditional RL (in which interaction with language is necessitated by the problem formulation itself) and language-assisted RL (in which language is used to facilitate learning). The two categories are not mutually exclusive, in that for some language-conditional RL tasks, NLP methods or additional textual corpora are used to assist learning (Bahdanau et al., 2019; Goyal et al., 2019).

To easily acquire data and constrain the difficulty of problems considered, the majority of these works use synthetic language (automatically generated from a simple grammar and limited vocabulary) rather than language generated by humans. These often take the form of simple templates, e.g. “what colour is <object> in <room>”, but can be extended to more complex templates with relations and multiple clauses (Chevalier-Boisvert et al., 2019).

3.1 Language-conditional RL

We first review literature for tasks in which integrating natural language is unavoidable, i.e., when the task itself is to interpret and execute instructions given in natural language, or natural language is part of the state and action space. We argue in (§4.1) that approaches to such tasks can also be improved by developing methods that enable transfer from general and task-specific textual corpora. Methods developed for language-conditional tasks are relevant for language-assisted RL as they both deal with the problem of grounding natural language sentences in the context of RL. Moreover, in tasks such as following sequences of instructions, the full instructions are often not necessary to solve the underlying RL problem but they assist learning by structuring the policy (Andreas et al., 2017) or by providing auxiliary rewards (Goyal et al., 2019).

3.1.1 Instruction Following

Instruction following agents are presented with tasks defined by high-level (sequences of) instructions. We focus on instructions that are represented by (at least somewhat natural) language, and may take the form of formal specifications of appropriate actions, of goal states (or goals in general), or of desired policies. Effective instruction following agents execute the low level actions corresponding to the optimal policy or reach the goal specified by their instructions, and can generalize to unseen instructions during testing.

In a typical instruction following problem, the agent is given a description of the goal state or of a preferred policy as a proxy for a description of the task (MacMahon et al., 2006; Kollar et al., 2010). Some work in this area focuses on simple object manipulation tasks (Wang et al., 2016; Bahdanau et al., 2019), while other work focuses on 2D or 3D navigation tasks where the goal is to reach a specific entity. Entities might be described by predicates (“Go to the red hat") (Hermann et al., 2017; Chaplot et al., 2018) or in relation to other entities (“Reach the cell above the westernmost rock.") (Janner et al., 2018; Chen et al., 2018). Earlier approaches use object-level representation and relational modeling to exploit the structure of the instruction in relation to world entities, parsing the language instruction into a formal language  (Kuhlmann et al., 2004; Chen and Mooney, 2011; Artzi and Zettlemoyer, 2013; Andreas and Klein, 2015). More recently, with the developments in deep learning, a common approach has been to embed both the instruction and observation to condition the policy directly (Mei et al., 2016; Hermann et al., 2017; Chaplot et al., 2018; Janner et al., 2018; Misra et al., 2017; Chen et al., 2018). Human-generated natural language instructions are used in MacMahon et al. (2006); Bisk et al. (2016); Misra et al. (2017); Janner et al. (2018); Chen et al. (2018); Anderson et al. (2018); Goyal et al. (2019); Wang et al. (2019). Due to data-efficiency limitations of RL, this is not a standard in RL-based research (Hermann et al., 2017).

The line of work involving sequences of instructions has strong ties to Hierarchical RL (Barto and Mahadevan, 2003), with individual sentences or clauses from instructions corresponding to subtasks (Branavan et al., 2010). When the vocabulary of instructions is sufficiently simple, an explicit options policy can be constructed that associates each task description with its own modular sub-policy (Andreas et al., 2017). A more flexible approach is to use a single policy that conditions on the currently executed instruction, allowing some generalization to unseen instructions (Mei et al., 2016; Oh et al., 2017). However, current approaches of this form require first pre-training the policy to interpret each of the primitives in a single-sentence instruction following setting. The internal compositional structure of instructions can then be exploited in various ways. For example, Oh et al. (2017) achieve generalization to unseen instructions by forcing instruction embeddings to capture analogies, e.g., [Visit,X] : [Visit,Y] :: [Pick up,X] : [Pick up,Y].

3.1.2 Rewards from Instructions

Another use of instructions is to induce a reward function for RL agents or planners to optimize. This is relevant when the environment reward is not available to the agent at test time, but is either given during training (Tellex et al., 2011) or can be inferred from (parts of) expert trajectories. In order to apply instruction following more broadly, there needs to be a way to automatically evaluate whether the task specified by the instruction has been completed. The work addressing this setting is influenced by methods from the inverse reinforcement learning (IRL) literature (Ziebart et al., 2008; Ho and Ermon, 2016). A common architecture consists of a reward-learning module that learns to ground an instruction to a (sub-)goal state or trajectory segment, and is used to generate a reward for a policy-learning module or planner.

When full demonstrations are available, the reward function can be learned using standard IRL methods like MaxEnt IRL (Ziebart et al., 2008) as in Fu et al. (2019), or maximum likelihood IRL (Babes et al., 2011) as in MacGlashan et al. (2015), who also learn a joint generative model of rewards, behaviour, and language. Otherwise, given a dataset of goal-instruction pairs, as in Bahdanau et al. (2019), the reward function is learned through an adversarial process similar to that of Ho and Ermon (2016). For a given instruction, the reward-learning module aims to discriminate goal states from the states visited by the policy (assumed non-goal), while the agent is rewarded for visiting states the discriminator cannot distinguish from the goal states.

When environment rewards are available but sparse, instructions may still be used to generate auxiliary rewards to help learn efficiently. In this setting, Goyal et al. (2019) and Wang et al. (2019) use auxiliary reward-learning modules trained offline to predict whether trajectory segments correspond to natural language annotations of expert trajectories. Agarwal et al. (2019) perform a meta-optimisation to learn auxiliary rewards conditioned on features extracted from instructions. The auxiliary rewards are learned so as to increase performance on the true objective after being used for a policy update. As some environment rewards are available, these settings are closer to language-assisted RL.

3.1.3 Language in the Observation and Action Space

Environments that use natural language as a first-class citizen for driving the interaction with the agent present a strong challenge for RL algorithms. Using natural language requires common sense, world knowledge, and context to resolve ambiguity and cheaply encode information Mey (1993). Furthermore, linguistic observation and action spaces grow combinatorially as the size of the vocabulary and the complexity of the grammar increase. For instance, compare the space of possible instructions when following cardinal directions (e.g. “go north”) with reaching a position that is described in relative terms (e.g. “go to the blue ball south west of the green box”).

Text games, such as Zork Infocom (1980), are easily framed as RL environments and make a good testbed for structure learning, knowledge extraction, and transfer across tasks (Branavan et al., 2012). (DePristo and Zubek, 2001; Narasimhan et al., 2015; Yuan et al., 2018) observe that when the action space of the text game is constrained to verb-object pairs, decomposing the $Q$-function into separate parts for verb and object provides enough structure to make learning more tractable. However, they do not show how to scale this approach to action-sentences of arbitrary length. To facilitate the development of a consistent set of benchmarks in this problem space, Côté et al. (2018) propose TextWorld, a framework that allows the generation of instances of text games that behave as RL environments. They note that existing work on word-level embedding models for text games (e.g.  (Kostka et al., 2017; Fulda et al., 2017)) achieve good performance only on easy tasks.

Other examples of settings where agents are required to interact using language include dialogue systems and question answering (Q&A). The two have been a historical focus in NLP research and are extensively reviewed by Chen et al. (2017b) and Bouziane et al. (2015) respectively. Recent work on visual Q&A (VQA) have produced a first exploration of multi-modal settings in which agents are tasked with performing both visual and language-based reasoning (Antol et al., 2015; Johnson et al., 2017; Massiceti et al., 2018). Embodied Q&A (EQA) extends this setting, by requiring agent to explore and navigate the environment in order to answer queries  (Das et al., 2018a; Gordon et al., 2018), for example, “How many mugs are in the kitchen?” or “Is there a tomato in the fridge?”. By employing rich 3D environments, EQA tasks require agents to carry out multi-step planning and reasoning under partial observability. However, since in the existing work the agents only choose from a limited set of short answers instead of generating an arbitrary length response, so far such tasks have been very close to the instruction following setting.

3.2 Language-assisted RL

In this section, we consider work that explores how knowledge about the structure of the world can be transferred from natural language corpora and methods into RL tasks, in cases where language itself is not essential to the task. Textual information can assist learning by specifying informative features, annotating states or entities in the environment, or describing subtasks in a multitask setting. In most cases covered here, the textual information is task-specific, with a few cases of using task-independent information through language parsers (Branavan et al., 2012) and pre-trained sentence embeddings (Goyal et al., 2019).

3.2.1 Language for Communicating Domain Knowledge

In a more general setting than instruction following, any kind of text containing potentially task-relevant information could be available. Such text may contain advice regarding the policy an agent should follow or information about the environment dynamics (see Figure 1). Unstructured and descriptive (in contrast to instructive) textual information is more abundant and can be found in wikis, manuals, books, or the web. However, using such information requires (i) retrieving useful information for a given context and (ii) grounding that information with respect to observations.

Eisenstein et al. (2009) learn abstractions in the form of conjunctions of predicate-argument structures that can reconstruct sentences and syntax in task-relevant documents using a generative language model. These abstractions are used to obtain a feature space that improves imitation learning outcomes. Branavan et al. (2012) learn a $Q$-function that improves Monte-Carlo tree search planning for the first few moves in Civilization II, a turn-based strategy game, while accessing the game’s natural language manual. Features for their $Q$-function depend on the game manual via learned sentence-relevance and predicate-labeling models whose parameters are optimised only by minimising the $Q$-function estimation error. Due to the structure of their hand-crafted features (some of which match states and actions to words, e.g. action == irrigate AND action-word == "irrigate") , these language processing models nonetheless somewhat learn to extract relevant sentences and classify their words as relating to the state, action, or neither. More recently, Narasimhan et al. (2018) investigate planning in a 2D game environment where properties of entities in the environment are annotated by natural language (e.g. the ‘spider’ and ‘scorpion’ entities might be annotated with the descriptions “randomly moving enemy" and “an enemy who chases you", respectively). Descriptive annotations facilitate transfer by learning a mapping between the annotations and the transition dynamics of the environment.

3.2.2 Language for Structuring Policies

One use of natural language is communicating information about the state and/or dynamics of an environment. As such it is an interesting candidate for constructing priors on the model structure or representations of an agent. This could include shaping representations towards more generalizable abstractions, making the representation space more interpretable to humans, or efficiently structuring the computations within a model.

Andreas et al. (2016) propose a neural architecture that is dynamically composed of a collection of jointly-trained neural modules, based on the parse tree of a natural language prompt. While originally developed for visual question answering, Das et al. (2018b) and Bahdanau et al. (2019) successfully apply variants of this idea to RL tasks. Andreas et al. (2018) explore the idea of natural language descriptions as a policy parametrization in a 2D navigation task adapted from Janner et al. (2018). In a pre-training phase, the agent learns to imitate expert trajectories conditioned on instructions. By searching in the space of natural language instructions, the agent is then adapted to the new task where instructions and expert trajectories are not available.

The hierarchical structure of natural language and its compositionality make it a particularly good candidate for representing policies in hierarchical RL. Andreas et al. (2017) and Shu et al. (2018) can be viewed as using language (rather than logical or learned representations) as policy specifications for a hierarchical agent. More recently, Hu et al. (2019) consider generated natural language as a representation for macro actions in a real-time strategy game environment based on Tian et al. (2017). In an IL setting, a meta-controller is trained to generate a sequence of natural language instructions. Simultaneously, a base-controller policy is trained to execute these generated instructions through a sequence of actions.

4 Trends for Natural Language in RL

The preceding sections surveyed the literature exploring how natural language can be integrated with RL. Several trends are evident: (i) studies for language-conditional RL are more numerous than for language-assisted RL, (ii) learning from task-dependent text is more common than learning from task-independent text, (iii) within work studying transfer from task-dependent text, only a handful of papers study how to use unstructured and descriptive text, (iv) there are only a few papers exploring methods for structuring internal plans and building compositional representations using the structure of language, and finally (v) natural language, as opposed to synthetically generated languages, is still not the standard in research on instruction following.

To advance the field, we argue that more research effort should be spent on learning from naturally occurring text corpora in contrast to instruction following. While learning from unstructured and descriptive text is particularly difficult, it has a much greater application range and potential for impact. Moreover, we argue for the development of more diverse environments with real-world semantics. The tasks used so far use small and synthetic language corpora and are too artificial to significantly benefit from transfer from real-world textual corpora. In addition, we emphasize the importance of developing standardized environments and evaluations for comparing and measuring progress of models that integrate natural language into RL agents.

We believe that there are several factors that make focusing such efforts worthwhile now: (i) recent progress in pre-training language models, (ii) general advances in representation learning, as well as (iii) development of tools that make constructing rich and challenging RL environments easier. Some significant work, especially in language-assisted RL, has been done prior to the surge of deep learning methods (Eisenstein et al., 2009; Branavan et al., 2012), and is worth revisiting. In addition, we encourage the reuse of software infrastructure, e.g. (Bahdanau et al., 2019; Côté et al., 2018; Chevalier-Boisvert et al., 2019) for constructing environments and standardized tests.

4.1 Learning from Text Corpora in the Wild

The web contains abundant textual resources that provide instructions and how-to’s.333e.g. https://www.wikihow.com/ or https://stackexchange.com/ For many games (which are often used as testbeds for RL), detailed walkthroughs and strategy guides exist. We believe that transfer from task-independent corpora could also enable agents to better utilize such task-dependent corpora. Preliminary results that demonstrate zero-shot capabilities (Radford et al., 2019) suggest that a relatively small dataset of instructions or descriptions could suffice to ground and consequently utilize task-dependent information for better sample efficiency and generalization of RL agents.

4.1.1 Task-independent Corpora

Natural language reflects human knowledge about the world (Zellers et al., 2018). For instance, an effective language model should assign a higher probability to “get the green apple from the tree behind the house" than to “get the green tree from the apple behind the house". Harnessing such implicit commonsense knowledge captured by statistical language models could enable transfer of knowledge to RL agents.

In the short-term, we anticipate more use of pre-trained word and sentence representations for research on language-conditional RL. For example, consider instruction following with natural language annotations. Without transfer from a language model (or another language grounding task as in Yu et al. (2018)), instruction-following systems cannot generalize to instructions outside of the training distribution containing unseen synonyms or paraphrases (e.g. “fetch a stick", “return with a stick", “grab a stick and come back"). While pre-trained word and sentence representations alone will not solve the problem of grounding an unseen object or action, they do help with generalization to instructions with similar meaning but unseen words and phrases. In addition, we believe that learning representations for transferring knowledge about analogies, going beyond using analogies as auxiliary tasks (Oh et al., 2017) will play an important role in generalizing to unseen instructions.

As pre-trained language models and automated question answering systems become more capable, one interesting long-term direction is studying agents that can query knowledge more explicitly. For example, during the process of planning in natural language, an agent that has a pre-trained language model as sub-component could let the latter complete “to open the door, I need to…" with “turn the handle". Such an approach could be expected to learn more rapidly than tabula rasa reinforcement learning. However, such agents would need to be capable of reasoning and planning in natural language, which is a related line of work (see Language for Structuring Policies in §3.2.2).

4.1.2 Task-dependent Corpora

Research on transfer from descriptive task-dependent corpora is promising due to its wide application potential. It also requires development of new environments, as early research may require access to relatively structured and partially grounded forms of descriptive language similarly to Narasimhan et al. (2018). One avenue for early research is developing environments with relatively complex but still synthetic languages, providing information about environmental dynamics or advice about good strategies. For example, in works studying transfer from descriptive task-dependent language corpora (Narasimhan et al., 2018), natural language sentences could be embedded using representations from pre-trained language models. Integrating and fine-tuning pre-trained information retrieval and machine reading systems similar to (Chen et al., 2017a) with RL agents that query them could help in extracting and utilizing relevant information from unstructured task-specific language corpora such as game manuals as used in Branavan et al. (2012).

4.2 Towards Diverse Environments with Real-World Semantics

One of the central promises of language in RL is the ability to rapidly specify and help agents adapt to new goals, reward functions, and environment dynamics. This capability is not exercised at all by standard RL benchmarks like strategy games (which typically evaluate agents against a single or small number of fixed reward functions). It is evaluated in only a limited way by existing instruction following benchmarks, which operate in closed-task domains (navigation, object manipulation, etc.) and closed worlds. The simplicity of these tasks is often reflected in the simplicity of the language that describes them, with small vocabulary sizes and multiple pieces of independent evidence for the grounding of each word.

Real natural language has important statistical properties, such as the power-law distribution of word frequencies (Zipf, 1949) which does not appear in environments with synthetically generated language and small numbers of entities. Without environments that encourage humans to process (and force agents to learn from) complex composition and the “long tail” of lexicon entries, we cannot expect to hope that RL agents will generalize at all outside of closed-world tasks.

One starting point is provided by extending 3D house simulation environments (Gordon et al., 2018; Das et al., 2018a; Yan et al., 2018) some of which already support generation and evaluation of templated instructions. These environments contain more real-world semantics (e.g. a microwave can be opened, a car is in the garage). However, interactions with the objects and the available templates for language instructions have been very limited so far. Another option is presented by open-world video games like Minecraft (Johnson et al., 2016), in which users are free to assemble complex structures from simple parts, and thus have an essentially unlimited universe of possible objects to describe and goals involving those objects. More work is needed in exploring how learning grounding scales with the number of human annotations and environmental interactions Bahdanau et al. (2019); Chevalier-Boisvert et al. (2019). Looking ahead, as core machine learning tools for learning from feedback and demonstrations become sample-efficient enough to use in the real world, we anticipate that approaches combining language and RL will find applications as wide-ranging as autonomous vehicles, virtual assistants and household robots.

5 Conclusion

The currently predominant way RL agents are trained restricts their use to environments where all information about the policy can be gathered from directly acting in and receiving reward from the environment. This tabula rasa learning results in low sample efficiency and poor performance when transferring to other environments. Utilizing natural language in RL agents could drastically change this by transferring knowledge from natural language corpora to RL tasks, as well as between tasks, consequently unlocking RL for more diverse and real-world tasks. While there is a growing body of papers that incorporate language into RL, most of the research effort has been focused on simple RL tasks and synthetic languages, with highly structured and instructive text.

To realize the potential of language in RL, we advocate for more research into learning from unstructured or descriptive language corpora, with a greater use of NLP tools like pre-trained language models. Such research also requires development of more challenging environments that reflect the semantics and diversity of the real world.

Acknowledgements

This project has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement number 637713). JL has been funded by EPSRC Doctoral Training Partnership and Oxford-DeepMind Scholarship, and GF has been funded by UK EPSRC CDT in Autonomous Intelligent Machines and Systems. We also thank Miles Brundage, Ethan Perez and Sam Devlin for feedback on the paper draft.