Ask Before You Act: Generalising to Novel Environments by Asking Questions

TL;DR

This paper explores how reinforcement learning agents can improve their ability to generalize in new environments by learning to ask yes-no questions to an oracle, thereby gaining useful information and reasoning about environment dynamics.

Contribution

The study introduces a method for RL agents to learn natural language question-asking as a tool for better environment understanding and generalization in temporally-extended tasks.

Findings

Agents asking questions outperform baselines in novel environments.

Question asking enhances understanding of environment dynamics.

Grounded language understanding enables relevant question generation.

Abstract

Solving temporally-extended tasks is a challenge for most reinforcement learning (RL) algorithms [arXiv:1906.07343]. We investigate the ability of an RL agent to learn to ask natural language questions as a tool to understand its environment and achieve greater generalisation performance in novel, temporally-extended environments. We do this by endowing this agent with the ability of asking "yes-no" questions to an all-knowing Oracle. This allows the agent to obtain guidance regarding the task at hand, while limiting the access to new information. To study the emergence of such natural language questions in the context of temporally-extended tasks we first train our agent in a Mini-Grid environment. We then transfer the trained agent to a different, harder environment. We observe a significant increase in generalisation performance compared to a baseline agent unable to ask questions. Through grounding its understanding of natural language in its environment, the agent can reason about the dynamics of its environment to the point that it can ask new, relevant questions when deployed in a novel environment.