Ask & Explore: Grounded Question Answering for Curiosity-Driven Exploration

TL;DR

This paper introduces a novel curiosity-driven exploration method using grounded question answering, enabling agents to ask and learn about specific environmental aspects to improve performance on sparse-reward tasks.

Contribution

It proposes a structured curiosity framework based on natural language questions, allowing agents to acquire targeted environmental knowledge for better exploration.

Findings

Questions reveal physical properties and spatial relationships of objects.

Structured curiosity improves efficiency in sparse-reward tasks.

Language-based questions guide targeted exploration.

Abstract

In many real-world scenarios where extrinsic rewards to the agent are extremely sparse, curiosity has emerged as a useful concept providing intrinsic rewards that enable the agent to explore its environment and acquire information to achieve its goals. Despite their strong performance on many sparse-reward tasks, existing curiosity approaches rely on an overly holistic view of state transitions, and do not allow for a structured understanding of specific aspects of the environment. In this paper, we formulate curiosity based on grounded question answering by encouraging the agent to ask questions about the environment and be curious when the answers to these questions change. We show that natural language questions encourage the agent to uncover specific knowledge about their environment such as the physical properties of objects as well as their spatial relationships with other objects, which serve as valuable curiosity rewards to solve sparse-reward tasks more efficiently.