A Data Source for Reasoning Embodied Agents

TL;DR

This paper introduces a new data generator for reasoning tasks involving embodied agents, combining world-state data with text queries to evaluate and improve neural reasoning models.

Contribution

It presents a novel data generation method that integrates world dynamics and agent actions, facilitating research on neural reasoning and database representations.

Findings

Pre-trained language models can answer some questions about world-states.

Graph-structured Transformers show potential but struggle with complex queries.

The data generator enables new research directions in neural reasoning models.

Abstract

Recent progress in using machine learning models for reasoning tasks has been driven by novel model architectures, large-scale pre-training protocols, and dedicated reasoning datasets for fine-tuning. In this work, to further pursue these advances, we introduce a new data generator for machine reasoning that integrates with an embodied agent. The generated data consists of templated text queries and answers, matched with world-states encoded into a database. The world-states are a result of both world dynamics and the actions of the agent. We show the results of several baseline models on instantiations of train sets. These include pre-trained language models fine-tuned on a text-formatted representation of the database, and graph-structured Transformers operating on a knowledge-graph representation of the database. We find that these models can answer some questions about the world-state, but struggle with others. These results hint at new research directions in designing neural reasoning models and database representations. Code to generate the data will be released at github.com/facebookresearch/neuralmemory