Learning to Solve Voxel Building Embodied Tasks from Pixels and Natural Language Instructions

TL;DR

This paper introduces a novel approach combining language models and reinforcement learning to enable embodied agents to build objects in a Minecraft-like environment based on natural language instructions, verifying action feasibility and relevance.

Contribution

It presents a new method that generates achievable sub-goals from instructions and completes sub-tasks with a pre-trained RL policy, advancing embodied task learning.

Findings

Formed the RL baseline at IGLU 2022 competition

Effectively generates achievable sub-goals from natural language instructions

Successfully completes sub-tasks with pre-trained RL policy

Abstract

The adoption of pre-trained language models to generate action plans for embodied agents is a promising research strategy. However, execution of instructions in real or simulated environments requires verification of the feasibility of actions as well as their relevance to the completion of a goal. We propose a new method that combines a language model and reinforcement learning for the task of building objects in a Minecraft-like environment according to the natural language instructions. Our method first generates a set of consistently achievable sub-goals from the instructions and then completes associated sub-tasks with a pre-trained RL policy. The proposed method formed the RL baseline at the IGLU 2022 competition.