Nebula: A discourse aware Minecraft Builder
Akshay Chaturvedi, Kate Thompson, Nicholas Asher

TL;DR
Nebula is a model that improves language-to-action tasks in a Minecraft environment by incorporating discourse and nonlinguistic context, significantly enhancing action prediction and understanding of shapes and locations.
Contribution
The paper introduces Nebula, a novel approach that leverages prior discourse and nonlinguistic context to improve language-to-action models in a Minecraft setting.
Findings
Doubles the net-action F1 score over baseline.
Successfully constructs shapes from descriptions.
Understands location descriptions effectively.
Abstract
When engaging in collaborative tasks, humans efficiently exploit the semantic structure of a conversation to optimize verbal and nonverbal interactions. But in recent "language to code" or "language to action" models, this information is lacking. We show how incorporating the prior discourse and nonlinguistic context of a conversation situated in a nonlinguistic environment can improve the "language to action" component of such interactions. We finetune an LLM to predict actions based on prior context; our model, Nebula, doubles the net-action F1 score over the baseline on this task of Jayannavar et al.(2020). We also investigate our model's ability to construct shapes and understand location descriptions using a synthetic dataset
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
Taxonomy
TopicsService-Oriented Architecture and Web Services
