GRILLBot: An Assistant for Real-World Tasks with Neural Semantic Parsing and Graph-Based Representations

TL;DR

GRILLBot is a multimodal voice assistant that uses neural semantic parsing and graph-based representations to guide users through complex, real-world tasks like cooking and home improvement, demonstrating advanced understanding and adaptability.

Contribution

It introduces a Neural Decision Parser and TaskGraph representations, enabling flexible, context-aware task execution and multimodal task enrichment in a real-world assistant system.

Findings

Won the 2022 Alexa Prize TaskBot Challenge

Supports conditional execution and multimodal task enrichment

Demonstrates effective handling of complex, long-running tasks

Abstract

GRILLBot is the winning system in the 2022 Alexa Prize TaskBot Challenge, moving towards the next generation of multimodal task assistants. It is a voice assistant to guide users through complex real-world tasks in the domains of cooking and home improvement. These are long-running and complex tasks that require flexible adjustment and adaptation. The demo highlights the core aspects, including a novel Neural Decision Parser for contextualized semantic parsing, a new "TaskGraph" state representation that supports conditional execution, knowledge-grounded chit-chat, and automatic enrichment of tasks with images and videos.