TWIZ-v2: The Wizard of Multimodal Conversational-Stimulus
Rafael Ferreira, Diogo Tavares, Diogo Silva, Rodrigo Val\'erio, Jo\~ao, Bordalo, In\^es Sim\~oes, Vasco Ramos, David Semedo, Jo\~ao Magalh\~aes

TL;DR
TWIZ-v2 is a multimodal conversational assistant designed to guide users through complex tasks using voice, images, and videos, with innovations in zero-shot flows and a specialized large language model.
Contribution
The paper introduces TWIZ-v2, a novel multimodal conversational system with robust zero-shot capabilities and a specialized LLM for complex manual task assistance.
Findings
Effective guidance through complex tasks
Supports multimodal stimuli including voice, images, and videos
Robust performance in unseen scenarios
Abstract
In this report, we describe the vision, challenges, and scientific contributions of the Task Wizard team, TWIZ, in the Alexa Prize TaskBot Challenge 2022. Our vision, is to build TWIZ bot as an helpful, multimodal, knowledgeable, and engaging assistant that can guide users towards the successful completion of complex manual tasks. To achieve this, we focus our efforts on three main research questions: (1) Humanly-Shaped Conversations, by providing information in a knowledgeable way; (2) Multimodal Stimulus, making use of various modalities including voice, images, and videos; and (3) Zero-shot Conversational Flows, to improve the robustness of the interaction to unseen scenarios. TWIZ is an assistant capable of supporting a wide range of tasks, with several innovative features such as creative cooking, video navigation through voice, and the robust TWIZ-LLM, a Large Language Model…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAI in Service Interactions · Speech and dialogue systems · Topic Modeling
MethodsFocus · Wizard: Unsupervised goats tracking algorithm
