Human in the loop approaches in multi-modal conversational task guidance system development

TL;DR

This paper explores the development of human-in-the-loop multi-modal conversational systems for task guidance, addressing technical challenges, dataset limitations, and introducing a new data collection tool with preliminary experiments.

Contribution

It highlights key challenges, reviews existing datasets, and presents a novel wizard-of-oz data collection tool with initial pilot results.

Findings

Identified challenges in developing conversational task guidance systems.

Reviewed limitations of current datasets.

Developed a new data collection tool and conducted pilot experiments.

Abstract

Development of task guidance systems for aiding humans in a situated task remains a challenging problem. The role of search (information retrieval) and conversational systems for task guidance has immense potential to help the task performers achieve various goals. However, there are several technical challenges that need to be addressed to deliver such conversational systems, where common supervised approaches fail to deliver the expected results in terms of overall performance, user experience and adaptation to realistic conditions. In this preliminary work we first highlight some of the challenges involved during the development of such systems. We then provide an overview of existing datasets available and highlight their limitations. We finally develop a model-in-the-loop wizard-of-oz based data collection tool and perform a pilot experiment.