Training an adaptive dialogue policy for interactive learning of visually grounded word meanings

TL;DR

This paper introduces a multi-modal dialogue system that interactively learns visually grounded word meanings from a human tutor, optimizing dialogue strategies to improve learning efficiency and accuracy.

Contribution

It develops an adaptive dialogue policy that balances classifier accuracy and tutoring costs, integrating semantic parsing with visual classifiers in an interactive setting.

Findings

Dialogue initiative affects learning performance

Expressing confidence levels improves learning efficiency

Handling elliptical and incremental turns enhances system robustness

Abstract

We present a multi-modal dialogue system for interactive learning of perceptually grounded word meanings from a human tutor. The system integrates an incremental, semantic parsing/generation framework - Dynamic Syntax and Type Theory with Records (DS-TTR) - with a set of visual classifiers that are learned throughout the interaction and which ground the meaning representations that it produces. We use this system in interaction with a simulated human tutor to study the effects of different dialogue policies and capabilities on the accuracy of learned meanings, learning rates, and efforts/costs to the tutor. We show that the overall performance of the learning agent is affected by (1) who takes initiative in the dialogues; (2) the ability to express/use their confidence level about visual attributes; and (3) the ability to process elliptical and incrementally constructed dialogue turns. Ultimately, we train an adaptive dialogue policy which optimises the trade-off between classifier accuracy and tutoring costs.