SimpleMTOD: A Simple Language Model for Multimodal Task-Oriented Dialogue with Symbolic Scene Representation

TL;DR

SimpleMTOD is a transformer-based multimodal dialogue model that effectively integrates visual scene semantics using symbolic tokens, achieving state-of-the-art results with a minimalist design and transfer learning from GPT-2.

Contribution

It introduces symbolic scene representation with local and de-localized tokens in a simple, transfer learning-based architecture for multimodal task-oriented dialogue.

Findings

Achieves state-of-the-art BLEU score of 0.327 in response generation.

Performs on par with state-of-the-art in disambiguation, coreference, and dialog state tracking.

Uses a minimalist approach without task-specific architectural modifications.

Abstract

SimpleMTOD is a simple language model which recasts several sub-tasks in multimodal task-oriented dialogues as sequence prediction tasks. SimpleMTOD is built on a large-scale transformer-based auto-regressive architecture, which has already proven to be successful in uni-modal task-oriented dialogues, and effectively leverages transfer learning from pre-trained GPT-2. In-order to capture the semantics of visual scenes, we introduce both local and de-localized tokens for objects within a scene. De-localized tokens represent the type of an object rather than the specific object itself and so possess a consistent meaning across the dataset. SimpleMTOD achieves a state-of-the-art BLEU score (0.327) in the Response Generation sub-task of the SIMMC 2.0 test-std dataset while performing on par in other multimodal sub-tasks: Disambiguation, Coreference Resolution, and Dialog State Tracking. This is despite taking a minimalist approach for extracting visual (and non-visual) information. In addition the model does not rely on task-specific architectural changes such as classification heads.