Simple Dialogue System with AUDITED

TL;DR

This paper introduces AUDITED, a multimodal dialogue system leveraging auxiliary unsupervised visual and textual data, with novel neighbor embedding methods and a discriminator to enhance performance across text and image tasks.

Contribution

The paper presents AUDITED, a new multimodal dialogue system that integrates neighbor embedding techniques and a discriminator for improved multimodal understanding.

Findings

Improved performance on Multimodal Dialogue Dataset (MMD)

Enhanced results on SIMMC dataset

Effective use of neighbor embeddings for context modeling

Abstract

We devise a multimodal conversation system for dialogue utterances composed of text, image or both modalities. We leverage Auxiliary UnsuperviseD vIsual and TExtual Data (AUDITED). To improve the performance of text-based task, we utilize translations of target sentences from English to French to form the assisted supervision. For the image-based task, we employ the DeepFashion dataset in which we seek nearest neighbor images of positive and negative target images of the MMD data. These nearest neighbors form the nearest neighbor embedding providing an external context for target images. We form two methods to create neighbor embedding vectors, namely Neighbor Embedding by Hard Assignment (NEHA) and Neighbor Embedding by Soft Assignment (NESA) which generate context subspaces per target image. Subsequently, these subspaces are learnt by our pipeline as a context for the target data. We also propose a discriminator which switches between the image- and text-based tasks. We show improvements over baselines on the large-scale Multimodal Dialogue Dataset (MMD) and SIMMC.