ContextQFormer: A New Context Modeling Method for Multi-Turn Multi-Modal Conversations

TL;DR

This paper introduces ContextQFormer, a novel context modeling module with memory for enhancing multi-turn multi-modal dialogue systems, supported by a new dataset TMDialog, leading to improved interaction capabilities.

Contribution

The paper proposes ContextQFormer with a memory-based context modeling approach and introduces TMDialog, a new dataset for multi-turn multi-modal dialogue research.

Findings

ContextQFormer improves available rate by 2%-4% over baselines.

TMDialog contains longer conversations to support multi-turn dialogue research.

Experimental results validate the effectiveness of ContextQFormer.

Abstract

Multi-modal large language models have demonstrated remarkable zero-shot abilities and powerful image-understanding capabilities. However, the existing open-source multi-modal models suffer from the weak capability of multi-turn interaction, especially for long contexts. To address the issue, we first introduce a context modeling module, termed ContextQFormer, which utilizes a memory block to enhance the presentation of contextual information. Furthermore, to facilitate further research, we carefully build a new multi-turn multi-modal dialogue dataset (TMDialog) for pre-training, instruction-tuning, and evaluation, which will be open-sourced lately. Compared with other multi-modal dialogue datasets, TMDialog contains longer conversations, which supports the research of multi-turn multi-modal dialogue. In addition, ContextQFormer is compared with three baselines on TMDialog and experimental results illustrate that ContextQFormer achieves an improvement of 2%-4% in available rate over baselines.