Sparkles: Unlocking Chats Across Multiple Images for Multimodal Instruction-Following Models

TL;DR

Sparkles introduces a new multimodal dialogue dataset and benchmark, enabling instruction-following models to better understand and converse across multiple images without losing single-image performance.

Contribution

The paper presents SparklesDialogue and SparklesEval datasets, and SparklesChat, a multimodal model trained on these resources for improved multi-image dialogue understanding.

Findings

Enhanced multi-image dialogue comprehension in SparklesChat

Maintains single-image understanding capabilities

Resources are publicly available for further research

Abstract

Large language models exhibit enhanced zero-shot performance on various tasks when fine-tuned with instruction-following data. Multimodal instruction-following models extend these capabilities by integrating both text and images. However, existing models such as MiniGPT-4 and LLaVA face challenges in maintaining dialogue coherence in scenarios involving multiple images. A primary reason is the lack of a specialized dataset for this critical application. To bridge these gaps, we introduce SparklesDialogue, the first machine-generated dialogue dataset tailored for word-level interleaved multi-image and text interactions. Furthermore, we construct SparklesEval, a GPT-assisted benchmark for quantitatively assessing a model's conversational competence across multiple images and dialogue turns. We then present SparklesChat, a multimodal instruction-following model for open-ended dialogues across multiple images. Our experiments validate the effectiveness of training SparklesChat with SparklesDialogue based on MiniGPT-4 and LLaVA-v1.5, which enhances comprehension across multiple images and dialogue turns, and does not compromise single-image understanding capabilities. Qualitative evaluations further demonstrate SparklesChat's generality in handling real-world applications. All resources related to this study are publicly available at https://github.com/HYPJUDY/Sparkles.