Chain of Questions: Guiding Multimodal Curiosity in Language Models

TL;DR

The paper introduces the Chain of Questions framework, enabling multimodal language models to dynamically generate questions that guide sensory modality engagement, improving reasoning accuracy and interpretability in complex environments.

Contribution

It presents a novel curiosity-driven reasoning approach for multimodal models, enhancing their ability to select relevant sensory modalities during interaction.

Findings

Improves model accuracy on multimodal tasks

Enhances interpretability of reasoning process

Demonstrates effectiveness on a new multimodal benchmark

Abstract

Reasoning capabilities in large language models (LLMs) have substantially advanced through methods such as chain-of-thought and explicit step-by-step explanations. However, these improvements have not yet fully transitioned to multimodal contexts, where models must proactively decide which sensory modalities such as vision, audio, or spatial perception to engage when interacting with complex real-world environments. In this paper, we introduce the Chain of Questions (CoQ) framework, a curiosity-driven reasoning approach that encourages multimodal language models to dynamically generate targeted questions regarding their surroundings. These generated questions guide the model to selectively activate relevant modalities, thereby gathering critical information necessary for accurate reasoning and response generation. We evaluate our framework on a novel multimodal benchmark dataset, assembled by integrating WebGPT, ScienceQA, AVSD, and ScanQA datasets. Experimental results demonstrate that our CoQ method improves a foundation model's ability to effectively identify and integrate pertinent sensory information. This leads to improved accuracy, interpretability, and alignment of the reasoning process with diverse multimodal tasks.