Separate First, Fuse Later: Mitigating Cross-Modal Interference in Audio-Visual LLMs Reasoning with Modality-Specific Chain-of-Thought

TL;DR

The paper introduces SFFL, a framework for audio-visual reasoning that reduces cross-modal interference by enforcing modality-specific reasoning and selective evidence fusion, improving accuracy and robustness.

Contribution

SFFL is a novel approach that enforces separate reasoning for audio and visual modalities and uses reinforcement learning to optimize modality preference.

Findings

Achieves 5.16% average accuracy gain on AVQA benchmarks.

Yields 11.17% improvement on a cross-modal hallucination benchmark.

Demonstrates enhanced robustness and reduced hallucinations.

Abstract

Audio and vision provide complementary evidence for audio-visual question answering, yet current audio-visual large language models may suffer from cross-modal interference: information from one modality misguides the interpretation of another, thereby inducing hallucinations. We attribute this issue to uncontrolled cross-modal interactions during intermediate reasoning. To mitigate this, we propose Separate First, Fuse Later (SFFL), an audio-visual reasoning framework designed to reduce cross-modal interference. SFFL enforces modality-specific chain-of-thought reasoning, producing separate audio and visual reasoning traces and integrating evidence for answering. We construct modality-preference labels via a data pipeline under different modality input settings. We use these labels as an auxiliary reward in reinforcement learning to encourage a instance-dependent preference for modality cues when answering. We further introduce a modality-specific reasoning mechanism that preserves modality isolation during the separated reasoning stage while enabling full access to cross-modal information at the evidence fusion stage. Experiments demonstrate consistent improvements in both accuracy and robustness, yielding an average relative gain of 5.16\% on general AVQA benchmarks and 11.17\% on a cross-modal hallucination benchmark.