Team of One: Cracking Complex Video QA with Model Synergy

TL;DR

This paper introduces a novel multi-model reasoning framework for open-ended video question answering, significantly improving reasoning depth, robustness, and generalization on complex real-world datasets without retraining models.

Contribution

It presents a prompting-and-response integration mechanism that coordinates multiple heterogeneous Video-Language Models via structured chains of thought, guided by an external LLM for response selection and fusion.

Findings

Outperforms existing baselines across all evaluation metrics.

Demonstrates superior generalization and robustness in complex scenarios.

Provides a lightweight, extensible reasoning strategy without retraining models.

Abstract

We propose a novel framework for open-ended video question answering that enhances reasoning depth and robustness in complex real-world scenarios, as benchmarked on the CVRR-ES dataset. Existing Video-Large Multimodal Models (Video-LMMs) often exhibit limited contextual understanding, weak temporal modeling, and poor generalization to ambiguous or compositional queries. To address these challenges, we introduce a prompting-and-response integration mechanism that coordinates multiple heterogeneous Video-Language Models (VLMs) via structured chains of thought, each tailored to distinct reasoning pathways. An external Large Language Model (LLM) serves as an evaluator and integrator, selecting and fusing the most reliable responses. Extensive experiments demonstrate that our method significantly outperforms existing baselines across all evaluation metrics, showcasing superior generalization and robustness. Our approach offers a lightweight, extensible strategy for advancing multimodal reasoning without requiring model retraining, setting a strong foundation for future Video-LMM development.