MAGNET: A Multi-agent Framework for Finding Audio-Visual Needles by Reasoning over Multi-Video Haystacks

TL;DR

This paper introduces AVHaystacks, a new large-scale benchmark for audio-visual reasoning across multiple videos, and proposes MAGNET, a multi-agent framework that significantly improves multi-video retrieval and temporal grounding in complex scenarios.

Contribution

The paper presents AVHaystacks, a comprehensive benchmark for multi-video reasoning, and introduces MAGNET, a novel multi-agent framework that enhances audio-visual understanding in large-scale retrieval tasks.

Findings

MAGNET achieves up to 89% improvement in BLEU@4 scores.

MAGNET outperforms baseline methods on AVHaystacks benchmark.

New metrics STEM and MTGS enable better evaluation of multi-video reasoning.

Abstract

Large multimodal models (LMMs) have shown remarkable progress in audio-visual understanding, yet they struggle with real-world scenarios that require complex reasoning across extensive video collections. Existing benchmarks for video question answering remain limited in scope, typically involving one clip per query, which falls short of representing the challenges of large-scale, audio-visual retrieval and reasoning encountered in practical applications. To bridge this gap, we introduce a novel task named AV-HaystacksQA, where the goal is to identify salient segments across different videos in response to a query and link them together to generate the most informative answer. To this end, we present AVHaystacks, an audio-visual benchmark comprising 3100 annotated QA pairs designed to assess the capabilities of LMMs in multi-video retrieval and temporal grounding task. Additionally, we propose a model-agnostic, multi-agent framework MAGNET to address this challenge, achieving up to 89% and 65% relative improvements over baseline methods on BLEU@4 and GPT evaluation scores in QA task on our proposed AVHaystacks. To enable robust evaluation of multi-video retrieval and temporal grounding for optimal response generation, we introduce two new metrics, STEM, which captures alignment errors between a ground truth and a predicted step sequence and MTGS, to facilitate balanced and interpretable evaluation of segment-level grounding performance. Project: https://schowdhury671.github.io/magnet_project/