Active Perception Agent for Omnimodal Audio-Video Understanding

TL;DR

OmniAgent is a novel active perception system that dynamically orchestrates unimodal tools for fine-grained audio-visual understanding, achieving state-of-the-art results without additional training.

Contribution

It introduces the first fully active perception agent for omnimodal reasoning, shifting from passive to active multimodal inquiry with a novel audio-guided perception paradigm.

Findings

Achieves 10-20% higher accuracy on benchmarks.

Outperforms existing models without training.

Demonstrates effective dynamic tool orchestration.

Abstract

Omnimodal large language models have made significant strides in unifying audio and visual modalities; however, they often face challenges in fine-grained cross-modal understanding and have difficulty with multimodal alignment. To address these limitations, we introduce OmniAgent, to our best knowledge, the first fully active perception agent that dynamically orchestrates specialized unimodal tools to achieve more fine-grained omnimodal reasoning. Unlike previous works that rely on rigid, static workflows and dense frame-captioning, we demonstrate a paradigm shift from passive response generation to active multimodal inquiry. OmniAgent employs dynamic planning to autonomously orchestrate tool invocation on demand, strategically concentrating perceptual attention on task-relevant cues. Central to our approach is a novel coarse-to-fine audio-guided perception paradigm, which leverages audio cues to localize temporal events and guide subsequent reasoning. Extensive empirical evaluations on three audio-video understanding benchmarks demonstrate that OmniAgent achieves state-of-the-art performance, surpassing leading open-source and closed-source models by substantial margins of 10% - 20% accuracy without training.