QMAVIS: Long Video-Audio Understanding using Fusion of Large Multimodal Models

TL;DR

QMAVIS is a novel long video-audio understanding pipeline that combines large multimodal models, language models, and speech recognition to analyze videos from minutes to over an hour, significantly improving understanding accuracy.

Contribution

Introduces QMAVIS, a late fusion pipeline for long video-audio understanding, addressing the challenge of analyzing extended videos beyond short clips with improved performance.

Findings

38.75% improvement over state-of-the-art models on VideoMME dataset

Up to 2% improvement on PerceptionTest and EgoSchema datasets

Effective extraction of scene nuances and overarching narratives in long videos

Abstract

Large Multimodal Models (LMMs) for video-audio understanding have traditionally been evaluated only on shorter videos of a few minutes long. In this paper, we introduce QMAVIS (Q Team-Multimodal Audio Video Intelligent Sensemaking), a novel long video-audio understanding pipeline built through a late fusion of LMMs, Large Language Models, and speech recognition models. QMAVIS addresses the gap in long-form video analytics, particularly for longer videos of a few minutes to beyond an hour long, opening up new potential applications in sensemaking, video content analysis, embodied AI, etc. Quantitative experiments using QMAVIS demonstrated a 38.75% improvement over state-of-the-art video-audio LMMs like VideoLlaMA2 and InternVL2 on the VideoMME (with subtitles) dataset, which comprises long videos with audio information. Evaluations on other challenging video understanding datasets like PerceptionTest and EgoSchema saw up to 2% improvement, indicating competitive performance. Qualitative experiments also showed that QMAVIS is able to extract the nuances of different scenes in a long video audio content while understanding the overarching narrative. Ablation studies were also conducted to ascertain the impact of each component in the fusion pipeline.