ViSIL: Unified Evaluation of Information Loss in Multimodal Video Captioning

TL;DR

This paper introduces ViSIL, a novel information-theoretic metric for evaluating multimodal video summaries, effectively quantifying information loss across diverse formats and correlating well with human and AI performance.

Contribution

The paper presents ViSIL, a unified, quantitative metric for assessing information coverage in multimodal video summaries, addressing limitations of traditional metrics.

Findings

ViSIL correlates significantly with human and VLM performance on VQA tasks.

ViSIL enables selection of summaries that optimize information retention and processing speed.

Outperforms text summaries by 7% in VQA accuracy without extra computational cost.

Abstract

Multimodal video captioning condenses dense footage into a structured format of keyframes and natural language. By creating a cohesive multimodal summary, this approach anchors generative AI in rich semantic evidence and serves as a lightweight proxy for high-efficiency retrieval. However, traditional metrics like BLEU or ROUGE fail to quantify information coverage across disparate modalities, such as comparing a paragraph of text to a sequence of keyframes. To address this, we propose the Video Summary Information Loss (ViSIL) score, an information-theoretic framework that quantifies the video information not captured by a summary via vision-language model (VLM) inference. By measuring the information loss, ViSIL is a unified metric that enables direct comparison across multimodal summary formats despite their structural discrepancies. Our results demonstrate that ViSIL scores show a statistically significant correlation with both human and VLM performance on Video Question Answering (VQA) tasks. ViSIL also enables summary selection to optimize the trade-off between information loss and processing speed, establishing a Pareto-optimal frontier that outperforms text summaries by $7\%$ in VQA accuracy without increasing processing load.