From Content to Audience: A Multimodal Annotation Framework for Broadcast Television Analytics

TL;DR

This paper systematically evaluates multimodal annotation pipelines for Italian broadcast TV, analyzing model performance and deploying a framework for content-based audience analytics using multimodal data.

Contribution

It introduces a domain-specific benchmark and compares nine models across different pipeline architectures, highlighting model-dependent gains and operational deployment for audience analysis.

Findings

Larger models leverage temporal continuity effectively.

Smaller models degrade with extended multimodal input due to token overload.

The deployed pipeline enables correlational analysis of audience engagement.

Abstract

Automated semantic annotation of broadcast television content presents distinctive challenges, combining structured audiovisual composition, domain-specific editorial patterns, and strict operational constraints. While multimodal large language models (MLLMs) have demonstrated strong general-purpose video understanding capabilities, their comparative effectiveness across pipeline architectures and input configurations in broadcast-specific settings remains empirically undercharacterized. This paper presents a systematic evaluation of multimodal annotation pipelines applied to broadcast television news in the Italian setting. We construct a domain-specific benchmark of clips labeled across four semantic dimensions: visual environment classification, topic classification, sensitive content detection, and named entity recognition. Two different pipeline architectures are evaluated across nine frontier models, including Gemini 3.0 Pro, LLaMA 4 Maverick, Qwen-VL variants, and Gemma 3, under progressively enriched input strategies combining visual signals, automatic speech recognition, speaker diarization, and metadata. Experimental results demonstrate that gains from video input are strongly model-dependent: larger models effectively leverage temporal continuity, while smaller models show performance degradation under extended multimodal context, likely due to token overload. Beyond benchmarking, the selected pipeline is deployed on 14 full broadcast episodes, with minute-level annotations integrated with normalized audience measurement data provided by an Italian media company. This integration enables correlational analysis of topic-level audience sensitivity and generational engagement divergence, demonstrating the operational viability of the proposed framework for content-based audience analytics.