LAVA: Language Driven Scalable and Versatile Traffic Video Analytics

TL;DR

LAVA introduces a natural language-driven system for scalable, flexible traffic video analytics that outperforms traditional SQL-based methods in accuracy and speed by leveraging novel sampling, detection, and trajectory extraction techniques.

Contribution

The paper presents LAVA, a novel system enabling natural language queries for traffic video analysis, with new components for efficient sampling, open-world detection, and trajectory extraction.

Findings

Improves F1-scores for selection queries by 14%.

Reduces MPAE for aggregation queries by 0.39.

Achieves 86% top-k precision and 9.6x faster processing than baselines.

Abstract

In modern urban environments, camera networks generate massive amounts of operational footage -- reaching petabytes each day -- making scalable video analytics essential for efficient processing. Many existing approaches adopt an SQL-based paradigm for querying such large-scale video databases; however, this constrains queries to rigid patterns with predefined semantic categories, significantly limiting analytical flexibility. In this work, we explore a language-driven video analytics paradigm aimed at enabling flexible and efficient querying of high-volume video data driven by natural language. Particularly, we build \textsc{Lava}, a system that accepts natural language queries and retrieves traffic targets across multiple levels of granularity and arbitrary categories. \textsc{Lava} comprises three main components: 1) a multi-armed bandit-based efficient sampling method for video segment-level localization; 2) a video-specific open-world detection module for object-level retrieval; and 3) a long-term object trajectory extraction scheme for temporal object association, yielding complete trajectories for object-of-interests. To support comprehensive evaluation, we further develop a novel benchmark by providing diverse, semantically rich natural language predicates and fine-grained annotations for multiple videos. Experiments on this benchmark demonstrate that \textsc{Lava} improves $F_1$-scores for selection queries by $\mathbf{14\%}$, reduces MPAE for aggregation queries by $\mathbf{0.39}$, and achieves top-$k$ precision of $\mathbf{86\%}$, while processing videos $ \mathbf{9.6\times} $ faster than the most accurate baseline. Our code and dataset are available at https://github.com/yuyanrui/LAVA.