Integrated Semantic and Temporal Alignment for Interactive Video Retrieval

TL;DR

This paper presents an integrated video retrieval system combining semantic and temporal alignment techniques, featuring novel components for handling out-of-knowledge queries and temporal event alignment, advancing real-world video search capabilities.

Contribution

The paper introduces QUEST and DANTE, innovative modules for out-of-knowledge query resolution and temporal event alignment, integrated into a scalable video retrieval framework.

Findings

System effectively handles out-of-knowledge queries.

DANTE achieves efficient temporal alignment in complex videos.

Framework outperforms existing methods on TRAKE challenge.

Abstract

The growing volume of video data and the introduction of complex retrieval challenges, such as the Temporal Retrieval and Alignment of Key Events (TRAKE) task at the Ho Chi Minh City AI Challenge 2025, expose critical limitations in existing systems. Many methodologies lack scalable, holistic architectures and rely on "frozen" embedding models that fail on out-of-knowledge (OOK) or real-world queries. This paper introduces the comprehensive video retrieval framework developed by team AIO\_Owlgorithms to address these gaps. Our system features an architecture integrating TransNetV2 for scene segmentation, BEiT-3 for visual embeddings in Milvus, and Gemini OCR for metadata in Elasticsearch. We propose two components: (1) \textbf{QUEST} (Query Understanding and External Search for Out-of-Knowledge Tasks), a two-branch framework that leverages a Large Language Model (LLM) for query rewriting and an external image search pathway to resolve OOK queries; and (2) \textbf{DANTE} (Dynamic Alignment of Narrative Temporal Events), a dynamic programming algorithm that efficiently solves the temporally-incoherent TRAKE task. These contributions form a robust and intelligent system that significantly advances the state-of-the-art in handling complex, real-world video search queries.