RAPID: Retrieval-Augmented Parallel Inference Drafting for Text-Based Video Event Retrieval

TL;DR

RAPID is a novel retrieval system that uses large language models to enrich and process text queries for more accurate video event retrieval, especially when queries lack context.

Contribution

The paper introduces RAPID, a retrieval-augmented system leveraging LLMs and prompt-based learning to improve text-based video event retrieval with incomplete queries.

Findings

RAPID outperforms traditional methods on custom datasets.

The system demonstrates high speed and accuracy in real-world scenarios.

Superior performance shown in the Ho Chi Minh City AI Challenge 2024.

Abstract

Retrieving events from videos using text queries has become increasingly challenging due to the rapid growth of multimedia content. Existing methods for text-based video event retrieval often focus heavily on object-level descriptions, overlooking the crucial role of contextual information. This limitation is especially apparent when queries lack sufficient context, such as missing location details or ambiguous background elements. To address these challenges, we propose a novel system called RAPID (Retrieval-Augmented Parallel Inference Drafting), which leverages advancements in Large Language Models (LLMs) and prompt-based learning to semantically correct and enrich user queries with relevant contextual information. These enriched queries are then processed through parallel retrieval, followed by an evaluation step to select the most relevant results based on their alignment with the original query. Through extensive experiments on our custom-developed dataset, we demonstrate that RAPID significantly outperforms traditional retrieval methods, particularly for contextually incomplete queries. Our system was validated for both speed and accuracy through participation in the Ho Chi Minh City AI Challenge 2024, where it successfully retrieved events from over 300 hours of video. Further evaluation comparing RAPID with the baseline proposed by the competition organizers demonstrated its superior effectiveness, highlighting the strength and robustness of our approach.