PolySmart and VIREO @ TRECVid 2024 Ad-hoc Video Search

TL;DR

This paper introduces generation-augmented retrieval techniques for TRECVid 2024 AVS, enhancing textual query understanding through multiple generations and leveraging large language models to improve video search performance.

Contribution

It proposes a novel approach combining multiple generation methods and LLM-based query rephrasing to address out-of-vocabulary issues in video search.

Findings

Fusion of original and generated queries improves retrieval performance.

Generated queries produce diverse rank lists, enhancing search results.

Manual rephrasing with GPT-4 ensures concept bank consistency.

Abstract

This year, we explore generation-augmented retrieval for the TRECVid AVS task. Specifically, the understanding of textual query is enhanced by three generations, including Text2Text, Text2Image, and Image2Text, to address the out-of-vocabulary problem. Using different combinations of them and the rank list retrieved by the original query, we submitted four automatic runs. For manual runs, we use a large language model (LLM) (i.e., GPT4) to rephrase test queries based on the concept bank of the search engine, and we manually check again to ensure all the concepts used in the rephrased queries are in the bank. The result shows that the fusion of the original and generated queries outperforms the original query on TV24 query sets. The generated queries retrieve different rank lists from the original query.