Zero-shot Audio Topic Reranking using Large Language Models

TL;DR

This paper explores zero-shot reranking of video search results using large language models to improve topic retrieval accuracy without domain-specific training, leveraging multiple text sources for enhanced performance.

Contribution

It introduces a zero-shot reranking approach with LLMs for video archive search, comparing different text sources and analyzing their information consistency.

Findings

Reranking significantly improves retrieval accuracy.

Using multiple text sources enhances reranking performance.

Fact-checking reveals insights into source information consistency.

Abstract

Multimodal Video Search by Examples (MVSE) investigates using video clips as the query term for information retrieval, rather than the more traditional text query. This enables far richer search modalities such as images, speaker, content, topic, and emotion. A key element for this process is highly rapid and flexible search to support large archives, which in MVSE is facilitated by representing video attributes with embeddings. This work aims to compensate for any performance loss from this rapid archive search by examining reranking approaches. In particular, zero-shot reranking methods using large language models (LLMs) are investigated as these are applicable to any video archive audio content. Performance is evaluated for topic-based retrieval on a publicly available video archive, the BBC Rewind corpus. Results demonstrate that reranking significantly improves retrieval ranking without requiring any task-specific in-domain training data. Furthermore, three sources of information (ASR transcriptions, automatic summaries and synopses) as input for LLM reranking were compared. To gain a deeper understanding and further insights into the performance differences and limitations of these text sources, we employ a fact-checking approach to analyse the information consistency among them.