Write What You Want: Applying Text-to-video Retrieval to Audiovisual Archives

TL;DR

This paper explores applying text-to-video retrieval techniques to audiovisual archives, proposing a classifier-enhanced workflow for better in-situ query results and demonstrating its potential through a real-world archive application and human-centered evaluation.

Contribution

It introduces a classifier-enhanced workflow for text-to-video retrieval in AV archives and demonstrates its effectiveness on a real-world dataset with user evaluation.

Findings

Improved retrieval accuracy with the proposed workflow.

Successful application to a real-world AV archive.

Positive human-centered evaluation results.

Abstract

Audiovisual (AV) archives, as an essential reservoir of our cultural assets, are suffering from the issue of accessibility. The complex nature of the medium itself made processing and interaction an open challenge still in the field of computer vision, multimodal learning, and human-computer interaction, as well as in culture and heritage. In recent years, with the raising of video retrieval tasks, methods in retrieving video content with natural language (text-to-video retrieval) gained quite some attention and have reached a performance level where real-world application is on the horizon. Appealing as it may sound, such methods focus on retrieving videos using plain visual-focused descriptions of what has happened in the video and finding videos such as instructions. It is too early to say such methods would be the new paradigms for accessing and encoding complex video content into high-dimensional data, but they are indeed innovative attempts and foundations to build future exploratory interfaces for AV archives (e.g. allow users to write stories and retrieve related snippets in the archive, or encoding video content at high-level for visualisation). This work filled the application gap by examining such text-to-video retrieval methods from an implementation point of view and proposed and verified a classifier-enhanced workflow to allow better results when dealing with in-situ queries that might have been different from the training dataset. Such a workflow is then applied to the real-world archive from T\'el\'evision Suisse Romande (RTS) to create a demo. At last, a human-centred evaluation is conducted to understand whether the text-to-video retrieval methods improve the overall experience of accessing AV archives.