Latent Wander: an Alternative Interface for Interactive and Serendipitous Discovery of Large AV Archives

TL;DR

This paper introduces a novel interface for exploring large audiovisual archives by leveraging text-to-video retrieval models, enabling semantic and emotionally rich search, and providing an immersive exploration experience.

Contribution

It develops a pipeline for encoding AV content into feature vectors, extends retrieval to emotionally descriptive queries, and creates an interactive latent space interface.

Findings

Successfully encoded raw archive videos into text-to-video features

Extended retrieval to emotionally rich descriptions

Created an interactive latent space exploration prototype

Abstract

Audiovisual (AV) archives are invaluable for holistically preserving the past. Unlike other forms, AV archives can be difficult to explore. This is not only because of its complex modality and sheer volume but also the lack of appropriate interfaces beyond keyword search. The recent rise in text-to-video retrieval tasks in computer science opens the gate to accessing AV content more naturally and semantically, able to map natural language descriptive sentences to matching videos. However, applications of this model are rarely seen. The contribution of this work is threefold. First, working with RTS (T\'el\'evision Suisse Romande), we identified the key blockers in a real archive for implementing such models. We built a functioning pipeline for encoding raw archive videos to the text-to-video feature vectors. Second, we designed and verified a method to encode and retrieve videos using emotionally abundant descriptions not supported in the original model. Third, we proposed an initial prototype for immersive and interactive exploration of AV archives in a latent space based on the previously mentioned encoding of videos.