OVEL: Large Language Model as Memory Manager for Online Video Entity Linking

TL;DR

This paper introduces OVEL, a novel online video entity linking task that connects mentions in live videos to a knowledge base, utilizing a memory-augmented large language model to improve accuracy and timeliness.

Contribution

It defines the OVEL task, creates the LIVE dataset for live scenarios, and proposes a memory-based LLM approach with a new evaluation metric for real-time entity linking.

Findings

The proposed method achieves high accuracy in online video entity linking.

The memory-augmented LLM improves retrieval speed and robustness.

Experimental results demonstrate the effectiveness of the approach.

Abstract

In recent years, multi-modal entity linking (MEL) has garnered increasing attention in the research community due to its significance in numerous multi-modal applications. Video, as a popular means of information transmission, has become prevalent in people's daily lives. However, most existing MEL methods primarily focus on linking textual and visual mentions or offline videos's mentions to entities in multi-modal knowledge bases, with limited efforts devoted to linking mentions within online video content. In this paper, we propose a task called Online Video Entity Linking OVEL, aiming to establish connections between mentions in online videos and a knowledge base with high accuracy and timeliness. To facilitate the research works of OVEL, we specifically concentrate on live delivery scenarios and construct a live delivery entity linking dataset called LIVE. Besides, we propose an evaluation metric that considers timelessness, robustness, and accuracy. Furthermore, to effectively handle OVEL task, we leverage a memory block managed by a Large Language Model and retrieve entity candidates from the knowledge base to augment LLM performance on memory management. The experimental results prove the effectiveness and efficiency of our method.