HIPPO-Video: Simulating Watch Histories with Large Language Models for Personalized Video Highlighting

TL;DR

HIPPO-Video introduces a new dataset generated by an LLM-based simulator to capture realistic user watch histories for personalized video highlighting, enabling better user-centric content selection.

Contribution

The paper presents HIPPO-Video, a large-scale personalized video highlighting dataset created with an LLM-based user simulator, and proposes HiPHer, a method leveraging this data for improved personalized saliency prediction.

Findings

HiPHer outperforms existing methods in personalized saliency prediction.

The dataset captures diverse user preferences across 170 categories.

Extensive experiments validate the effectiveness of our approach.

Abstract

The exponential growth of video content has made personalized video highlighting an essential task, as user preferences are highly variable and complex. Existing video datasets, however, often lack personalization, relying on isolated videos or simple text queries that fail to capture the intricacies of user behavior. In this work, we introduce HIPPO-Video, a novel dataset for personalized video highlighting, created using an LLM-based user simulator to generate realistic watch histories reflecting diverse user preferences. The dataset includes 2,040 (watch history, saliency score) pairs, covering 20,400 videos across 170 semantic categories. To validate our dataset, we propose HiPHer, a method that leverages these personalized watch histories to predict preference-conditioned segment-wise saliency scores. Through extensive experiments, we demonstrate that our method outperforms existing generic and query-based approaches, showcasing its potential for highly user-centric video highlighting in real-world scenarios.