Frozen LVLMs for Micro-Video Recommendation: A Systematic Study of Feature Extraction and Fusion

TL;DR

This paper systematically evaluates how to best extract and fuse features from frozen LVLMs for micro-video recommendation, revealing key principles and proposing a novel fusion framework that improves performance.

Contribution

It provides the first comprehensive empirical analysis of LVLM feature extraction and integration strategies, introducing the Dual Feature Fusion (DFF) framework for enhanced recommendation accuracy.

Findings

Intermediate hidden states outperform caption-based features.

ID embeddings are essential and fusion is superior to replacement.

Layer-wise effectiveness of decoder features varies significantly.

Abstract

Frozen Large Video Language Models (LVLMs) are increasingly employed in micro-video recommendation due to their strong multimodal understanding. However, their integration lacks systematic empirical evaluation: practitioners typically deploy LVLMs as fixed black-box feature extractors without systematically comparing alternative representation strategies. To address this gap, we present the first systematic empirical study along two key design dimensions: (i) integration strategies with ID embeddings, specifically replacement versus fusion, and (ii) feature extraction paradigms, comparing LVLM-generated captions with intermediate decoder hidden states. Extensive experiments on representative LVLMs reveal three key principles: (1) intermediate hidden states consistently outperform caption-based representations, as natural-language summarization inevitably discards fine-grained visual semantics crucial for recommendation; (2) ID embeddings capture irreplaceable collaborative signals, rendering fusion strictly superior to replacement; and (3) the effectiveness of intermediate decoder features varies significantly across layers. Guided by these insights, we propose the Dual Feature Fusion (DFF) Framework, a lightweight and plug-and-play approach that adaptively fuses multi-layer representations from frozen LVLMs with item ID embeddings. DFF achieves state-of-the-art performance on two real-world micro-video recommendation benchmarks, consistently outperforming strong baselines and providing a principled approach to integrating off-the-shelf large vision-language models into micro-video recommender systems.