Hi-SAM: A Hierarchical Structure-Aware Multi-modal Framework for Large-Scale Recommendation

TL;DR

Hi-SAM introduces a hierarchical, structure-aware multi-modal recommendation framework that improves semantic tokenization and models user-item hierarchies, leading to better performance especially in cold-start situations.

Contribution

The paper presents Hi-SAM, a novel framework combining a disentangled semantic tokenizer and a hierarchical transformer to enhance multi-modal recommendation accuracy.

Findings

Achieves a 6.55% gain in core online metrics.

Outperforms state-of-the-art baselines on real-world datasets.

Effective in cold-start recommendation scenarios.

Abstract

Multi-modal recommendation has gained traction as items possess rich attributes like text and images. Semantic ID-based approaches effectively discretize this information into compact tokens. However, two challenges persist: (1) Suboptimal Tokenization: existing methods (e.g., RQ-VAE) lack disentanglement between shared cross-modal semantics and modality-specific details, causing redundancy or collapse; (2) Architecture-Data Mismatch: vanilla Transformers treat semantic IDs as flat streams, ignoring the hierarchy of user interactions, items, and tokens. Expanding items into multiple tokens amplifies length and noise, biasing attention toward local details over holistic semantics. We propose Hi-SAM, a Hierarchical Structure-Aware Multi-modal framework with two designs: (1) Disentangled Semantic Tokenizer (DST): unifies modalities via geometry-aware alignment and quantizes them via a coarse-to-fine strategy. Shared codebooks distill consensus while modality-specific ones recover nuances from residuals, enforced by mutual information minimization; (2) Hierarchical Memory-Anchor Transformer (HMAT): splits positional encoding into inter- and intra-item subspaces via Hierarchical RoPE to restore hierarchy. It inserts Anchor Tokens to condense items into compact memory, retaining details for the current item while accessing history only through compressed summaries. Experiments on real-world datasets show consistent improvements over SOTA baselines, especially in cold-start scenarios. Deployed on a large-scale social platform serving millions of users, Hi-SAM achieved a 6.55% gain in the core online metric.