EARN: Efficient Inference Acceleration for LLM-based Generative Recommendation by Register Tokens

TL;DR

EARN is a novel inference acceleration framework for LLM-based generative recommendation that compresses interaction history into register tokens, significantly reducing latency and memory use while maintaining high accuracy.

Contribution

The paper introduces EARN, a new method leveraging layer-wise attention insights to efficiently compress information into register tokens, enabling faster inference in LLMRec systems.

Findings

Achieves up to 3.79x speedup in inference

Reduces KV Cache by 80.8%

Maintains better accuracy than finetuning methods

Abstract

Large Language Model-based generative recommendation (LLMRec) has achieved notable success, but it suffers from high inference latency due to massive computational overhead and memory pressure of KV Cache. Existing KV Cache reduction methods face critical limitations: cache compression offers marginal acceleration given recommendation tasks' short decoding steps, while prompt compression risks discarding vital interaction history. Through systematic analysis of attention patterns in LLMRec, we uncover two pivotal insights: 1) layer-wise attention sparsity inversion where early layers retain dense informative patterns while later layers exhibit high redundancy, and 2) dual attention sinks phenomenon where attention scores concentrate on both head and tail tokens of input sequences. Motivated by these insights, we propose EARN, an efficient inference framework that leverages the early layers to compress information into register tokens placed at the input sequence boundaries, then focuses solely on these tokens in the subsequent layers. Extensive experiments on three datasets, two LLMRec methods and two LLM architectures demonstrate EARN's superiority, achieving up to 3.79x speedup and 80.8% KV Cache reduction with better accuracy than the general finetuning approach. Our work bridges the efficiency-effectiveness gap in LLMRec, offering practical deployment advantages for industrial scenarios.