IGD: Token Decisiveness Modeling via Information Gain in LLMs for Personalized Recommendation

TL;DR

This paper introduces a novel method called IGD that models token decisiveness in LLMs for recommendation, using information gain to improve token handling during training and decoding, leading to better recommendation accuracy.

Contribution

The paper proposes a new perspective on token decisiveness using information gain, and develops IGD, a strategy that enhances LLM-based recommendation by emphasizing high-decisiveness tokens.

Findings

IGD improves recommendation accuracy across four benchmark datasets.

Tokens with low information gain often dominate training but contribute little to discrimination.

Rebalancing token importance based on IG enhances model performance.

Abstract

Large Language Models (LLMs) have shown strong potential for recommendation by framing item prediction as a token-by-token language generation task. However, existing methods treat all item tokens equally, simply pursuing likelihood maximization during both optimization and decoding. This overlooks crucial token-level differences in decisiveness-many tokens contribute little to item discrimination yet can dominate optimization or decoding. To quantify token decisiveness, we propose a novel perspective that models item generation as a decision process, measuring token decisiveness by the Information Gain (IG) each token provides in reducing uncertainty about the generated item. Our empirical analysis reveals that most tokens have low IG but often correspond to high logits, disproportionately influencing training loss and decoding, which may impair model performance. Building on these insights, we introduce an Information Gain-based Decisiveness-aware Token handling (IGD) strategy that integrates token decisiveness into both tuning and decoding. Specifically, IGD downweights low-IG tokens during tuning and rebalances decoding to emphasize tokens with high IG. In this way, IGD moves beyond pure likelihood maximization, effectively prioritizing high-decisiveness tokens. Extensive experiments on four benchmark datasets with two LLM backbones demonstrate that IGD consistently improves recommendation accuracy, achieving significant gains on widely used ranking metrics compared to strong baselines.