C2T-ID: Converting Semantic Codebooks to Textual Document Identifiers for Generative Search

TL;DR

This paper introduces C2T-ID, a novel method for creating semantically rich document identifiers that balance expressiveness and search efficiency, improving generative retrieval performance.

Contribution

C2T-ID combines hierarchical clustering with keyword extraction to generate semantic, natural language-like docids that outperform existing methods in retrieval tasks.

Findings

C2T-ID outperforms baseline docid methods on Natural Questions.

It effectively balances semantic richness with manageable search spaces.

Experimental results show improved retrieval accuracy.

Abstract

Designing document identifiers (docids) that carry rich semantic information while maintaining tractable search spaces is a important challenge in generative retrieval (GR). Popular codebook methods address this by building a hierarchical semantic tree and constraining generation to its child nodes, yet their numeric identifiers cannot leverage the large language model's pretrained natural language understanding. Conversely, using text as docid provides more semantic expressivity but inflates the decoding space, making the system brittle to early-step errors. To resolve this trade-off, we propose C2T-ID: (i) first construct semantic numerical docid via hierarchical clustering; (ii) then extract high-frequency metadata keywords and iteratively replace each numeric label with its cluster's top-K keywords; and (iii) an optional two-level semantic smoothing step further enhances the fluency of C2T-ID. Experiments on Natural Questions and Taobao's product search demonstrate that C2T-ID significantly outperforms atomic, semantic codebook, and pure-text docid baselines, demonstrating its effectiveness in balancing semantic expressiveness with search space constraints.