Differentiable Geometric Indexing for End-to-End Generative Retrieval

TL;DR

This paper introduces Differentiable Geometric Indexing (DGI), a novel method that unifies index construction and retrieval in generative search by addressing optimization and geometric conflicts, leading to improved performance especially on long-tail data.

Contribution

DGI systematically resolves optimization and geometric conflicts in generative retrieval by employing soft teacher forcing, symmetric weight sharing, and cosine similarity, enabling end-to-end differentiability and geometric fidelity.

Findings

DGI outperforms existing baselines on large-scale industry datasets.

DGI shows superior robustness in long-tail retrieval scenarios.

DGI effectively balances semantic relevance and popularity bias.

Abstract

Generative Retrieval (GR) has emerged as a promising paradigm to unify indexing and search within a single probabilistic framework. However, existing approaches suffer from two intrinsic conflicts: (1) an Optimization Blockage, where the non-differentiable nature of discrete indexing creates a gradient blockage, decoupling index construction from the downstream retrieval objective; and (2) a Geometric Conflict, where standard unnormalized inner-product objectives induce norm-inflation instability, causing popular "hub" items to geometrically overshadow relevant long-tail items. To systematically resolve these misalignments, we propose Differentiable Geometric Indexing (DGI). First, to bridge the optimization gap, DGI enforces Operational Unification. It employs Soft Teacher Forcing via Gumbel-Softmax to establish a fully differentiable pathway, combined with Symmetric Weight Sharing to effectively align the quantizer's indexing space with the retriever's decoding space. Second, to restore geometric fidelity, DGI introduces Isotropic Geometric Optimization. We replace inner-product logits with scaled cosine similarity on the unit hypersphere to effectively decouple popularity bias from semantic relevance. Extensive experiments on large-scale industry search datasets and online e-commerce platform demonstrate that DGI outperforms competitive sparse, dense, and generative baselines. Notably, DGI exhibits superior robustness in long-tail scenarios, validating the necessity of harmonizing structural differentiability with geometric isotropy.