Extracting Document Relations from Search Corpus by Marginalizing over User Queries

TL;DR

This paper introduces EDR-MQ, a novel framework that discovers document relationships by analyzing co-occurrence patterns across diverse user queries, eliminating the need for manual annotations or predefined taxonomies.

Contribution

The paper presents a new query marginalization approach and MC-RAG method to estimate document relationships from search results without labeled data.

Findings

Successfully identifies topical clusters and evidence chains.

Reveals cross-domain connections not found by traditional methods.

Adapts to different user perspectives and information needs.

Abstract

Understanding relationships between documents in large-scale corpora is essential for knowledge discovery and information organization. However, existing approaches rely heavily on manual annotation or predefined relationship taxonomies. We propose EDR-MQ (Extracting Document Relations by Marginalizing over User Queries), a novel framework that discovers document relationships through query marginalization. EDR-MQ is based on the insight that strongly related documents often co-occur in results across diverse user queries, enabling us to estimate joint probabilities between document pairs by marginalizing over a collection of queries. To enable this query marginalization approach, we develop Multiply Conditioned Retrieval-Augmented Generation (MC-RAG), which employs conditional retrieval where subsequent document retrievals depend on previously retrieved content. By observing co-occurrence patterns across diverse queries, EDR-MQ estimates joint probabilities between document pairs without requiring labeled training data or predefined taxonomies. Experimental results show that our query marginalization approach successfully identifies meaningful document relationships, revealing topical clusters, evidence chains, and cross-domain connections that are not apparent through traditional similarity-based methods. Our query-driven framework offers a practical approach to document organization that adapts to different user perspectives and information needs.