Structure and Semantics Preserving Document Representations

TL;DR

This paper introduces a deep metric learning approach that integrates document content and inter-document relationships to improve retrieval, using a novel quintuplet loss and flexible margins for better semantic and structural preservation.

Contribution

It presents a holistic, fine-tunable model that combines intra-document content with inter-document relations using a novel loss function for enhanced retrieval performance.

Findings

Outperforms existing methods on multiple document retrieval datasets.

Effectively encodes both semantic relevance and structural relationships.

Supports query projection during inference for practical application.

Abstract

Retrieving relevant documents from a corpus is typically based on the semantic similarity between the document content and query text. The inclusion of structural relationship between documents can benefit the retrieval mechanism by addressing semantic gaps. However, incorporating these relationships requires tractable mechanisms that balance structure with semantics and take advantage of the prevalent pre-train/fine-tune paradigm. We propose here a holistic approach to learning document representations by integrating intra-document content with inter-document relations. Our deep metric learning solution analyzes the complex neighborhood structure in the relationship network to efficiently sample similar/dissimilar document pairs and defines a novel quintuplet loss function that simultaneously encourages document pairs that are semantically relevant to be closer and structurally unrelated to be far apart in the representation space. Furthermore, the separation margins between the documents are varied flexibly to encode the heterogeneity in relationship strengths. The model is fully fine-tunable and natively supports query projection during inference. We demonstrate that it outperforms competing methods on multiple datasets for document retrieval tasks.