Patent Representation Learning via Self-supervision

TL;DR

This paper introduces a self-supervised contrastive learning framework for patent embeddings that leverages intra-document section-based views, improving retrieval and classification without relying on annotations.

Contribution

It proposes section-based augmentation for patent embedding learning, addressing limitations of dropout methods and exploiting patent structure for better representations.

Findings

Outperforms citation-and IPC-supervised baselines in retrieval and classification.

Section-specific embeddings improve task performance, with claims aiding retrieval and background aiding classification.

The method is scalable and avoids reliance on brittle annotations.

Abstract

This paper presents a simple yet effective contrastive learning framework for learning patent embeddings by leveraging multiple views from within the same document. We first identify a patent-specific failure mode of SimCSE style dropout augmentation: it produces overly uniform embeddings that lose semantic cohesion. To remedy this, we propose section-based augmentation, where different sections of a patent (e.g., abstract, claims, background) serve as complementary views. This design introduces natural semantic and structural diversity, mitigating over-dispersion and yielding embeddings that better preserve both global structure and local continuity. On large-scale benchmarks, our fully self-supervised method matches or surpasses citation-and IPC-supervised baselines in prior-art retrieval and classification, while avoiding reliance on brittle or incomplete annotations. Our analysis further shows that different sections specialize for different tasks-claims and summaries benefit retrieval, while background sections aid classification-highlighting the value of patents' inherent discourse structure for representation learning. These results highlight the value of exploiting intra-document views for scalable and generalizable patent understanding.