DistillER: Knowledge Distillation in Entity Resolution with Large Language Models

TL;DR

This paper introduces DistillER, a framework that uses knowledge distillation to transfer the capabilities of large language models to smaller, more efficient models for entity resolution, balancing effectiveness and computational cost.

Contribution

It systematically explores data selection, knowledge elicitation, and distillation algorithms to improve LLM-based entity resolution without requiring gold labels.

Findings

Supervised fine-tuning on noisy labels outperforms other KD strategies.

DistillER achieves better effectiveness and efficiency than existing methods.

High-quality explanations can be generated by distilled models.

Abstract

Recent advances in Entity Resolution (ER) have leveraged Large Language Models (LLMs), achieving strong performance but at the cost of substantial computational resources or high financial overhead. Existing LLM-based ER approaches operate either in unsupervised settings and rely on very large and costly models, or in supervised settings and require ground-truth annotations, leaving a critical gap between time efficiency and effectiveness. To make LLM-powered ER more practical, we investigate Knowledge Distillation (KD) as a means to transfer knowledge from large, effective models (Teachers) to smaller, more efficient models (Students) without requiring gold labels. We introduce DistillER, the first framework that systematically bridges this gap across three dimensions: (i) Data Selection, where we study strategies for identifying informative subsets of data; (ii) Knowledge Elicitation, where we compare single- and multi-teacher settings across LLMs and smaller language models (SLMs); and (iii) Distillation Algorithms, where we evaluate supervised fine-tuning and reinforcement learning approaches. Our experiments reveal that supervised fine-tuning of Students on noisy labels generated by LLM Teachers consistently outperforms alternative KD strategies, while also enabling high-quality explanation generation. Finally, we benchmark DistillER against established supervised and unsupervised ER methods based on LLMs and SLMs, demonstrating significant improvements in both effectiveness and efficiency.