# Cleaning Noisy and Heterogeneous Metadata for Record Linking Across Scholarly Big Datasets

## TL;DR

This paper presents a system for cleaning noisy, heterogeneous scholarly metadata and accurately linking records across large datasets using supervised methods, citation info, and BM25-based blocking.

## Contribution

It introduces a novel system combining supervised feature extraction, citation data, and BM25 blocking for high-precision record matching in scholarly datasets.

## Key findings

- Outperforms baseline string similarity methods.
- Achieves high accuracy in matching CiteSeerX with other datasets.
- Successfully deployed in CiteSeerX for metadata cleaning.

## Abstract

Automatically extracted metadata from scholarly documents in PDF formats is usually noisy and heterogeneous, often containing incomplete fields and erroneous values. One common way of cleaning metadata is to use a bibliographic reference dataset. The challenge is to match records between corpora with high precision. The existing solution which is based on information retrieval and string similarity on titles works well only if the titles are cleaned. We introduce a system designed to match scholarly document entities with noisy metadata against a reference dataset. The blocking function uses the classic BM25 algorithm to find the matching candidates from the reference data that has been indexed by ElasticSearch. The core components use supervised methods which combine features extracted from all available metadata fields. The system also leverages available citation information to match entities. The combination of metadata and citation achieves high accuracy that significantly outperforms the baseline method on the same test dataset. We apply this system to match the database of CiteSeerX against Web of Science, PubMed, and DBLP. This method will be deployed in the CiteSeerX system to clean metadata and link records to other scholarly big datasets.