Metadata Archaeology: Unearthing Data Subsets by Leveraging Training Dynamics

TL;DR

This paper introduces a unified framework called Metadata Archaeology that leverages training dynamics to identify and curate subsets of data with specific properties, improving data quality and diversity handling in machine learning.

Contribution

It proposes a novel, efficient method to infer dataset metadata by analyzing learning dynamics, without relying on prior labels or assumptions.

Findings

Effective in identifying mislabeled data

Classifies minority-group samples accurately

Enables scalable human data auditing

Abstract

Modern machine learning research relies on relatively few carefully curated datasets. Even in these datasets, and typically in `untidy' or raw data, practitioners are faced with significant issues of data quality and diversity which can be prohibitively labor intensive to address. Existing methods for dealing with these challenges tend to make strong assumptions about the particular issues at play, and often require a priori knowledge or metadata such as domain labels. Our work is orthogonal to these methods: we instead focus on providing a unified and efficient framework for Metadata Archaeology -- uncovering and inferring metadata of examples in a dataset. We curate different subsets of data that might exist in a dataset (e.g. mislabeled, atypical, or out-of-distribution examples) using simple transformations, and leverage differences in learning dynamics between these probe suites to infer metadata of interest. Our method is on par with far more sophisticated mitigation methods across different tasks: identifying and correcting mislabeled examples, classifying minority-group samples, prioritizing points relevant for training and enabling scalable human auditing of relevant examples.