Class-wise Autoencoders Measure Classification Difficulty And Detect Label Mistakes

TL;DR

This paper presents a novel autoencoder-based framework that measures classification difficulty and detects label mistakes by analyzing reconstruction error ratios across classes, correlating well with model error rates and mislabel detection.

Contribution

It introduces reconstruction error ratios (RERs) as a new method to analyze dataset difficulty and identify label errors, providing a systematic and interpretable approach.

Findings

RERs strongly correlate with classification error rates.

RERs effectively detect label mistakes under various noise conditions.

The framework applies across multiple visual datasets.

Abstract

We introduce a new framework for analyzing classification datasets based on the ratios of reconstruction errors between autoencoders trained on individual classes. This analysis framework enables efficient characterization of datasets on the sample, class, and entire dataset levels. We define reconstruction error ratios (RERs) that probe classification difficulty and allow its decomposition into (1) finite sample size and (2) Bayes error and decision-boundary complexity. Through systematic study across 19 popular visual datasets, we find that our RER-based dataset difficulty probe strongly correlates with error rate for state-of-the-art (SOTA) classification models. By interpreting sample-level classification difficulty as a label mistakenness score, we further find that RERs achieve SOTA performance on mislabel detection tasks on hard datasets under symmetric and asymmetric label noise. Our code is publicly available at https://github.com/voxel51/reconstruction-error-ratios.