Robust Classification under Noisy Labels: A Geometry-Aware Reliability Framework for Foundation Models

TL;DR

This paper introduces a geometry-aware framework for robust classification with foundation models in noisy label scenarios, leveraging local geometry and reliability estimation to improve performance without retraining.

Contribution

It proposes a novel two-stage, geometry-informed inference method that enhances robustness to label noise in foundation models without requiring model retraining.

Findings

Outperforms standard K-NN in noisy conditions

Improves robustness on CIFAR-10 and DermaMNIST datasets

Uses local geometry for reliability estimation

Abstract

Foundation models (FMs) pretrained on large datasets have become fundamental for various downstream machine learning tasks, in particular in scenarios where obtaining perfectly labeled data is prohibitively expensive. In this paper, we assume an FM has to be fine-tuned with noisy data and present a two-stage framework to ensure robust classification in the presence of label noise without model retraining. Recent work has shown that simple k-nearest neighbor (kNN) approaches using an embedding derived from an FM can achieve good performance even in the presence of severe label noise. Our work is motivated by the fact that these methods make use of local geometry. In this paper, following a similar two-stage procedure, reliability estimation followed by reliability-weighted inference, we show that improved performance can be achieved by introducing geometry information. For a given instance, our proposed inference uses a local neighborhood of training data, obtained using the non-negative kernel (NNK) neighborhood construction. We propose several methods for reliability estimation that can rely less on distance and local neighborhood as the label noise increases. Our evaluation on CIFAR-10 and DermaMNIST shows that our methods improve robustness across various noise conditions, surpassing standard K-NN approaches and recent adaptive-neighborhood baselines.