How Does a Neural Network's Architecture Impact Its Robustness to Noisy Labels?

TL;DR

This paper investigates how neural network architecture influences robustness to noisy labels, proposing a formal framework and demonstrating that better alignment with the target function enhances robustness and can outperform existing methods.

Contribution

It introduces a formal framework linking architecture-target alignment to robustness and provides empirical evidence supporting this connection.

Findings

Network robustness correlates with architecture-target alignment.

Aligned architectures outperform state-of-the-art noisy-label methods.

Better alignment can lead to improved test accuracy even with noisy labels.

Abstract

Noisy labels are inevitable in large real-world datasets. In this work, we explore an area understudied by previous works -- how the network's architecture impacts its robustness to noisy labels. We provide a formal framework connecting the robustness of a network to the alignments between its architecture and target/noise functions. Our framework measures a network's robustness via the predictive power in its representations -- the test performance of a linear model trained on the learned representations using a small set of clean labels. We hypothesize that a network is more robust to noisy labels if its architecture is more aligned with the target function than the noise. To support our hypothesis, we provide both theoretical and empirical evidence across various neural network architectures and different domains. We also find that when the network is well-aligned with the target function, its predictive power in representations could improve upon state-of-the-art (SOTA) noisy-label-training methods in terms of test accuracy and even outperform sophisticated methods that use clean labels.