Forget Me Not: Fighting Local Overfitting with Knowledge Fusion and Distillation

TL;DR

This paper investigates local overfitting in deep neural networks, introduces a score to measure it, and proposes a knowledge fusion and distillation method to recover forgotten knowledge, improving performance especially with noisy labels.

Contribution

It introduces a novel score for local overfitting, links it to double descent, and presents a two-stage knowledge fusion and distillation approach to mitigate overfitting effects.

Findings

Effective detection of local overfitting using the new score.

Knowledge fusion and distillation improve performance with label noise.

Method reduces training and inference complexity.

Abstract

Overfitting in deep neural networks occurs less frequently than expected. This is a puzzling observation, as theory predicts that greater model capacity should eventually lead to overfitting -- yet this is rarely seen in practice. But what if overfitting does occur, not globally, but in specific sub-regions of the data space? In this work, we introduce a novel score that measures the forgetting rate of deep models on validation data, capturing what we term local overfitting: a performance degradation confined to certain regions of the input space. We demonstrate that local overfitting can arise even without conventional overfitting, and is closely linked to the double descent phenomenon. Building on these insights, we introduce a two-stage approach that leverages the training history of a single model to recover and retain forgotten knowledge: first, by aggregating checkpoints into an ensemble, and then by distilling it into a single model of the original size, thus enhancing performance without added inference cost. Extensive experiments across multiple datasets, modern architectures, and training regimes validate the effectiveness of our approach. Notably, in the presence of label noise, our method -- Knowledge Fusion followed by Knowledge Distillation -- outperforms both the original model and independently trained ensembles, achieving a rare win-win scenario: reduced training and inference complexity.