Towards Robust Influence Functions with Flat Validation Minima

TL;DR

This paper introduces a new influence function estimation method that improves reliability in deep neural networks by focusing on flat validation minima, addressing issues caused by sharp validation risk.

Contribution

The work establishes a theoretical link between influence estimation error and validation risk sharpness, and proposes a novel influence function tailored for flat minima.

Findings

Our method outperforms existing influence estimation techniques.

Flat validation minima lead to more reliable influence estimates.

Experimental validation across multiple tasks confirms the effectiveness.

Abstract

The Influence Function (IF) is a widely used technique for assessing the impact of individual training samples on model predictions. However, existing IF methods often fail to provide reliable influence estimates in deep neural networks, particularly when applied to noisy training data. This issue does not stem from inaccuracies in parameter change estimation, which has been the primary focus of prior research, but rather from deficiencies in loss change estimation, specifically due to the sharpness of validation risk. In this work, we establish a theoretical connection between influence estimation error, validation set risk, and its sharpness, underscoring the importance of flat validation minima for accurate influence estimation. Furthermore, we introduce a novel estimation form of Influence Function specifically designed for flat validation minima. Experimental results across various tasks validate the superiority of our approach.