Efficient Estimation of Influence of a Training Instance

TL;DR

This paper introduces an efficient influence estimation method for neural networks, inspired by dropout, to improve interpretability and dataset quality, demonstrated on BERT and VGGNet.

Contribution

It proposes a novel dropout-inspired approach to estimate training influence efficiently, enhancing interpretability and dataset cleaning capabilities.

Findings

Effectively captures training influence on models.

Improves interpretability of error predictions.

Aids in dataset cleansing for better generalization.

Abstract

Understanding the influence of a training instance on a neural network model leads to improving interpretability. However, it is difficult and inefficient to evaluate the influence, which shows how a model's prediction would be changed if a training instance were not used. In this paper, we propose an efficient method for estimating the influence. Our method is inspired by dropout, which zero-masks a sub-network and prevents the sub-network from learning each training instance. By switching between dropout masks, we can use sub-networks that learned or did not learn each training instance and estimate its influence. Through experiments with BERT and VGGNet on classification datasets, we demonstrate that the proposed method can capture training influences, enhance the interpretability of error predictions, and cleanse the training dataset for improving generalization.