Interaction-Aware Influence Functions for Group Attribution

TL;DR

This paper introduces interaction-aware influence functions that account for pairwise interactions between training examples, improving group influence estimation and data selection for model training.

Contribution

It proposes a second-order expansion of influence functions to capture interactions, enhancing group influence estimation and data selection strategies.

Findings

Estimator better tracks leave-group-out retraining effects across datasets and models.

Outperforms prior influence and similarity baselines in instruction-tuning data selection.

Improves influence estimation accuracy over standard first-order methods.

Abstract

Influence functions approximate how removing a training example changes a quantity of interest, called the target function, such as a held-out loss. To estimate the influence of a group of examples, the standard practice is to sum the individual influences of its members. However, this sum does not capture how examples jointly affect the target: a pair of examples may be redundant or complementary, but the sum cannot distinguish these cases. We propose an interaction-aware influence function that characterizes how interactions between examples influence the target. By expanding the target to second order around the trained parameters, we obtain an estimator that augments the standard sum with a pairwise interaction term that captures the alignment between two examples' effects on the target. We empirically evaluate our estimator in two settings. First, on six dataset-model pairs spanning logistic regression, MLPs, and ResNet-9, our estimator tracks leave-group-out retraining substantially better than first-order influence across all settings. Second, when used as a greedy selection rule for instruction-tuning data on Llama-3.1-8B, it beats prior influence-based and representation-similarity baselines on five of seven downstream tasks, in a regime where standard influence-based selection underperforms random selection.