HyperINF: Unleashing the HyperPower of the Schulz's Method for Data Influence Estimation

TL;DR

HyperINF introduces a novel influence function approximation method using the hyperpower technique, achieving high accuracy and efficiency on large models by leveraging low-rank Hessian approximations and Schulz's iterative algorithm.

Contribution

This paper presents HyperINF, a new influence estimation method that combines hyperpower iteration with low-rank Hessian approximation for scalable and accurate influence analysis.

Findings

HyperINF outperforms existing baselines in accuracy and stability.

HyperINF achieves minimal memory and computational overhead on large models.

HyperINF improves downstream tasks like data attribution and model fine-tuning.

Abstract

Influence functions provide a principled method to assess the contribution of individual training samples to a specific target. Yet, their high computational costs limit their applications on large-scale models and datasets. Existing methods proposed for influence function approximation have significantly reduced the computational overheads. However, they mostly suffer from inaccurate estimation due to the lack of strong convergence guarantees from the algorithm. The family of hyperpower methods are well-known for their rigorous convergence guarantees on matrix inverse approximation, while the matrix multiplication operation can involve intractable memory and computation costs on large-scale models. We propose HyperINF, an efficient and accurate influence function approximation method which leverages the hyperpower method, specifically Schulz's iterative algorithm. To deal with the computation-intensive matrix multiplication, we incorporate the generalized fisher information (GFIM) as a low-rank approximation of the Hessian matrix, which reduces the memory and computation overheads to constant costs independent of ranks on LoRA-tuned models. We first demonstrate the superior accuracy and stability of HyperINF compared to other baselines through a synthetic convergence simulation for matrix inversion. We further validate the efficacy of HyperINF through extensive real-world data attribution tasks, including mislabeled data detection and data selection for LLM and VLM fine-tuning. On LoRA-tuned models, HyperINF achieves superior downstream performance with minimal memory and computational overhead, while other baselines suffer from significant degradation. Our codebase is available at https://github.com/Blackzxy/HyperINF.