Token-wise Influential Training Data Retrieval for Large Language Models

TL;DR

This paper introduces RapidIn, a scalable and efficient framework for identifying influential training data for large language models, significantly speeding up influence estimation while maintaining accuracy.

Contribution

RapidIn is the first framework to enable fast, scalable influence estimation for LLMs through gradient compression and multi-GPU parallelization.

Findings

Achieves over 6,326x speedup in influence estimation

Compresses gradient vectors by over 200,000x

Supports multi-GPU parallelization for further acceleration

Abstract

Given a Large Language Model (LLM) generation, how can we identify which training data led to this generation? In this paper, we proposed RapidIn, a scalable framework adapting to LLMs for estimating the influence of each training data. The proposed framework consists of two stages: caching and retrieval. First, we compress the gradient vectors by over 200,000x, allowing them to be cached on disk or in GPU/CPU memory. Then, given a generation, RapidIn efficiently traverses the cached gradients to estimate the influence within minutes, achieving over a 6,326x speedup. Moreover, RapidIn supports multi-GPU parallelization to substantially accelerate caching and retrieval. Our empirical result confirms the efficiency and effectiveness of RapidIn.