FASTTRACK: Fast and Accurate Fact Tracing for LLMs

TL;DR

FASTTRACK leverages Large Language Models to improve fact tracing accuracy and efficiency by validating supportive evidence and reducing the training data scope, significantly outperforming existing methods.

Contribution

The paper introduces FASTTRACK, a novel LLM-based method that enhances fact tracing by validating evidence and clustering data, addressing limitations of relevance and computational cost.

Findings

Over 100% improvement in F1 score over state-of-the-art methods.

FASTTRACK is 33 times faster than TracIn.

Achieves superior accuracy and efficiency in fact tracing.

Abstract

Fact tracing seeks to identify specific training examples that serve as the knowledge source for a given query. Existing approaches to fact tracing rely on assessing the similarity between each training sample and the query along a certain dimension, such as lexical similarity, gradient, or embedding space. However, these methods fall short of effectively distinguishing between samples that are merely relevant and those that actually provide supportive evidence for the information sought by the query. This limitation often results in suboptimal effectiveness. Moreover, these approaches necessitate the examination of the similarity of individual training points for each query, imposing significant computational demands and creating a substantial barrier for practical applications. This paper introduces FASTTRACK, a novel approach that harnesses the capabilities of Large Language Models (LLMs) to validate supportive evidence for queries and at the same time clusters the training database towards a reduced extent for LLMs to trace facts. Our experiments show that FASTTRACK substantially outperforms existing methods in both accuracy and efficiency, achieving more than 100\% improvement in F1 score over the state-of-the-art methods while being X33 faster than \texttt{TracIn}.