Revitalizing Black-Box Interpretability: Actionable Interpretability for LLMs via Proxy Models

TL;DR

This paper introduces a cost-effective proxy framework for LLM interpretability, enabling scalable, high-fidelity explanations that guide model optimization tasks like prompt compression and data sanitation.

Contribution

It proposes a novel proxy-based interpretability method with a verification mechanism, significantly reducing computational costs while maintaining explanation fidelity.

Findings

Proxy explanations achieve over 90% fidelity with 11% of the oracle's cost.

The framework effectively guides prompt compression and poisoned data removal.

Open-source code and datasets support future research.

Abstract

Post-hoc explanations provide transparency and are essential for guiding model optimization, such as prompt engineering and data sanitation. However, applying model-agnostic techniques to Large Language Models (LLMs) is hindered by prohibitive computational costs, rendering these tools dormant for real-world applications. To revitalize model-agnostic interpretability, we propose a budget-friendly proxy framework that leverages efficient models to approximate the decision boundaries of expensive LLMs. We introduce a screen-and-apply mechanism to statistically verify local alignment before deployment. Our empirical evaluation confirms that proxy explanations achieve over 90% fidelity with only 11% of the oracle's cost. Building on this foundation, we demonstrate the actionable utility of our framework in prompt compression and poisoned example removal. Results show that reliable proxy explanations effectively guide optimization, transforming interpretability from a passive observation tool into a scalable primitive for LLM development. Additionally, we open-source code and datasets to facilitate future research.