Token-Efficient Change Detection in LLM APIs

TL;DR

This paper introduces a cost-effective black-box change detection method for LLM APIs using Border Inputs, achieving high performance with significantly reduced costs and no access to model internals.

Contribution

The paper proposes B3IT, a novel black-box change detection scheme leveraging Border Inputs, which reduces costs by 30x while maintaining state-of-the-art accuracy.

Findings

Border Inputs are easily identified for non-reasoning endpoints.

B3IT achieves performance comparable to grey-box methods.

Cost is reduced by a factor of 30 compared to existing approaches.

Abstract

Remote change detection in LLMs is a difficult problem. Existing methods are either too expensive for deployment at scale, or require initial white-box access to model weights or grey-box access to log probabilities. We aim to achieve both low cost and strict black-box operation, observing only output tokens. Our approach hinges on specific inputs we call Border Inputs, for which there exists more than one output top token. From a statistical perspective, optimal change detection depends on the model's Jacobian and the Fisher information of the output distribution. Analyzing these quantities in low-temperature regimes shows that border inputs enable powerful change detection tests. Building on this insight, we propose the Black-Box Border Input Tracking (B3IT) scheme. Extensive in-vivo and in-vitro experiments show that border inputs are easily found for non-reasoning tested endpoints, and achieve performance on par with the best available grey-box approaches. B3IT reduces costs by $30\times$ compared to existing methods, while operating in a strict black-box setting.