Inhibitory Attacks on Backdoor-based Fingerprinting for Large Language Models

TL;DR

This paper introduces two novel attacks, TFA and SVA, that effectively compromise LLM fingerprinting methods, highlighting the need for more robust protection techniques for ensemble-based LLMs.

Contribution

The paper presents the first comprehensive analysis of fingerprinting attacks on ensemble LLMs and proposes two effective methods to inhibit fingerprint responses.

Findings

TFA and SVA successfully inhibit fingerprint responses.

Proposed attacks outperform existing methods.

Ensemble LLM fingerprinting requires improved robustness.

Abstract

The widespread adoption of Large Language Model (LLM) in commercial and research settings has intensified the need for robust intellectual property protection. Backdoor-based LLM fingerprinting has emerged as a promising solution for this challenge. In practical application, the low-cost multi-model collaborative technique, LLM ensemble, combines diverse LLMs to leverage their complementary strengths, garnering significant attention and practical adoption. Unfortunately, the vulnerability of existing LLM fingerprinting for the ensemble scenario is unexplored. In order to comprehensively assess the robustness of LLM fingerprinting, in this paper, we propose two novel fingerprinting attack methods: token filter attack (TFA) and sentence verification attack (SVA). The TFA gets the next token from a unified set of tokens created by the token filter mechanism at each decoding step. The SVA filters out fingerprint responses through a sentence verification mechanism based on perplexity and voting. Experimentally, the proposed methods effectively inhibit the fingerprint response while maintaining ensemble performance. Compared with state-of-the-art attack methods, the proposed method can achieve better performance. The findings necessitate enhanced robustness in LLM fingerprinting.