Hidden in Plain Sight: Exploring Chat History Tampering in Interactive Language Models

TL;DR

This paper reveals how chat history tampering in LLMs like ChatGPT and Llama can manipulate model behavior, introduces a systematic method to inject fake history, and demonstrates significant impact on model outputs.

Contribution

It proposes a novel prompt template search method using LLM-guided genetic algorithms to enable chat history tampering in black-box LLMs.

Findings

Tampering can increase disallowed response success rate to 97% on ChatGPT.

Effective templates can be automatically generated and optimized.

Chat history tampering affects model behavior over time.

Abstract

Large Language Models (LLMs) such as ChatGPT and Llama have become prevalent in real-world applications, exhibiting impressive text generation performance. LLMs are fundamentally developed from a scenario where the input data remains static and unstructured. To behave interactively, LLM-based chat systems must integrate prior chat history as context into their inputs, following a pre-defined structure. However, LLMs cannot separate user inputs from context, enabling chat history tampering. This paper introduces a systematic methodology to inject user-supplied history into LLM conversations without any prior knowledge of the target model. The key is to utilize prompt templates that can well organize the messages to be injected, leading the target LLM to interpret them as genuine chat history. To automatically search for effective templates in a WebUI black-box setting, we propose the LLM-Guided Genetic Algorithm (LLMGA) that leverages an LLM to generate and iteratively optimize the templates. We apply the proposed method to popular real-world LLMs including ChatGPT and Llama-2/3. The results show that chat history tampering can enhance the malleability of the model's behavior over time and greatly influence the model output. For example, it can improve the success rate of disallowed response elicitation up to 97% on ChatGPT. Our findings provide insights into the challenges associated with the real-world deployment of interactive LLMs.