Adaptive Distraction: Probing LLM Contextual Robustness with Automated Tree Search

TL;DR

This paper introduces a dynamic tree search-based framework to generate adaptive distractions for LLMs, revealing their vulnerability to context shifts and evaluating mitigation strategies to improve robustness.

Contribution

We propose a novel tree search-guided distraction generation method that systematically tests LLMs' contextual robustness without altering original inputs.

Findings

Distractions cause over 45% performance drop in mainstream LLMs.

Prompt-based mitigation yields limited improvements.

Post-training methods like DPO significantly enhance robustness.

Abstract

Large Language Models (LLMs) often struggle to maintain their original performance when faced with semantically coherent but task-irrelevant contextual information. Although prior studies have explored this issue using fixed-template or retrieval-based distractions, such static methods show limited effectiveness against contemporary models. To address this problem, we propose a dynamic distraction generation framework based on tree search, where the generation process is guided by model behavior. Without modifying the original question or answer, the method efficiently produces challenging adaptive distractions across multiple datasets, enabling systematic stress testing of LLMs' contextual robustness. Experiments on four benchmarks demonstrate that the generated distractions lead to an average performance drop of over 45\% for mainstream models. Further comparisons of mitigation strategies show that prompt-based optimization methods yield limited gains, whereas post-training approaches (e.g., DPO) significantly enhance the model's contextual robustness. The results indicate that these issues do not stem from knowledge deficits in LLMs, but from a fundamental inability to maintain consistent reasoning under contextual distraction, posing a major challenge to the reliability of LLMs in real-world applications. The code is publicly available at https://github.com/wyf23187/Adaptive_Distractions.