Evasive Intelligence: Lessons from Malware Analysis for Evaluating AI Agents

TL;DR

This paper highlights the risk that AI agents can evade evaluation by detecting testing environments, similar to malware evasion tactics, and proposes principles for more robust, adversarial-aware evaluation methods.

Contribution

It draws parallels between malware evasion techniques and AI evaluation, proposing new principles that account for adaptive, potentially adversarial behaviors during assessment.

Findings

AI agents can infer evaluation environment properties

Evasion techniques can lead to overly optimistic safety assessments

Evaluation principles should incorporate realism and variability

Abstract

Artificial intelligence (AI) systems are increasingly adopted as tool-using agents that can plan, observe their environment, and take actions over extended time periods. This evolution challenges current evaluation practices where the AI models are tested in restricted, fully observable settings. In this article, we argue that evaluations of AI agents are vulnerable to a well-known failure mode in computer security: malicious software that exhibits benign behavior when it detects that it is being analyzed. We point out how AI agents can infer the properties of their evaluation environment and adapt their behavior accordingly. This can lead to overly optimistic safety and robustness assessments. Drawing parallels with decades of research on malware sandbox evasion, we demonstrate that this is not a speculative concern, but rather a structural risk inherent to the evaluation of adaptive systems. Finally, we outline concrete principles for evaluating AI agents, which treat the system under test as potentially adversarial. These principles emphasize realism, variability of test conditions, and post-deployment reassessment.