AttackPilot: Autonomous Inference Attacks Against ML Services With LLM-Based Agents

TL;DR

AttackPilot is an autonomous LLM-based agent that performs inference attacks on ML services, achieving near-expert performance with minimal cost, and can help non-experts assess ML service risks effectively.

Contribution

This work introduces AttackPilot, the first autonomous agent capable of conducting inference attacks on ML services without human intervention, utilizing LLMs like GPT-4o.

Findings

100% task completion rate on 20 services

Near-expert attack performance achieved

Low average token cost of $0.627 per run

Abstract

Inference attacks have been widely studied and offer a systematic risk assessment of ML services; however, their implementation and the attack parameters for optimal estimation are challenging for non-experts. The emergence of advanced large language models presents a promising yet largely unexplored opportunity to develop autonomous agents as inference attack experts, helping address this challenge. In this paper, we propose AttackPilot, an autonomous agent capable of independently conducting inference attacks without human intervention. We evaluate it on 20 target services. The evaluation shows that our agent, using GPT-4o, achieves a 100.0% task completion rate and near-expert attack performance, with an average token cost of only $0.627 per run. The agent can also be powered by many other representative LLMs and can adaptively optimize its strategy under service constraints. We further perform trace analysis, demonstrating that design choices, such as a multi-agent framework and task-specific action spaces, effectively mitigate errors such as bad plans, inability to follow instructions, task context loss, and hallucinations. We anticipate that such agents could empower non-expert ML service providers, auditors, or regulators to systematically assess the risks of ML services without requiring deep domain expertise.