AutoMIA: Improved Baselines for Membership Inference Attack via Agentic Self-Exploration

TL;DR

AutoMIA introduces an agentic, automated framework for membership inference attacks that self-explores and refines strategies, outperforming static heuristics across various models.

Contribution

It presents AutoMIA, a novel, model-agnostic framework that automates membership inference attack strategies through self-exploration and closed-loop refinement.

Findings

AutoMIA matches or exceeds state-of-the-art attack performance.

Eliminates manual feature engineering in membership inference.

Demonstrates robustness across different large models.

Abstract

Membership Inference Attacks (MIAs) serve as a fundamental auditing tool for evaluating training data leakage in machine learning models. However, existing methodologies predominantly rely on static, handcrafted heuristics that lack adaptability, often leading to suboptimal performance when transferred across different large models. In this work, we propose AutoMIA, an agentic framework that reformulates membership inference as an automated process of self-exploration and strategy evolution. Given high-level scenario specifications, AutoMIA self-explores the attack space by generating executable logits-level strategies and progressively refining them through closed-loop evaluation feedback. By decoupling abstract strategy reasoning from low-level execution, our framework enables a systematic, model-agnostic traversal of the attack search space. Extensive experiments demonstrate that AutoMIA consistently matches or outperforms state-of-the-art baselines while eliminating the need for manual feature engineering.