Automated Membership Inference Attacks: Discovering MIA Signal Computations using LLM Agents

TL;DR

AutoMIA employs large language model agents to automate and enhance the discovery of membership inference attack strategies, achieving state-of-the-art performance and revealing new vulnerabilities in machine learning models.

Contribution

This paper introduces AutoMIA, the first framework using LLM agents to automate MIA design, enabling systematic exploration and discovery of novel attack strategies.

Findings

AutoMIA discovers MIAs with up to 0.18 higher AUC than existing methods.

LLM agents effectively explore a vast space of attack strategies.

AutoMIA demonstrates the scalability and effectiveness of LLMs in security research.

Abstract

Membership inference attacks (MIAs), which enable adversaries to determine whether specific data points were part of a model's training dataset, have emerged as an important framework to understand, assess, and quantify the potential information leakage associated with machine learning systems. Designing effective MIAs is a challenging task that usually requires extensive manual exploration of model behaviors to identify potential vulnerabilities. In this paper, we introduce AutoMIA -- a novel framework that leverages large language model (LLM) agents to automate the design and implementation of new MIA signal computations. By utilizing LLM agents, we can systematically explore a vast space of potential attack strategies, enabling the discovery of novel strategies. Our experiments demonstrate AutoMIA can successfully discover new MIAs that are specifically tailored to user-configured target model and dataset, resulting in improvements of up to 0.18 in absolute AUC over existing MIAs. This work provides the first demonstration that LLM agents can serve as an effective and scalable paradigm for designing and implementing MIAs with SOTA performance, opening up new avenues for future exploration.