What Really is a Member? Discrediting Membership Inference via Poisoning

TL;DR

This paper demonstrates that membership inference tests for language models can be easily poisoned, undermining their reliability even under relaxed definitions of membership, and reveals a fundamental accuracy-robustness trade-off.

Contribution

It introduces a poisoning attack against membership inference tests and provides theoretical and empirical evidence of its effectiveness, challenging their trustworthiness.

Findings

Poisoning can cause inference tests to fail significantly

There is a trade-off between test accuracy and robustness to poisoning

Empirical validation shows attacks degrade test performance below random chance

Abstract

Membership inference tests aim to determine whether a particular data point was included in a language model's training set. However, recent works have shown that such tests often fail under the strict definition of membership based on exact matching, and have suggested relaxing this definition to include semantic neighbors as members as well. In this work, we show that membership inference tests are still unreliable under this relaxation - it is possible to poison the training dataset in a way that causes the test to produce incorrect predictions for a target point. We theoretically reveal a trade-off between a test's accuracy and its robustness to poisoning. We also present a concrete instantiation of this poisoning attack and empirically validate its effectiveness. Our results show that it can degrade the performance of existing tests to well below random.