On the amplification of security and privacy risks by post-hoc explanations in machine learning models

TL;DR

This paper systematically analyzes how post-hoc explanation methods in machine learning models can unintentionally leak sensitive information and enable security attacks, proposing new attack techniques and quantifying the associated risks.

Contribution

It introduces novel explanation-guided black-box attacks, quantifies privacy leakage, and demonstrates significant improvements over prior methods in attack efficiency and information leakage.

Findings

Explanation methods can be exploited for black-box evasion attacks with fewer queries.

Membership information leakage is significantly higher than previously reported.

Explanation-guided model extraction attacks require fewer queries, increasing attack efficiency.

Abstract

A variety of explanation methods have been proposed in recent years to help users gain insights into the results returned by neural networks, which are otherwise complex and opaque black-boxes. However, explanations give rise to potential side-channels that can be leveraged by an adversary for mounting attacks on the system. In particular, post-hoc explanation methods that highlight input dimensions according to their importance or relevance to the result also leak information that weakens security and privacy. In this work, we perform the first systematic characterization of the privacy and security risks arising from various popular explanation techniques. First, we propose novel explanation-guided black-box evasion attacks that lead to 10 times reduction in query count for the same success rate. We show that the adversarial advantage from explanations can be quantified as a reduction in the total variance of the estimated gradient. Second, we revisit the membership information leaked by common explanations. Contrary to observations in prior studies, via our modified attacks we show significant leakage of membership information (above 100% improvement over prior results), even in a much stricter black-box setting. Finally, we study explanation-guided model extraction attacks and demonstrate adversarial gains through a large reduction in query count.