On the Trade-Off Between Transparency and Security in Adversarial Machine Learning

TL;DR

This paper explores the inherent conflict between transparency and security in adversarial machine learning, demonstrating through empirical and game-theoretic analysis that increased transparency can compromise security.

Contribution

It provides a large-scale empirical evaluation of transferability in adversarial attacks and models the transparency-security trade-off using game theory, revealing that transparency can weaken security.

Findings

Attackers succeed more when matching defender's model status

Obscurity can enhance defender security in adversarial settings

Game-theoretic analysis confirms transparency-security trade-off

Abstract

Transparency and security are both central to Responsible AI, but they may conflict in adversarial settings. We investigate the strategic effect of transparency for agents through the lens of transferable adversarial example attacks. In transferable adversarial example attacks, attackers maliciously perturb their inputs using surrogate models to fool a defender's target model. These models can be defended or undefended, with both players having to decide which to use. Using a large-scale empirical evaluation of nine attacks across 181 models, we find that attackers are more successful when they match the defender's decision; hence, obscurity could be beneficial to the defender. With game theory, we analyze this trade-off between transparency and security by modeling this problem as both a Nash game and a Stackelberg game, and comparing the expected outcomes. Our analysis confirms that only knowing whether a defender's model is defended or not can sometimes be enough to damage its security. This result serves as an indicator of the general trade-off between transparency and security, suggesting that transparency in AI systems can be at odds with security. Beyond adversarial machine learning, our work illustrates how game-theoretic reasoning can uncover conflicts between transparency and security.