Technical Report on the CleverHans v2.1.0 Adversarial Examples Library

TL;DR

This technical report introduces CleverHans v2.1.0, a library offering standardized tools for generating adversarial examples and benchmarking model robustness, facilitating consistent evaluation of adversarial defenses.

Contribution

The report details the core functionalities, versioning, and benchmarking procedures of the CleverHans library, promoting standardized adversarial testing in machine learning.

Findings

Provides a unified framework for adversarial attacks and defenses.

Enables consistent benchmarking of model robustness.

Facilitates comparison of adversarial training methods.

Abstract

CleverHans is a software library that provides standardized reference implementations of adversarial example construction techniques and adversarial training. The library may be used to develop more robust machine learning models and to provide standardized benchmarks of models' performance in the adversarial setting. Benchmarks constructed without a standardized implementation of adversarial example construction are not comparable to each other, because a good result may indicate a robust model or it may merely indicate a weak implementation of the adversarial example construction procedure. This technical report is structured as follows. Section 1 provides an overview of adversarial examples in machine learning and of the CleverHans software. Section 2 presents the core functionalities of the library: namely the attacks based on adversarial examples and defenses to improve the robustness of machine learning models to these attacks. Section 3 describes how to report benchmark results using the library. Section 4 describes the versioning system.