On Continuity of Robust and Accurate Classifiers

TL;DR

This paper explores the relationship between the continuity of classifiers and their robustness and accuracy, proposing that continuity may hinder optimal robustness and providing empirical and theoretical insights into this phenomenon.

Contribution

It introduces a framework for analyzing the continuity of hypotheses in learning theory and demonstrates that discontinuous hypotheses can outperform continuous ones in robustness and accuracy.

Findings

Continuous hypotheses are less effective than discontinuous ones in some tasks.

Robustness and accuracy may conflict with the continuity of the hypothesis.

Theoretical analysis links adversarial examples to the continuity properties of classifiers.

Abstract

The reliability of a learning model is key to the successful deployment of machine learning in various applications. However, it is difficult to describe the phenomenon due to the complicated nature of the problems in machine learning. It has been shown that adversarial training can improve the robustness of the hypothesis. However, this improvement usually comes at the cost of decreased performance on natural samples. Hence, it has been suggested that robustness and accuracy of a hypothesis are at odds with each other. In this paper, we put forth the alternative proposal that it is the continuity of a hypothesis that is incompatible with its robustness and accuracy in many of these scenarios. In other words, a continuous function cannot effectively learn the optimal robust hypothesis. We introduce a framework for a rigorous study of harmonic and holomorphic hypothesis in learning theory terms and provide empirical evidence that continuous hypotheses do not perform as well as discontinuous hypotheses in some common machine learning tasks. From a practical point of view, our results suggests that a robust and accurate learning rule would train different continuous hypotheses for different regions of the domain. From a theoretical perspective, our analysis explains the adversarial examples phenomenon in these situations as a conflict between the continuity of a sequence of functions and its uniform convergence to a discontinuous function. Given that many of the contemporary machine learning models are continuous functions, it is important to theoretically study the continuity of robust and accurate classifiers as it is consequential in their construction, analysis and evaluation.