Inverse classification with logistic and softmax classifiers: efficient optimization

TL;DR

This paper presents efficient methods for solving inverse classification problems, such as counterfactuals and adversarial examples, for logistic and softmax classifiers, enabling real-time applications with high-dimensional data.

Contribution

It introduces closed-form and fast iterative solutions for inverse classification problems specific to logistic and softmax models, significantly improving computational efficiency.

Findings

Closed-form solution for logistic regression inverse classification.

Fast iterative method for softmax classifier inverse classification.

Achieves near machine precision solutions in milliseconds to seconds.

Abstract

In recent years, a certain type of problems have become of interest where one wants to query a trained classifier. Specifically, one wants to find the closest instance to a given input instance such that the classifier's predicted label is changed in a desired way. Examples of these "inverse classification" problems are counterfactual explanations, adversarial examples and model inversion. All of them are fundamentally optimization problems over the input instance vector involving a fixed classifier, and it is of interest to achieve a fast solution for interactive or real-time applications. We focus on solving this problem efficiently with the squared Euclidean distance for two of the most widely used classifiers: logistic regression and softmax classifier. Owing to special properties of these models, we show that the optimization can be solved in closed form for logistic regression, and iteratively but extremely fast for the softmax classifier. This allows us to solve either case exactly (to nearly machine precision) in a runtime of milliseconds to around a second even for very high-dimensional instances and many classes.