Distance Learner: Incorporating Manifold Prior to Model Training

TL;DR

Distance Learner introduces a novel approach that predicts the distance of data points from class manifolds, improving classification boundaries and robustness against adversarial attacks in high-dimensional data.

Contribution

The paper proposes Distance Learner, a method that incorporates manifold priors by predicting distances to class manifolds, enhancing classification and out-of-distribution detection.

Findings

Outperforms standard classifiers in synthetic datasets

Achieves robustness comparable to adversarial training methods

Learns more meaningful classification boundaries

Abstract

The manifold hypothesis (real world data concentrates near low-dimensional manifolds) is suggested as the principle behind the effectiveness of machine learning algorithms in very high dimensional problems that are common in domains such as vision and speech. Multiple methods have been proposed to explicitly incorporate the manifold hypothesis as a prior in modern Deep Neural Networks (DNNs), with varying success. In this paper, we propose a new method, Distance Learner, to incorporate this prior for DNN-based classifiers. Distance Learner is trained to predict the distance of a point from the underlying manifold of each class, rather than the class label. For classification, Distance Learner then chooses the class corresponding to the closest predicted class manifold. Distance Learner can also identify points as being out of distribution (belonging to neither class), if the distance to the closest manifold is higher than a threshold. We evaluate our method on multiple synthetic datasets and show that Distance Learner learns much more meaningful classification boundaries compared to a standard classifier. We also evaluate our method on the task of adversarial robustness, and find that it not only outperforms standard classifier by a large margin, but also performs at par with classifiers trained via state-of-the-art adversarial training.