Interpolating Discriminant Functions in High-Dimensional Gaussian Latent Mixtures

TL;DR

This paper studies high-dimensional binary classification under a Gaussian mixture model, proposing a corrected estimator that interpolates training data and achieves minimax optimality, with performance depending on label encoding.

Contribution

It introduces a correction method for interpolating discriminant functions in high-dimensional Gaussian mixtures, improving estimation of the hyperplane intercept.

Findings

Corrected estimator achieves minimax optimality in many scenarios.

Interpolation property depends on label encoding.

Naive plug-in estimate fails to consistently estimate the intercept.

Abstract

This paper considers binary classification of high-dimensional features under a postulated model with a low-dimensional latent Gaussian mixture structure and non-vanishing noise. A generalized least squares estimator is used to estimate the direction of the optimal separating hyperplane. The estimated hyperplane is shown to interpolate on the training data. While the direction vector can be consistently estimated as could be expected from recent results in linear regression, a naive plug-in estimate fails to consistently estimate the intercept. A simple correction, that requires an independent hold-out sample, renders the procedure minimax optimal in many scenarios. The interpolation property of the latter procedure can be retained, but surprisingly depends on the way the labels are encoded.