Weight Vector Tuning and Asymptotic Analysis of Binary Linear Classifiers

TL;DR

This paper introduces a scalar-based weight vector tuning method for binary linear classifiers, improving performance and computational efficiency, especially for LDA under high noise, supported by asymptotic analysis using random matrix theory.

Contribution

It proposes a novel scalar parameterization for weight vector tuning in linear classifiers, simplifying hyperparameter optimization and enhancing classifier performance.

Findings

Weight vector tuning compensates for non-optimal hyperparameters.

Tuning improves LDA performance under high noise.

Asymptotic misclassification probability converges to a data-dependent limit.

Abstract

Unlike its intercept, a linear classifier's weight vector cannot be tuned by a simple grid search. Hence, this paper proposes weight vector tuning of a generic binary linear classifier through the parameterization of a decomposition of the discriminant by a scalar which controls the trade-off between conflicting informative and noisy terms. By varying this parameter, the original weight vector is modified in a meaningful way. Applying this method to a number of linear classifiers under a variety of data dimensionality and sample size settings reveals that the classification performance loss due to non-optimal native hyperparameters can be compensated for by weight vector tuning. This yields computational savings as the proposed tuning method reduces to tuning a scalar compared to tuning the native hyperparameter, which may involve repeated weight vector generation along with its burden of optimization, dimensionality reduction, etc., depending on the classifier. It is also found that weight vector tuning significantly improves the performance of Linear Discriminant Analysis (LDA) under high estimation noise. Proceeding from this second finding, an asymptotic study of the misclassification probability of the parameterized LDA classifier in the growth regime where the data dimensionality and sample size are comparable is conducted. Using random matrix theory, the misclassification probability is shown to converge to a quantity that is a function of the true statistics of the data. Additionally, an estimator of the misclassification probability is derived. Finally, computationally efficient tuning of the parameter using this estimator is demonstrated on real data.