Classification with High-Dimensional Sparse Samples

TL;DR

This paper analyzes the conditions under which high-dimensional binary classification is consistent when sample sizes and alphabet size grow, introducing classifiers with different error decay properties.

Contribution

It extends previous results to the case of unequal training and test sample sizes and introduces classifiers with improved error decay in sparse high-dimensional settings.

Findings

Asymptotic consistency requires m=o(min{N^2, Nn})

Finer error decay results for sparse samples: -log(P_e) proportional to min{N^2, Nn}/m

Weighted coincidence classifier achieves positive error exponent J

Abstract

The task of the binary classification problem is to determine which of two distributions has generated a length-$n$ test sequence. The two distributions are unknown; two training sequences of length $N$, one from each distribution, are observed. The distributions share an alphabet of size $m$, which is significantly larger than $n$ and $N$. How does $N,n,m$ affect the probability of classification error? We characterize the achievable error rate in a high-dimensional setting in which $N,n,m$ all tend to infinity, under the assumption that probability of any symbol is $O(m^{-1})$. The results are: 1. There exists an asymptotically consistent classifier if and only if $m=o(\min\{N^2,Nn\})$. This extends the previous consistency result in [1] to the case $N\neq n$. 2. For the sparse sample case where $\max\{n,N\}=o(m)$, finer results are obtained: The best achievable probability of error decays as $-\log(P_e)=J \min\{N^2, Nn\}(1+o(1))/m$ with $J>0$. 3. A weighted coincidence-based classifier has non-zero generalized error exponent $J$. 4. The $\ell_2$-norm based classifier has J=0.