Transfer learning via Regularized Linear Discriminant Analysis

TL;DR

This paper introduces novel transfer learning methods for high-dimensional linear discriminant analysis, leveraging source data to improve classification accuracy in small sample, high-dimensional settings.

Contribution

It proposes regularized random-effects LDA with strategies for optimal weight determination, supported by asymptotic analysis and extensive numerical validation.

Findings

Improved classification accuracy in high-dimensional, small-sample scenarios.

Effective weight strategies derived from asymptotic analysis.

Successful application to proteomics-based cardiovascular risk prediction.

Abstract

Linear discriminant analysis is a widely used method for classification. However, the high dimensionality of predictors combined with small sample sizes often results in large classification errors. To address this challenge, it is crucial to leverage data from related source models to enhance the classification performance of a target model. We propose to address this problem in the framework of transfer learning. In this paper, we present novel transfer learning methods via regularized random-effects linear discriminant analysis, where the discriminant direction is estimated as a weighted combination of ridge estimates obtained from both the target and source models. Multiple strategies for determining these weights are introduced and evaluated, including one that minimizes the estimation risk of the discriminant vector and another that minimizes the classification error. Utilizing results from random matrix theory, we explicitly derive the asymptotic values of these weights and the associated classification error rates in the high-dimensional setting, where $p/n \rightarrow \gamma$, with $p$ representing the predictor dimension and $n$ the sample size. We also provide geometric interpretations of various weights and a guidance on which weights to choose. Extensive numerical studies, including simulations and analysis of proteomics-based 10-year cardiovascular disease risk classification, demonstrate the effectiveness of the proposed approach.