Statistical Mechanical Development of a Sparse Bayesian Classifier

TL;DR

This paper introduces a Bayesian-based sparse classifier that reduces data dimensionality by pruning redundant features, improving generalization, and demonstrating practical effectiveness in colon cancer classification.

Contribution

It develops a theoretically grounded, computationally feasible sparse Bayesian classifier with proven near-optimal performance and practical validation on real-world data.

Findings

Effective data dimensionality reduction through pruning

Near-optimal performance in large systems

Successful application to colon cancer classification

Abstract

The demand for extracting rules from high dimensional real world data is increasing in various fields. However, the possible redundancy of such data sometimes makes it difficult to obtain a good generalization ability for novel samples. To resolve this problem, we provide a scheme that reduces the effective dimensions of data by pruning redundant components for bicategorical classification based on the Bayesian framework. First, the potential of the proposed method is confirmed in ideal situations using the replica method. Unfortunately, performing the scheme exactly is computationally difficult. So, we next develop a tractable approximation algorithm, which turns out to offer nearly optimal performance in ideal cases when the system size is large. Finally, the efficacy of the developed classifier is experimentally examined for a real world problem of colon cancer classification, which shows that the developed method can be practically useful.