Incremental Sparse Bayesian Ordinal Regression

TL;DR

This paper introduces ISBOR, an incremental sparse Bayesian method for ordinal regression that efficiently learns relevant basis functions, improves scalability, and provides uncertainty estimates, outperforming existing basis function-based approaches.

Contribution

The paper presents a novel incremental sparse Bayesian ordinal regression algorithm that automatically optimizes hyper-parameters and avoids large matrix inversions, enhancing scalability and accuracy.

Findings

ISBOR achieves higher accuracy than existing methods.

It provides automatic hyper-parameter tuning.

The approach is computationally efficient on large datasets.

Abstract

Ordinal Regression (OR) aims to model the ordering information between different data categories, which is a crucial topic in multi-label learning. An important class of approaches to OR models the problem as a linear combination of basis functions that map features to a high dimensional non-linear space. However, most of the basis function-based algorithms are time consuming. We propose an incremental sparse Bayesian approach to OR tasks and introduce an algorithm to sequentially learn the relevant basis functions in the ordinal scenario. Our method, called Incremental Sparse Bayesian Ordinal Regression (ISBOR), automatically optimizes the hyper-parameters via the type-II maximum likelihood method. By exploiting fast marginal likelihood optimization, ISBOR can avoid big matrix inverses, which is the main bottleneck in applying basis function-based algorithms to OR tasks on large-scale datasets. We show that ISBOR can make accurate predictions with parsimonious basis functions while offering automatic estimates of the prediction uncertainty. Extensive experiments on synthetic and real word datasets demonstrate the efficiency and effectiveness of ISBOR compared to other basis function-based OR approaches.