Quadruply Stochastic Gradient Method for Large Scale Nonlinear Semi-Supervised Ordinal Regression AUC Optimization

TL;DR

This paper introduces a scalable doubly stochastic gradient method for large-scale semi-supervised ordinal regression that optimizes AUC, providing theoretical convergence guarantees and demonstrating efficiency and effectiveness on benchmark datasets.

Contribution

It proposes a novel unbiased AUC optimization objective for semi-supervised ordinal regression and a scalable doubly stochastic gradient algorithm with proven convergence.

Findings

Method converges at rate O(1/t)

Achieves comparable generalization performance to existing methods

Demonstrates efficiency and scalability on large datasets

Abstract

Semi-supervised ordinal regression (S$^2$OR) problems are ubiquitous in real-world applications, where only a few ordered instances are labeled and massive instances remain unlabeled. Recent researches have shown that directly optimizing concordance index or AUC can impose a better ranking on the data than optimizing the traditional error rate in ordinal regression (OR) problems. In this paper, we propose an unbiased objective function for S$^2$OR AUC optimization based on ordinal binary decomposition approach. Besides, to handle the large-scale kernelized learning problems, we propose a scalable algorithm called QS$^3$ORAO using the doubly stochastic gradients (DSG) framework for functional optimization. Theoretically, we prove that our method can converge to the optimal solution at the rate of $O(1/t)$, where $t$ is the number of iterations for stochastic data sampling. Extensive experimental results on various benchmark and real-world datasets also demonstrate that our method is efficient and effective while retaining similar generalization performance.