Low-Complexity Nonparametric Bayesian Online Prediction with Universal Guarantees

TL;DR

This paper introduces a low-complexity nonparametric Bayesian online predictor for discrete labels with universal guarantees, leveraging feature space discretization and recursive Bayesian methods for efficient, adaptive prediction.

Contribution

It presents a novel online prediction algorithm that automatically learns relevant feature scales and achieves universal asymptotic performance with logarithmic time complexity.

Findings

Achieves asymptotic normalized log loss close to true conditional entropy.

Operates with $O( ext{log} n)$ time complexity per sample.

Outperforms standard methods in experiments on challenging datasets.

Abstract

We propose a novel nonparametric online predictor for discrete labels conditioned on multivariate continuous features. The predictor is based on a feature space discretization induced by a full-fledged k-d tree with randomly picked directions and a recursive Bayesian distribution, which allows to automatically learn the most relevant feature scales characterizing the conditional distribution. We prove its pointwise universality, i.e., it achieves a normalized log loss performance asymptotically as good as the true conditional entropy of the labels given the features. The time complexity to process the $n$-th sample point is $O(\log n)$ in probability with respect to the distribution generating the data points, whereas other exact nonparametric methods require to process all past observations. Experiments on challenging datasets show the computational and statistical efficiency of our algorithm in comparison to standard and state-of-the-art methods.