Learning from Non-Random Data in Hilbert Spaces: An Optimal Recovery Perspective

TL;DR

This paper develops a framework for regression in Hilbert spaces that minimizes worst-case error without relying on probabilistic data assumptions, connecting optimal recovery with kernel methods.

Contribution

It introduces a semidefinite programming approach for worst-case error calculation and links optimal recovery to kernel ridgeless regression in Hilbert spaces.

Findings

Semidefinite program for finite-dimensional Hilbert spaces.

Optimal recovery formula matches kernel ridgeless regression in some cases.

Numerical experiments validate theoretical results.

Abstract

The notion of generalization in classical Statistical Learning is often attached to the postulate that data points are independent and identically distributed (IID) random variables. While relevant in many applications, this postulate may not hold in general, encouraging the development of learning frameworks that are robust to non-IID data. In this work, we consider the regression problem from an Optimal Recovery perspective. Relying on a model assumption comparable to choosing a hypothesis class, a learner aims at minimizing the worst-case error, without recourse to any probabilistic assumption on the data. We first develop a semidefinite program for calculating the worst-case error of any recovery map in finite-dimensional Hilbert spaces. Then, for any Hilbert space, we show that Optimal Recovery provides a formula which is user-friendly from an algorithmic point-of-view, as long as the hypothesis class is linear. Interestingly, this formula coincides with kernel ridgeless regression in some cases, proving that minimizing the average error and worst-case error can yield the same solution. We provide numerical experiments in support of our theoretical findings.