A Structural Approach to Coordinate-Free Statistics

TL;DR

This paper develops a general, coordinate-free framework for statistical learning in topological vector spaces, extending classical results like OLS and Gaussian disintegrations, and applies it to machine learning tasks such as kriging and SVMs.

Contribution

It introduces a novel, structural approach to coordinate-free statistics, generalizing classical estimators and disintegrations to broad topological vector spaces, with applications to machine learning.

Findings

Extended OLS to general topological vector spaces.

Proved continuous disintegration for Gaussian measures.

Developed a coordinate-free SVM classifier.

Abstract

We consider the question of learning in general topological vector spaces. By exploiting known (or parametrized) covariance structures, our Main Theorem demonstrates that any continuous linear map corresponds to a certain isomorphism of embedded Hilbert spaces. By inverting this isomorphism and extending continuously, we construct a version of the Ordinary Least Squares estimator in absolute generality. Our Gauss-Markov theorem demonstrates that OLS is a "best linear unbiased estimator", extending the classical result. We construct a stochastic version of the OLS estimator, which is a continuous disintegration exactly for the class of "uncorrelated implies independent" (UII) measures. As a consequence, Gaussian measures always exhibit continuous disintegrations through continuous linear maps, extending a theorem of the first author. Applying this framework to some problems in machine learning, we prove a useful representation theorem for covariance tensors, and show that OLS defines a good kriging predictor for vector-valued arrays on general index spaces. We also construct a support-vector machine classifier in this setting. We hope that our article shines light on some deeper connections between probability theory, statistics and machine learning, and may serve as a point of intersection for these three communities.