How to Implement A Priori Information: A Statistical Mechanics Approach

TL;DR

This paper introduces a flexible framework for incorporating complex prior knowledge into learning models using a statistical mechanics approach, especially effective with limited data and complex tasks.

Contribution

It presents a novel method to decompose and combine prior information into quadratic components, enabling the construction of both convex and non-convex error functionals for improved learning.

Findings

The framework handles ambiguous priors with non-convex functionals.

Numerical solutions to stationarity equations are feasible despite their complexity.

The approach extends classical regularization by incorporating non-Gaussian processes.

Abstract

A new general framework is presented for implementing complex a priori knowledge, having in mind especially situations where the number of available training data is small compared to the complexity of the learning task. A priori information is hereby decomposed into simple components represented by quadratic building blocks (quadratic concepts) which are then combined by conjunctions and disjunctions to built more complex, problem specific error functionals. While conjunction of quadratic concepts leads to classical quadratic regularization functionals, disjunctions, representing ambiguous priors, result in non--convex error functionals. These go beyond classical quadratic regularization approaches and correspond, in Bayesian interpretation, to non--gaussian processes. Numerical examples show that the resulting stationarity equations, despite being in general nonlinear, inhomogeneous (integro--)differential equations, are not necessarily difficult to solve. Appendix A relates the formalism of statistical mechanics to statistics and Appendix B describes the framework of Bayesian decision theory.