Functional Risk Minimization

TL;DR

The paper introduces Functional Risk Minimization (FRM), a new framework that compares functions rather than outputs, improving performance across various learning paradigms and offering insights into generalization in over-parameterized models.

Contribution

FRM generalizes ERM by comparing functions instead of outputs, capturing realistic noise and aiding understanding of generalization in modern models.

Findings

FRM improves performance in supervised, unsupervised, and RL tasks.

FRM subsumes ERM for many loss functions.

FRM offers a perspective on model simplicity and generalization in over-parameterized regimes.

Abstract

The field of Machine Learning has changed significantly since the 1970s. However, its most basic principle, Empirical Risk Minimization (ERM), remains unchanged. We propose Functional Risk Minimization~(FRM), a general framework where losses compare functions rather than outputs. This results in better performance in supervised, unsupervised, and RL experiments. In the FRM paradigm, for each data point $(x_i,y_i)$ there is function $f_{\theta_i}$ that fits it: $y_i = f_{\theta_i}(x_i)$. This allows FRM to subsume ERM for many common loss functions and to capture more realistic noise processes. We also show that FRM provides an avenue towards understanding generalization in the modern over-parameterized regime, as its objective can be framed as finding the simplest model that fits the training data.