Statistical Mechanics of Support Vector Regression

TL;DR

This paper applies statistical mechanics to analyze Support Vector Regression, revealing phase transitions and double-descent phenomena in generalization error, and extends SVM theory to neural variability in continuous decoding tasks.

Contribution

It introduces a statistical mechanics framework for understanding SVR in neural contexts, highlighting phase transitions and the regularizing role of the tolerance parameter.

Findings

Identifies a phase transition in training error at a critical load.

Discovers a double-descent phenomenon in generalization error.

Shows the regularizing effect of the epsilon parameter.

Abstract

A key problem in deep learning and computational neuroscience is relating the geometrical properties of neural representations to task performance. Here, we consider this problem for continuous decoding tasks where neural variability may affect task precision. Using methods from statistical mechanics, we study the average-case learning curves for $\varepsilon$-insensitive Support Vector Regression ($\varepsilon$-SVR) and discuss its capacity as a measure of linear decodability. Our analysis reveals a phase transition in training error at a critical load, capturing the interplay between the tolerance parameter $\varepsilon$ and neural variability. We uncover a double-descent phenomenon in the generalization error, showing that $\varepsilon$ acts as a regularizer, both suppressing and shifting these peaks. Theoretical predictions are validated both with toy models and deep neural networks, extending the theory of Support Vector Machines to continuous tasks with inherent neural variability.