Support Vector Regression: Risk Quadrangle Framework

TL;DR

This paper unifies Support Vector Regression within the Risk Quadrangle framework, revealing new theoretical insights, including asymptotic unbiasedness and distributional robustness, supported by a case study.

Contribution

It introduces a unified RQ-based framework for SVR, connecting it to risk measures and distributionally robust optimization, and demonstrates equivalences and new formulations.

Findings

$ ext{ε}$-SVR and $ ext{ν}$-SVR minimize Vapnik error and CVaR norm

Both SVR variants are asymptotically unbiased estimators of symmetric quantiles

SVR can be formulated as a deviation minimization and distributionally robust problem

Abstract

This paper investigates Support Vector Regression (SVR) within the framework of the Risk Quadrangle (RQ) theory. Every RQ includes four stochastic functionals -- error, regret, risk, and \emph{deviation}, bound together by a so-called statistic. The RQ framework unifies stochastic optimization, risk management, and statistical estimation. Within this framework, both $\varepsilon$-SVR and $\nu$-SVR are shown to reduce to the minimization of the \emph{Vapnik error} and the Conditional Value-at-Risk (CVaR) norm, respectively. The Vapnik error and CVaR norm define quadrangles with a statistic equal to the average of two symmetric quantiles. Therefore, RQ theory implies that $\varepsilon$-SVR and $\nu$-SVR are asymptotically unbiased estimators of the average of two symmetric conditional quantiles. Moreover, the equivalence between $\varepsilon$-SVR and $\nu$-SVR is demonstrated in a general stochastic setting. Additionally, SVR is formulated as a deviation minimization problem. Another implication of the RQ theory is the formulation of $\nu$-SVR as a Distributionally Robust Regression (DRR) problem. Finally, an alternative dual formulation of SVR within the RQ framework is derived. Theoretical results are validated with a case study.