Variance-based regularization with convex objectives

TL;DR

This paper introduces a convex surrogate for variance in risk minimization, enabling efficient trade-offs between approximation and estimation errors, with theoretical guarantees and improved out-of-sample performance.

Contribution

It proposes a novel variance-based regularization method using convex objectives, combining distributionally robust optimization and empirical likelihood techniques.

Findings

Achieves faster convergence rates than ERM in some scenarios.

Provides certificates of optimality for the estimator.

Improves out-of-sample performance in classification tasks.

Abstract

We develop an approach to risk minimization and stochastic optimization that provides a convex surrogate for variance, allowing near-optimal and computationally efficient trading between approximation and estimation error. Our approach builds off of techniques for distributionally robust optimization and Owen's empirical likelihood, and we provide a number of finite-sample and asymptotic results characterizing the theoretical performance of the estimator. In particular, we show that our procedure comes with certificates of optimality, achieving (in some scenarios) faster rates of convergence than empirical risk minimization by virtue of automatically balancing bias and variance. We give corroborating empirical evidence showing that in practice, the estimator indeed trades between variance and absolute performance on a training sample, improving out-of-sample (test) performance over standard empirical risk minimization for a number of classification problems.