Second-Order Bounds for [0,1]-Valued Regression via Betting Loss

TL;DR

This paper introduces the betting loss, a new loss function for [0,1]-valued regression, that achieves variance-dependent second-order bounds without prior variance knowledge, improving upon first-order bounds.

Contribution

The paper proposes the betting loss, a novel variance-adaptive loss function that provides second-order bounds in [0,1]-valued regression without needing variance estimation.

Findings

Betting loss achieves variance-dependent bounds.

It is variance-adaptive without prior variance knowledge.

Improves upon first-order bounds in regression.

Abstract

We consider the $[0,1]$-valued regression problem in the i.i.d. setting. In a related problem called cost-sensitive classification, \citet{foster21efficient} have shown that the log loss minimizer achieves an improved generalization bound compared to that of the squared loss minimizer in the sense that the bound scales with the cost of the best classifier, which can be arbitrarily small depending on the problem at hand. Such a result is often called a first-order bound. For $[0,1]$-valued regression, we first show that the log loss minimizer leads to a similar first-order bound. We then ask if there exists a loss function that achieves a variance-dependent bound (also known as a second order bound), which is a strict improvement upon first-order bounds. We answer this question in the affirmative by proposing a novel loss function called the betting loss. Our result is ``variance-adaptive'' in the sense that the bound is attained \textit{without any knowledge about the variance}, which is in contrast to modeling label (or reward) variance or the label distribution itself explicitly as part of the function class such as distributional reinforcement learning.