On the alpha-loss Landscape in the Logistic Model

TL;DR

This paper investigates the optimization landscape of the alpha-loss family in logistic models, revealing how landscape properties change with alpha and implications for optimization complexity.

Contribution

It provides a detailed analysis of the alpha-loss landscape across different alpha values, connecting geometric properties to optimization difficulty.

Findings

Landscape properties vary with alpha, affecting optimization complexity.

Alpha-loss includes exponential, log, and 0-1 losses as special cases.

Results inform the choice of loss functions for better learning performance.

Abstract

We analyze the optimization landscape of a recently introduced tunable class of loss functions called $\alpha$-loss, $\alpha \in (0,\infty]$, in the logistic model. This family encapsulates the exponential loss ($\alpha = 1/2$), the log-loss ($\alpha = 1$), and the 0-1 loss ($\alpha = \infty$) and contains compelling properties that enable the practitioner to discern among a host of operating conditions relevant to emerging learning methods. Specifically, we study the evolution of the optimization landscape of $\alpha$-loss with respect to $\alpha$ using tools drawn from the study of strictly-locally-quasi-convex functions in addition to geometric techniques. We interpret these results in terms of optimization complexity via normalized gradient descent.