Empirical Risk Minimization for Losses without Variance

TL;DR

This paper develops a robust empirical risk minimization approach for heavy-tailed data with infinite variance, utilizing Catoni's method to achieve better performance than traditional truncation-based methods.

Contribution

It introduces a novel empirical risk minimization framework under heavy-tailed distributions using Catoni's robust estimation, with theoretical guarantees and practical algorithms.

Findings

Catoni's method provides reliable risk estimates in heavy-tailed settings.

The proposed approach outperforms baseline methods in numerical experiments.

Theoretical bounds on excess risk are established using generalized chaining techniques.

Abstract

This paper considers an empirical risk minimization problem under heavy-tailed settings, where data does not have finite variance, but only has $p$-th moment with $p \in (1,2)$. Instead of using estimation procedure based on truncated observed data, we choose the optimizer by minimizing the risk value. Those risk values can be robustly estimated via using the remarkable Catoni's method (Catoni, 2012). Thanks to the structure of Catoni-type influence functions, we are able to establish excess risk upper bounds via using generalized generic chaining methods. Moreover, we take computational issues into consideration. We especially theoretically investigate two types of optimization methods, robust gradient descent algorithm and empirical risk-based methods. With an extensive numerical study, we find that the optimizer based on empirical risks via Catoni-style estimation indeed shows better performance than other baselines. It indicates that estimation directly based on truncated data may lead to unsatisfactory results.