On the benefits of maximum likelihood estimation for Regression and Forecasting

TL;DR

This paper advocates for using maximum likelihood estimation (MLE) as a flexible and theoretically sound approach for designing loss functions in regression and forecasting, offering advantages over direct metric minimization.

Contribution

It introduces a practical MLE-based framework for regression and forecasting that incorporates domain knowledge and can optimize various target metrics, with theoretical and empirical validation.

Findings

MLE-based methods outperform direct metric minimization in excess risk bounds.

The approach achieves superior empirical performance across diverse datasets.

Theoretical results demonstrate competitiveness with any estimator for the target metric.

Abstract

We advocate for a practical Maximum Likelihood Estimation (MLE) approach towards designing loss functions for regression and forecasting, as an alternative to the typical approach of direct empirical risk minimization on a specific target metric. The MLE approach is better suited to capture inductive biases such as prior domain knowledge in datasets, and can output post-hoc estimators at inference time that can optimize different types of target metrics. We present theoretical results to demonstrate that our approach is competitive with any estimator for the target metric under some general conditions. In two example practical settings, Poisson and Pareto regression, we show that our competitive results can be used to prove that the MLE approach has better excess risk bounds than directly minimizing the target metric. We also demonstrate empirically that our method instantiated with a well-designed general purpose mixture likelihood family can obtain superior performance for a variety of tasks across time-series forecasting and regression datasets with different data distributions.