Robust Regression with Student's T: The Role of Degrees of Freedom

TL;DR

This paper investigates robust linear regression using Student's t distribution, comparing methods for estimating degrees of freedom, and demonstrates that accurate estimation improves regression performance through extensive simulations.

Contribution

It introduces a method for estimating degrees of freedom in Student's t regression using adjusted profile likelihood, showing its effectiveness over fixed degrees of freedom approaches.

Findings

Estimating degrees of freedom improves regression accuracy.

The proposed method performs comparably to maximum likelihood with known true values.

Proper calibration of degrees of freedom is crucial for robust regression.

Abstract

Linear regression estimators are known to be sensitive to outliers, and one alternative to obtain a robust and efficient estimator of the regression parameter is to model the error with Student's $t$ distribution. In this article, we compare estimators of the degrees of freedom parameter in the $t$ distribution using frequentist and Bayesian methods, and then study properties of the corresponding estimated regression coefficient. We also include the comparison with some recommended approaches in the literature, including fixing the degrees of freedom and robust regression using the Huber loss. Our extensive simulations on both synthetic and real data demonstrate that estimating the degrees of freedom via the adjusted profile log-likelihood approach yields regression coefficient estimators with high accuracy, performing comparably to the maximum likelihood estimators where the degrees of freedom are fixed at their true values. These findings provide a detailed synthesis of $t$-based robust regression and underscore a key insight: the proper calibration of the degrees of freedom is as crucial as the choice of the robust distribution itself for achieving optimal performance. The {\tt R} package that implements our method is available at https://github.com/amanda-ng518/RobustTRegression.