When Does More Regularization Imply Fewer Degrees of Freedom? Sufficient Conditions and Counter Examples from Lasso and Ridge Regression

TL;DR

This paper investigates when regularization reduces degrees of freedom, providing conditions under which it does and counterexamples where it does not, especially in lasso and ridge regression.

Contribution

It establishes sufficient conditions for regularization to decrease degrees of freedom and presents counterexamples where it fails, clarifying the behavior of common regularization methods.

Findings

Regularization can increase degrees of freedom in simple models.

Certain regularization scenarios guarantee a reduction in degrees of freedom.

Counterexamples show regularization may not always improve model simplicity.

Abstract

Regularization aims to improve prediction performance of a given statistical modeling approach by moving to a second approach which achieves worse training error but is expected to have fewer degrees of freedom, i.e., better agreement between training and prediction error. We show here, however, that this expected behavior does not hold in general. In fact, counter examples are given that show regularization can increase the degrees of freedom in simple situations, including lasso and ridge regression, which are the most common regularization approaches in use. In such situations, the regularization increases both training error and degrees of freedom, and is thus inherently without merit. On the other hand, two important regularization scenarios are described where the expected reduction in degrees of freedom is indeed guaranteed: (a) all symmetric linear smoothers, and (b) linear regression versus convex constrained linear regression (as in the constrained variant of ridge regression and lasso).