A Ridge Too Far: Correcting Over-Shrinkage via Negative Regularization

TL;DR

This paper introduces a negative regularization approach in ridge regression to counteract over-shrinkage, especially effective in small-data settings with weak signal directions.

Contribution

It formalizes weak-spectrum underfitting, derives a sign-switching result, and develops criteria for automatic negative regularization selection.

Findings

Negative regularization increases spectral complexity along weak eigendirections.

Sign-switch behavior is observed under conservative baseline shrinkage.

Experiments verify the feasibility and effectiveness of negative regularization.

Abstract

Conventional regularization is designed to control variance, but in small-data regression it can also aggravate underfitting when predictive signal is concentrated in weak directions of a restricted representation. We study a negative-capable ridge family that permits a feasible negative region whenever the estimator remains well posed, and show that negative regularization acts there as controlled anti-shrinkage by increasing effective complexity most strongly along weak eigendirections. Building on this mechanism, we formalize weak-spectrum underfitting, derive a sign-switch result under conservative baseline shrinkage, and study criterion-based automatic selection over the full negative-capable family. Synthetic and semi-synthetic experiments support the theory by verifying feasibility, spectral complexity increase, sign-switch behavior, and effective recovery of negative adjustments in the predicted regimes.