Orthogonal and Linear Regressions and Pencils of Confocal Quadrics

TL;DR

This paper introduces a geometric framework using confocal quadrics to improve regression methods, providing new regularization techniques and insights into error-in-variables models with applications in statistics.

Contribution

It develops a novel geometric approach linking confocal quadrics with regression analysis, generalizing classical results and introducing new regularization procedures.

Findings

Explicit construction of confocal quadrics related to hyperplanar moments of inertia.

New regularization methods for orthogonal least squares, lasso, and ridge regression.

Geometric characterization of least squares hyperplanes and applications to restricted regressions.

Abstract

This paper enhances and develops bridges between statistics, mechanics, and geometry. For a given system of points in $\mathbb R^k$ representing a sample of full rank, we construct an explicit pencil of confocal quadrics with the following properties: (i) All the hyperplanes for which the hyperplanar moments of inertia for the given system of points are equal, are tangent to the same quadrics from the pencil of quadrics. As an application, we develop regularization procedures for the orthogonal least square method, analogues of lasso and ridge methods from linear regression. (ii) For any given point $P$ among all the hyperplanes that contain it, the best fit is the tangent hyperplane to the quadric from the confocal pencil corresponding to the maximal Jacobi coordinate of the point $P$; the worst fit among the hyperplanes containing $P$ is the tangent hyperplane to the ellipsoid from the confocal pencil that contains $P$. The confocal pencil of quadrics provides a universal tool to solve the restricted principal component analysis restricted at any given point. Both results (i) and (ii) can be seen as generalizations of the classical result of Pearson on orthogonal regression. They have natural and important applications in the statistics of the errors-in-variables models (EIV). For the classical linear regressions we provide a geometric characterization of hyperplanes of least squares in a given direction among all hyperplanes which contain a given point. The obtained results have applications in restricted regressions, both ordinary and orthogonal ones. For the latter, a new formula for test statistic is derived. The developed methods and results are illustrated in natural statistics examples.