The Equivariance Criterion in a Linear Model for Fixed-X Cases

TL;DR

This paper investigates the use of the equivariance criterion in fixed-X linear models, deriving optimal estimators and confirming the approach's theoretical validity within a statistical estimation framework.

Contribution

It extends the linear model to multiple populations, derives minimum risk equivariant estimators, and provides theoretical insights into the optimality of least squares in machine learning.

Findings

Derived minimum risk equivariant estimators for coefficients and covariance.

Confirmed the optimality of least squares in the fixed-X linear model setting.

Linked the estimation problem to mixtures of normal samples and functional equivariance.

Abstract

The field of machine have seen rising applications of equivariance criterion. However, there is no systematic way to justify its usage, including why it works, whether there is an optimal solution and if so, what form it carries. In this article, we explored the usage of equivariance criterion in a normal linear model with fixed-$X$ and extended the model to allow multiple populations, which, in turn, leads to a multivariate invariant location-scale transformation group, compared than the commonly used univariate one. The minimum risk equivariant estimators of the coefficient vector and the diagonal covariance matrix were derived, which were consistent with literature works. This work serves as an early exploration of the usage of equivariance criterion in machine learning, where we confirmed that the least square approach widely used in machine learning indeed carries optimality in some sense at least in the framework of estimation. Meanwhile, the problems can be shown to be equivalent to a mixture from $p$ independent normal samples and via the principle of functional equivariance, an alternative proof can be derived. However, such an approach carries its own limitation with a strong tie to equivariance criterion.