An Intrinsic Treatment of Stochastic Linear Regression

TL;DR

This paper provides a rigorous, intrinsic mathematical framework for stochastic linear regression, clarifying its fundamental concepts and correcting misconceptions, with illustrative examples and a systematic arrangement of known results.

Contribution

It offers the first systematic, intrinsic treatment of stochastic linear regression, including new results and a clear classification of fundamental concepts.

Findings

Provides a rigorous mathematical framework for stochastic linear regression.

Clarifies misconceptions and conceptual issues in the field.

Arranges fundamental known results systematically for better understanding.

Abstract

Linear regression is perhaps one of the most popular statistical concepts, which permeates almost every scientific field of study. Due to the technical simplicity and wide applicability of linear regression, attention is almost always quickly directed to the algorithmic or computational side of linear regression. In particular, the underlying mathematics of stochastic linear regression itself as an entity usually gets either a peripheral treatment or a relatively in-depth but ad hoc treatment depending on the type of concerned problems; in other words, compared to the extensiveness of the study of mathematical properties of the "derivatives" of stochastic linear regression such as the least squares estimator, the mathematics of stochastic linear regression itself seems to have not yet received a due intrinsic treatment. Apart from the conceptual importance, a consequence of an insufficient or possibly inaccurate understanding of stochastic linear regression would be the recurrence for the role of stochastic linear regression in the important (and more sophisticated) context of structural equation modeling to be misperceived or taught in a misleading way. We believe this pity is rectifiable when the fundamental concepts are correctly classified. Accompanied by some illustrative, distinguishing examples and counterexamples, we intend to pave out the mathematical framework for stochastic linear regression, in a rigorous but non-technical way, by giving new results and pasting together several fundamental known results that are, we believe, both enlightening and conceptually useful, and that had not yet been systematically documented in the related literature. As a minor contribution, the way we arrange the fundamental known results would be the first attempt in the related literature.