Maximum Entropy Least Squares Solutions of Overdetermined Linear Systems

TL;DR

This paper develops a theoretical framework for an entropy-based weighted least squares method that enhances robustness against outliers in overdetermined linear systems by controlling the MSE and maximizing Shannon entropy.

Contribution

It introduces a novel entropy-maximizing formulation for weighted least squares, analyzes its optimality conditions, and demonstrates its robustness and unique solution properties.

Findings

Existence and local uniqueness of entropy-maximizing solutions

Global continuation of solutions under nondegeneracy conditions

Asymptotic behavior concentrates on consistent data subsets

Abstract

We investigate the theoretical foundations of a recently introduced entropy-based formulation of weighted least squares for the approximation of overdetermined linear systems, motivated by robust data fitting in the presence of sparse gross errors. The weight vector is interpreted as a discrete probability distribution and is determined by maximizing Shannon entropy under normalization and a prescribed mean squared error (MSE) constraint. Unlike classical ordinary least squares, where the error level is an output of the minimization process, here the MSE value plays the role of a control parameter, and entropy selects the least biased weight distribution achieving the prescribed accuracy. The resulting optimization problem is nonconvex due to the nonlinear coupling between the weights and the solution induced by the residual constraint. We analyze the associated optimality system and characterize stationary points through first- and second-order conditions. We prove the existence and local uniqueness of a smooth branch of entropy-maximizing configurations emanating from the ordinary least squares solution and establish its global continuation under suitable nondegeneracy conditions. Furthermore, we investigate the asymptotic regime as the prescribed MSE tends to zero and show that, under appropriate assumptions, the limiting configuration concentrates on a largest subset of data consistent with the linear model, thus suppressing the influence of outliers. Two numerical experiments illustrate the theoretical findings and confirm the robustness properties of the method.