Hebbian learning inspired estimation of the linear regression parameters from queries

TL;DR

This paper investigates a Hebbian learning-inspired approach for estimating linear regression parameters using query-based methods, demonstrating near-optimal convergence rates and advantages over non-adaptive strategies.

Contribution

It introduces a variation of Hebbian learning for linear regression, establishing bounds and showing faster convergence than non-adaptive query methods.

Findings

Achieves near-optimal convergence rates with query-based methods.

Hebbian learning variation outperforms non-adaptive query strategies.

Provides theoretical bounds for the proposed approach.

Abstract

Local learning rules in biological neural networks (BNNs) are commonly referred to as Hebbian learning. [26] links a biologically motivated Hebbian learning rule to a specific zeroth-order optimization method. In this work, we study a variation of this Hebbian learning rule to recover the regression vector in the linear regression model. Zeroth-order optimization methods are known to converge with suboptimal rate for large parameter dimension compared to first-order methods like gradient descent, and are therefore thought to be in general inferior. By establishing upper and lower bounds, we show, however, that such methods achieve near-optimal rates if only queries of the linear regression loss are available. Moreover, we prove that this Hebbian learning rule can achieve considerably faster rates than any non-adaptive method that selects the queries independently of the data.