The General Theory of Localization Methods

TL;DR

This paper introduces the localization method, a unifying machine learning framework based on localization kernels and local means, connecting various models and extending to modern architectures like Transformers.

Contribution

It provides a rigorous theoretical foundation for the localization method, linking it to many existing models and demonstrating its capacity to unify and extend current machine learning architectures.

Findings

Shows the connection between localization method and kernel methods, autoencoders, and Hopfield networks.

Demonstrates that Transformers can be constructed using hierarchical local models.

Provides a theoretical foundation for designing flexible, data-adaptive learning systems.

Abstract

This paper proposes a general machine learning framework called the localization method, which is fundamentally built on two core concepts: localization kernels and local means -- key components that underpin the self-attention mechanism. To establish a rigorous theoretical foundation, the framework is formally defined through two essential pillars: the formulation of the local(-ized) model and the localization trick. We systematically investigate the connections between the localization method and a wide range of existing machine learning models/methods, including (but not limited to) kernel methods, lazy learning, the MeanShift algorithm, relaxation labeling, Hopfield networks, local linear embedding (LLE), fuzzy inference, and denoising autoencoders (DAEs). By dissecting these relationships, we clarify the broader theoretical significance of the localization method and demonstrate its practical applicability across diverse machine learning tasks. Furthermore, we explore advanced extensions of the framework, such as adaptive kernels, hierarchical local models, and non-local models. Notably, we show that the Transformer -- a cornerstone of modern sequence modeling -- can be constructed using hierarchical local models, revealing the ability of the localization method to unify and generalize state-of-the-art architectures. This work not only provides a unified theoretical lens to reinterpret existing models but also offers new methodological tools for designing flexible, data-adaptive learning systems.