Learning Spectral Methods by Transformers

TL;DR

This paper demonstrates that multi-layered Transformers can learn spectral algorithms like PCA and clustering through unsupervised pre-training, showing theoretical and empirical capabilities beyond in-context learning.

Contribution

It provides a theoretical proof that Transformers can learn spectral methods and empirically verifies this on PCA and clustering tasks.

Findings

Transformers can learn spectral algorithms like PCA and clustering.

Pre-trained Transformers perform well on synthetic and real datasets.

Theoretical proof of spectral method learning by Transformers.

Abstract

Transformers demonstrate significant advantages as the building block of modern LLMs. In this work, we study the capacities of Transformers in performing unsupervised learning. We show that multi-layered Transformers, given a sufficiently large set of pre-training instances, are able to learn the algorithms themselves and perform statistical estimation tasks given new instances. This learning paradigm is distinct from the in-context learning setup and is similar to the learning procedure of human brains where skills are learned through past experience. Theoretically, we prove that pre-trained Transformers can learn the spectral methods and use the classification of bi-class Gaussian mixture model as an example. Our proof is constructive using algorithmic design techniques. Our results are built upon the similarities of multi-layered Transformer architecture with the iterative recovery algorithms used in practice. Empirically, we verify the strong capacity of the multi-layered (pre-trained) Transformer on unsupervised learning through the lens of both the PCA and the Clustering tasks performed on the synthetic and real-world datasets.