Continuity and Isolation Lead to Doubts or Dilemmas in Large Language Models

TL;DR

This paper reveals two fundamental phenomena, isolation and continuity, that limit Transformers' ability to learn simple patterns, supported by mathematical proofs and practical experiments.

Contribution

It introduces the phenomena of isolation and continuity in Transformers with compact positional encoding, providing mathematical proofs and empirical evidence of their limitations.

Findings

Isolation prevents learning multiple sequences simultaneously.

Continuity causes sequences to collapse into attractor basins.

Limitations occur in practical Transformer models.

Abstract

Understanding how Transformers work and how they process information is key to the theoretical and empirical advancement of these machines. In this work, we demonstrate the existence of two phenomena in Transformers, namely isolation and continuity. Both of these phenomena hinder Transformers to learn even simple pattern sequences. Isolation expresses that any learnable sequence must be isolated from another learnable sequence, and hence some sequences cannot be learned by a single Transformer at the same time. Continuity entails that an attractor basin forms around a learned sequence, such that any sequence falling in that basin will collapse towards the learned sequence. Here, we mathematically prove these phenomena emerge in all Transformers that use compact positional encoding, and design rigorous experiments, demonstrating that the theoretical limitations we shed light on occur on the practical scale.