Simulating Hard Attention Using Soft Attention

TL;DR

This paper explores how soft attention mechanisms in transformers can emulate hard attention by using techniques like unbounded positional embeddings and temperature scaling, enabling focused attention on specific input positions.

Contribution

It introduces methods for soft attention transformers to simulate hard attention, including the use of unbounded positional embeddings and temperature-dependent scaling.

Findings

Soft attention can simulate hard attention with unbounded positional embeddings.

Temperature scaling enables softmax transformers to mimic hard-attention behavior.

Transformers can recognize languages defined by linear temporal logic using these techniques.

Abstract

We study conditions under which transformers using soft attention can simulate hard attention, that is, effectively focus all attention on a subset of positions. First, we examine several subclasses of languages recognized by hard-attention transformers, which can be defined in variants of linear temporal logic. We demonstrate how soft-attention transformers can compute formulas of these logics using unbounded positional embeddings or temperature scaling. Second, we demonstrate how temperature scaling allows softmax transformers to simulate general hard-attention transformers, using a temperature that depends on the minimum gap between the maximum attention scores and other attention scores.