Theoretical Analysis of Hierarchical Language Recognition and Generation by Transformers without Positional Encoding

TL;DR

This paper proves that Transformers can recognize and generate hierarchical language structures efficiently without positional encoding, using causal masking and starting tokens, and suggests positional encoding may hinder generalization.

Contribution

It provides a constructive proof that Transformers can handle hierarchical language without positional encoding, challenging the necessity of positional information.

Findings

Transformers can recognize hierarchical structures without positional encoding.

Causal masking and starting tokens enable positional awareness.

Positional encoding may negatively impact generalization.

Abstract

In this study, we provide constructive proof that Transformers can recognize and generate hierarchical language efficiently with respect to model size, even without the need for a specific positional encoding. Specifically, we show that causal masking and a starting token enable Transformers to compute positional information and depth within hierarchical structures. We demonstrate that Transformers without positional encoding can generate hierarchical languages. Furthermore, we suggest that explicit positional encoding might have a detrimental effect on generalization with respect to sequence length.