RNNs are not Transformers (Yet): The Key Bottleneck on In-context Retrieval

TL;DR

This paper analyzes the limitations of RNNs compared to Transformers in solving algorithmic problems, showing that RNNs lack the retrieval capacity needed for certain tasks but can be enhanced to match Transformers' capabilities.

Contribution

The paper provides a theoretical analysis of RNNs' limitations and demonstrates how retrieval techniques and minimal modifications can bridge the gap with Transformers.

Findings

RNNs cannot solve tasks requiring perfect context retrieval.

Transformers can solve associative recall and graph problems easily.

Enhancing RNNs with retrieval methods closes the performance gap.

Abstract

This paper investigates the gap in representation powers of Recurrent Neural Networks (RNNs) and Transformers in the context of solving algorithmic problems. We focus on understanding whether RNNs, known for their memory efficiency in handling long sequences, can match the performance of Transformers, particularly when enhanced with Chain-of-Thought (CoT) prompting. Our theoretical analysis reveals that CoT improves RNNs but is insufficient to close the gap with Transformers. A key bottleneck lies in the inability of RNNs to perfectly retrieve information from the context, even with CoT: for several tasks that explicitly or implicitly require this capability, such as associative recall and determining if a graph is a tree, we prove that RNNs are not expressive enough to solve the tasks while Transformers can solve them with ease. Conversely, we prove that adopting techniques to enhance the in-context retrieval capability of RNNs, including Retrieval-Augmented Generation (RAG) and adding a single Transformer layer, can elevate RNNs to be capable of solving all polynomial-time solvable problems with CoT, hence closing the representation gap with Transformers.