Deep Learning: Our Miraculous Year 1990-1991

TL;DR

This paper highlights the revolutionary impact of neural network ideas developed during 1990-1991, which laid the groundwork for modern AI advancements like generative models, transformers, and deep recurrent networks, now pervasive worldwide.

Contribution

It documents the foundational principles of key deep learning architectures introduced during 1990-1991, which have become central to contemporary AI systems.

Findings

Foundations of Generative Adversarial Networks, Transformers, and Pre-training established in 1990-91.

LSTM and Highway Networks, inspired by 1991 work, are among the most cited AI papers.

Deep neural networks from this era are now embedded in billions of devices worldwide.

Abstract

The Deep Learning Artificial Neural Networks (NNs) of our team have revolutionised Machine Learning & AI. Many of the basic ideas behind this revolution were published within the 12 months of our "Annus Mirabilis" 1990-1991 at our lab in TU Munich. Back then, few people were interested. But a quarter century later, NNs based on our "Miraculous Year" were on over 3 billion devices, and used many billions of times per day, consuming a significant fraction of the world's compute. In particular, in 1990-91, we laid foundations of Generative AI, publishing principles of (1) Generative Adversarial Networks for Artificial Curiosity and Creativity (now used for deepfakes), (2) Transformers (the T in ChatGPT - see the 1991 Unnormalized Linear Transformer), (3) Pre-training for deep NNs (see the P in ChatGPT), (4) NN distillation (key for DeepSeek), and (5) recurrent World Models for Reinforcement Learning and Planning in partially observable environments. The year 1991 also marks the emergence of the defining features of (6) LSTM, the most cited AI paper of the 20th century (based on deep residual learning and constant error flow through residual NN connections), and (7) the most cited paper of the 21st century, based on our LSTM-inspired Highway Net that was 10 times deeper than previous feedforward NNs. As of 2025, the two most frequently cited scientific articles of all time (with the most Google Scholar citations within 3 years - manuals excluded) are both directly based on our 1991 work.