AI, Native Supercomputing and The Revival of Moore's Law

TL;DR

This paper proposes a new AI computing architecture inspired by Turing's ideas, emphasizing native linear algebra processing and collective streaming to overcome Moore's law limitations without traditional GPU or TPU hierarchies.

Contribution

It introduces a universal learning machine architecture that leverages collective streaming and native linear algebra to enhance AI computing efficiency beyond current hardware constraints.

Findings

Proposes a universal learning machine architecture.

Introduces collective streaming for data distribution and collection.

Suggests bypassing traditional GPU/TPU hierarchies.

Abstract

Based on Alan Turing's proposition on AI and computing machinery, which shaped Computing as we know it today, the new AI computing machinery should comprise a universal computer and a universal learning machine. The later should understand linear algebra natively to overcome the slowdown of Moore's law. In such a universal learnig machine, a computing unit does not need to keep the legacy of a universal computing core. The data can be distributed to the computing units, and the results can be collected from them through Collective Streaming, reminiscent of Collective Communication in Supercomputing. It is not necessary to use a GPU-like deep memory hierarchy, nor a TPU-like fine-grain mesh.