Towards a Categorical Foundation of Deep Learning: A Survey

TL;DR

This survey explores how category theory can provide a unifying and principled mathematical framework for deep learning, addressing current theoretical gaps and reproducibility issues.

Contribution

It reviews recent work applying category theory to deep learning, proposing a structured mathematical foundation for the field.

Findings

Categorical optics model gradient-based learning

Categorical algebras link classical CS to neural networks

String diagrams visually represent neural architectures

Abstract

The unprecedented pace of machine learning research has lead to incredible advances, but also poses hard challenges. At present, the field lacks strong theoretical underpinnings, and many important achievements stem from ad hoc design choices which are hard to justify in principle and whose effectiveness often goes unexplained. Research debt is increasing and many papers are found not to be reproducible. This thesis is a survey that covers some recent work attempting to study machine learning categorically. Category theory is a branch of abstract mathematics that has found successful applications in many fields, both inside and outside mathematics. Acting as a lingua franca of mathematics and science, category theory might be able to give a unifying structure to the field of machine learning. This could solve some of the aforementioned problems. In this work, we mainly focus on the application of category theory to deep learning. Namely, we discuss the use of categorical optics to model gradient-based learning, the use of categorical algebras and integral transforms to link classical computer science to neural networks, the use of functors to link different layers of abstraction and preserve structure, and, finally, the use of string diagrams to provide detailed representations of neural network architectures.