From Complexity to Clarity: Analytical Expressions of Deep Neural Network Weights via Clifford's Geometric Algebra and Convexity

TL;DR

This paper presents a geometric algebra framework for analyzing deep neural networks, revealing that optimal weights relate to wedge products of data and that training reduces to convex optimization over these geometric features.

Contribution

It introduces a novel geometric algebra approach to neural network analysis, connecting weights to wedge products and convex optimization over geometric data structures.

Findings

Optimal weights are wedge products of training samples.

Training reduces to convex optimization over wedge product features.

Identifies relevant geometric features via $\,l_1$ regularization.

Abstract

In this paper, we introduce a novel analysis of neural networks based on geometric (Clifford) algebra and convex optimization. We show that optimal weights of deep ReLU neural networks are given by the wedge product of training samples when trained with standard regularized loss. Furthermore, the training problem reduces to convex optimization over wedge product features, which encode the geometric structure of the training dataset. This structure is given in terms of signed volumes of triangles and parallelotopes generated by data vectors. The convex problem finds a small subset of samples via $\ell_1$ regularization to discover only relevant wedge product features. Our analysis provides a novel perspective on the inner workings of deep neural networks and sheds light on the role of the hidden layers.