Orthogonium : A Unified, Efficient Library of Orthogonal and 1-Lipschitz Building Blocks

TL;DR

Orthogonium is a comprehensive, efficient PyTorch library that provides orthogonal and 1-Lipschitz neural network layers, facilitating robust deep learning applications with mathematical guarantees and reduced computational overhead.

Contribution

The paper introduces Orthogonium, a unified library offering optimized, reliable orthogonal and 1-Lipschitz layers with support for standard convolution features, addressing fragmentation and inefficiency in existing tools.

Findings

Uncovered critical errors in existing implementations.

Enabled scalable experimentation on large benchmarks like ImageNet.

Provided standardized, reliable tools for robust deep learning.

Abstract

Orthogonal and 1-Lipschitz neural network layers are essential building blocks in robust deep learning architectures, crucial for certified adversarial robustness, stable generative models, and reliable recurrent networks. Despite significant advancements, existing implementations remain fragmented, limited, and computationally demanding. To address these issues, we introduce Orthogonium , a unified, efficient, and comprehensive PyTorch library providing orthogonal and 1-Lipschitz layers. Orthogonium provides access to standard convolution features-including support for strides, dilation, grouping, and transposed-while maintaining strict mathematical guarantees. Its optimized implementations reduce overhead on large scale benchmarks such as ImageNet. Moreover, rigorous testing within the library has uncovered critical errors in existing implementations, emphasizing the importance of standardized and reliable tools. Orthogonium thus significantly lowers adoption barriers, enabling scalable experimentation and integration across diverse applications requiring orthogonality and robust Lipschitz constraints. Orthogonium is available at https://github.com/deel-ai/orthogonium.