LipNeXt: Scaling up Lipschitz-based Certified Robustness to Billion-parameter Models

TL;DR

LipNeXt introduces a scalable, constraint-free, convolution-free 1-Lipschitz architecture that achieves state-of-the-art certified robustness on large models and datasets, demonstrating the potential of Lipschitz-based certification for modern deep learning.

Contribution

The paper presents LipNeXt, a novel 1-Lipschitz architecture that scales to billion-parameter models using manifold optimization and spatial shift modules, without convolutions or constraints.

Findings

Achieves state-of-the-art certified robustness on CIFAR-10/100 and Tiny-ImageNet.

Scales to 1-2 billion parameters on ImageNet, improving robustness over prior Lipschitz models.

Maintains efficient, stable low-precision training while providing deterministic robustness guarantees.

Abstract

Lipschitz-based certification offers efficient, deterministic robustness guarantees but has struggled to scale in model size, training efficiency, and ImageNet performance. We introduce \emph{LipNeXt}, the first \emph{constraint-free} and \emph{convolution-free} 1-Lipschitz architecture for certified robustness. LipNeXt is built using two techniques: (1) a manifold optimization procedure that updates parameters directly on the orthogonal manifold and (2) a \emph{Spatial Shift Module} to model spatial pattern without convolutions. The full network uses orthogonal projections, spatial shifts, a simple 1-Lipschitz $\beta$-Abs nonlinearity, and $L_2$ spatial pooling to maintain tight Lipschitz control while enabling expressive feature mixing. Across CIFAR-10/100 and Tiny-ImageNet, LipNeXt achieves state-of-the-art clean and certified robust accuracy (CRA), and on ImageNet it scales to 1-2B large models, improving CRA over prior Lipschitz models (e.g., up to $+8\%$ at $\varepsilon{=}1$) while retaining efficient, stable low-precision training. These results demonstrate that Lipschitz-based certification can benefit from modern scaling trends without sacrificing determinism or efficiency.