A Rescaling-Invariant Lipschitz Bound Based on Path-Metrics for Modern ReLU Network Parameterizations

TL;DR

This paper introduces a rescaling-invariant Lipschitz bound based on path-metrics that applies to various ReLU network architectures, improving robustness guarantees and enabling symmetry-aware pruning.

Contribution

It presents a novel Lipschitz inequality using the $ ext{ell}^1$-path-metric that is invariant to rescaling and applicable to complex ReLU-DAG architectures, sharpening prior bounds.

Findings

The new bound is rescaling-invariant and sharpens previous bounds.

It can be computed efficiently in two forward passes.

The pruning criterion based on this bound performs comparably to classical methods, immune to neuron-wise rescaling.

Abstract

Robustness with respect to weight perturbations underpins guarantees for generalization, pruning and quantization. Existing guarantees rely on Lipschitz bounds in parameter space, cover only plain feed-forward MLPs, and break under the ubiquitous neuron-wise rescaling symmetry of ReLU networks. We prove a new Lipschitz inequality expressed through the $\ell^1$-path-metric of the weights. The bound is (i) rescaling-invariant by construction and (ii) applies to any ReLU-DAG architecture with any combination of convolutions, skip connections, pooling, and frozen (inference-time) batch-normalization -- thus encompassing ResNets, U-Nets, VGG-style CNNs, and more. By respecting the network's natural symmetries, the new bound strictly sharpens prior parameter-space bounds and can be computed in two forward passes. To illustrate its utility, we derive from it a symmetry-aware pruning criterion and show -- through a proof-of-concept experiment on a ResNet-18 trained on ImageNet -- that its pruning performance matches that of classical magnitude pruning, while becoming totally immune to arbitrary neuron-wise rescalings.