Dynamical Low-Rank Compression of Neural Networks with Robustness under Adversarial Attacks

TL;DR

This paper presents a dynamical low-rank training method with spectral regularization that produces compact neural networks resilient to adversarial attacks, balancing compression and robustness effectively.

Contribution

It introduces a novel spectral regularizer within a dynamical low-rank training scheme to enhance adversarial robustness in compressed neural networks.

Findings

Achieves over 94% compression on various architectures.

Improves adversarial accuracy compared to uncompressed models.

Supports automatic rank adaptivity during training.

Abstract

Deployment of neural networks on resource-constrained devices demands models that are both compact and robust to adversarial inputs. However, compression and adversarial robustness often conflict. In this work, we introduce a dynamical low-rank training scheme enhanced with a novel spectral regularizer that controls the condition number of the low-rank core in each layer. This approach mitigates the sensitivity of compressed models to adversarial perturbations without sacrificing accuracy on clean data. The method is model- and data-agnostic, computationally efficient, and supports rank adaptivity to automatically compress the network at hand. Extensive experiments across standard architectures, datasets, and adversarial attacks show the regularized networks can achieve over 94% compression while recovering or improving adversarial accuracy relative to uncompressed baselines.