Towards Practical Control of Singular Values of Convolutional Layers

TL;DR

This paper introduces a tensor-train decomposition-based method to control singular values of convolutional layers, improving CNN properties like robustness and calibration with minimal expressivity loss.

Contribution

It presents a novel, computationally feasible approach to regulate singular values in CNNs using tensor-train decomposition, enhancing model robustness and efficiency.

Findings

Improved adversarial robustness of CNNs

Enhanced calibration and generalization performance

Maintained model expressivity with minimal reduction

Abstract

In general, convolutional neural networks (CNNs) are easy to train, but their essential properties, such as generalization error and adversarial robustness, are hard to control. Recent research demonstrated that singular values of convolutional layers significantly affect such elusive properties and offered several methods for controlling them. Nevertheless, these methods present an intractable computational challenge or resort to coarse approximations. In this paper, we offer a principled approach to alleviating constraints of the prior art at the expense of an insignificant reduction in layer expressivity. Our method is based on the tensor-train decomposition; it retains control over the actual singular values of convolutional mappings while providing structurally sparse and hardware-friendly representation. We demonstrate the improved properties of modern CNNs with our method and analyze its impact on the model performance, calibration, and adversarial robustness. The source code is available at: https://github.com/WhiteTeaDragon/practical_svd_conv