Transformed Low-Rank Parameterization Can Help Robust Generalization for Tensor Neural Networks

TL;DR

This paper investigates how transformed low-rank parameterization influences the generalization ability of tensor neural networks, providing theoretical bounds and practical insights into robustness and adversarial training effects.

Contribution

It is the first to derive generalization bounds for t-NNs with transformed low-rank weights and explains how adversarial training promotes low-rank regularization.

Findings

Transformed low-rank parameterization leads to sharper adversarial generalization bounds.

Adversarial training with gradient flow implicitly regularizes t-NNs towards low-rank weights.

The analysis applies to both exact and approximate transformed low-rank weights.

Abstract

Achieving efficient and robust multi-channel data learning is a challenging task in data science. By exploiting low-rankness in the transformed domain, i.e., transformed low-rankness, tensor Singular Value Decomposition (t-SVD) has achieved extensive success in multi-channel data representation and has recently been extended to function representation such as Neural Networks with t-product layers (t-NNs). However, it still remains unclear how t-SVD theoretically affects the learning behavior of t-NNs. This paper is the first to answer this question by deriving the upper bounds of the generalization error of both standard and adversarially trained t-NNs. It reveals that the t-NNs compressed by exact transformed low-rank parameterization can achieve a sharper adversarial generalization bound. In practice, although t-NNs rarely have exactly transformed low-rank weights, our analysis further shows that by adversarial training with gradient flow (GF), the over-parameterized t-NNs with ReLU activations are trained with implicit regularization towards transformed low-rank parameterization under certain conditions. We also establish adversarial generalization bounds for t-NNs with approximately transformed low-rank weights. Our analysis indicates that the transformed low-rank parameterization can promisingly enhance robust generalization for t-NNs.