# Tensor Dropout for Robust Learning

## TL;DR

This paper introduces tensor dropout, a method that enhances robustness and efficiency of CNNs by imposing low-rank tensor structures, demonstrating superior performance on image classification and brain MRI datasets.

## Contribution

The paper proposes tensor dropout with low-rank tensor structures, providing a novel approach for robust, compact, and efficient neural networks with theoretical validation.

## Key findings

- Outperforms existing methods on ImageNet and CIFAR-100
- Achieves state-of-the-art accuracy on UK Biobank brain MRI dataset
- Significantly improves robustness to noise and adversarial attacks

## Abstract

CNNs achieve remarkable performance by leveraging deep, over-parametrized architectures, trained on large datasets. However, they have limited generalization ability to data outside the training domain, and a lack of robustness to noise and adversarial attacks. By building better inductive biases, we can improve robustness and also obtain smaller networks that are more memory and computationally efficient. While standard CNNs use matrix computations, we study tensor layers that involve higher-order computations and provide better inductive bias. Specifically, we impose low-rank tensor structures on the weights of tensor regression layers to obtain compact networks, and propose tensor dropout, a randomization in the tensor rank for robustness. We show that our approach outperforms other methods for large-scale image classification on ImageNet and CIFAR-100. We establish a new state-of-the-art accuracy for phenotypic trait prediction on the largest dataset of brain MRI, the UK Biobank brain MRI dataset, where multi-linear structure is paramount. In all cases, we demonstrate superior performance and significantly improved robustness, both to noisy inputs and to adversarial attacks. We rigorously validate the theoretical validity of our approach by establishing the link between our randomized decomposition and non-linear dropout.