Building Compact and Robust Deep Neural Networks with Toeplitz Matrices

TL;DR

This paper proposes using Toeplitz matrices to design neural networks that are compact, cost-effective, and robust against adversarial attacks, addressing limitations of large models in real-world applications.

Contribution

It introduces a novel approach leveraging Toeplitz matrices to create neural networks that are both efficient and robust, improving over traditional unstructured models.

Findings

Neural networks with Toeplitz matrices are more compact and easier to train.

Such networks demonstrate increased robustness to adversarial examples.

The approach maintains high accuracy while reducing model size.

Abstract

Deep neural networks are state-of-the-art in a wide variety of tasks, however, they exhibit important limitations which hinder their use and deployment in real-world applications. When developing and training neural networks, the accuracy should not be the only concern, neural networks must also be cost-effective and reliable. Although accurate, large neural networks often lack these properties. This thesis focuses on the problem of training neural networks which are not only accurate but also compact, easy to train, reliable and robust to adversarial examples. To tackle these problems, we leverage the properties of structured matrices from the Toeplitz family to build compact and secure neural networks.