Tensor-to-Tensor Models with Fast Iterated Sum Features

TL;DR

The paper introduces a novel tensor-to-tensor neural network layer called the FIS layer, which efficiently processes high-dimensional tensor data with linear complexity, improving performance in image classification and anomaly detection tasks.

Contribution

It proposes the FIS layer based on corner trees and iterated sums, enabling efficient tensor processing with linear cost and seamless integration into existing neural networks.

Findings

Achieved similar accuracy to larger ResNet with fewer parameters.

Attained 97.3% AUROC on MVTec AD anomaly detection dataset.

Demonstrated effectiveness in classification and anomaly detection tasks.

Abstract

Data in the form of images or higher-order tensors is ubiquitous in modern deep learning applications. Owing to their inherent high dimensionality, the need for subquadratic layers processing such data is even more pressing than for sequence data. We propose a novel tensor-to-tensor layer with linear cost in the input size, utilizing the mathematical gadget of ``corner trees'' from the field of permutation counting. In particular, for order-two tensors, we provide an image-to-image layer that can be plugged into image processing pipelines. On the one hand, our method can be seen as a higher-order generalization of state-space models. On the other hand, it is based on a multiparameter generalization of the signature of iterated integrals (or sums). The proposed tensor-to-tensor concept is used to build a neural network layer called the Fast Iterated Sums (FIS) layer which integrates seamlessly with other layer types. We demonstrate the usability of the FIS layer with both classification and anomaly detection tasks. By replacing some layers of a smaller ResNet architecture with FIS, a similar accuracy (with a difference of only 0.1\%) was achieved in comparison to a larger ResNet while reducing the number of trainable parameters and multi-add operations. The FIS layer was also used to build an anomaly detection model that achieved an average AUROC of 97.3\% on the texture images of the popular MVTec AD dataset. The processing and modelling codes are publicly available at https://github.com/diehlj/fast-iterated-sums.