TensorProjection Layer: A Tensor-Based Dimension Reduction Method in Deep Neural Networks

TL;DR

This paper introduces the TensorProjection layer, a tensor-based dimension reduction method for deep neural networks that optimizes mode-wise projections during training, serving as an alternative to pooling or convolutional layers.

Contribution

The paper presents a novel tensor projection layer that reduces dimensions in tensor data within neural networks, optimized during training, and improves over traditional pooling methods.

Findings

Outperforms traditional pooling in image classification and segmentation tasks.

Effectively reduces feature dimensions while maintaining or improving accuracy.

Integrates seamlessly into existing neural network architectures.

Abstract

In this paper, we propose a dimension reduction method specifically designed for tensor-structured feature data in deep neural networks. The method is implemented as a hidden layer, called the TensorProjection layer, which transforms input tensors into output tensors with reduced dimensions through mode-wise projections. The projection directions are treated as model parameters of the layer and are optimized during model training. Our method can serve as an alternative to pooling layers for summarizing image data, or to convolutional layers as a technique for reducing the number of channels. We conduct experiments on tasks such as medical image classification and segmentation, integrating the TensorProjection layer into commonly used baseline architectures to evaluate its effectiveness. Numerical experiments indicate that the proposed method can outperform traditional downsampling methods, such as pooling layers, in our tasks, suggesting it as a promising alternative for feature summarization.