# Factorized Higher-Order CNNs with an Application to Spatio-Temporal Emotion Estimation

## TL;DR

This paper introduces a tensor factorization framework for efficient higher-order convolutions, enabling effective spatio-temporal emotion estimation from static images to videos with improved performance.

## Contribution

It unifies tensor decomposition and efficient convolution design, enabling transduction from 2D images to N-dimensional data like videos for emotion analysis.

## Key findings

- Superior performance on AffectNet, SEWA, and AFEW-VA datasets.
- Effective generalization from static images to videos.
- Reduced computational complexity of higher-order convolutions.

## Abstract

Training deep neural networks with spatio-temporal (i.e., 3D) or multidimensional convolutions of higher-order is computationally challenging due to millions of unknown parameters across dozens of layers. To alleviate this, one approach is to apply low-rank tensor decompositions to convolution kernels in order to compress the network and reduce its number of parameters. Alternatively, new convolutional blocks, such as MobileNet, can be directly designed for efficiency. In this paper, we unify these two approaches by proposing a tensor factorization framework for efficient multidimensional (separable) convolutions of higher-order. Interestingly, the proposed framework enables a novel higher-order transduction, allowing to train a network on a given domain (e.g., 2D images or N-dimensional data in general) and using transduction to generalize to higher-order data such as videos (or (N+K)-dimensional data in general), capturing for instance temporal dynamics while preserving the learnt spatial information. We apply the proposed methodology, coined CP-Higher-Order Convolution (HO-CPConv), to spatio-temporal facial emotion analysis. Most existing facial affect models focus on static imagery and discard all temporal information. This is due to the above-mentioned burden of training 3D convolutional nets and the lack of large bodies of video data annotated by experts. We address both issues with our proposed framework. Initial training is first done on static imagery before using transduction to generalize to the temporal domain. We demonstrate superior performance on three challenging large scale affect estimation datasets, AffectNet, SEWA, and AFEW-VA.