Factorized Higher-Order CNNs with an Application to Spatio-Temporal Emotion Estimation

TL;DR

This paper introduces a tensor factorization framework for efficient higher-order convolutions, enabling effective spatio-temporal emotion estimation from static images to videos with improved performance.

Contribution

It unifies tensor decomposition and efficient convolution design, enabling transduction from 2D images to N-dimensional data like videos for emotion analysis.

Findings

Superior performance on AffectNet, SEWA, and AFEW-VA datasets.

Effective generalization from static images to videos.

Reduced computational complexity of higher-order convolutions.

Abstract

Training deep neural networks with spatio-temporal (i.e., 3D) or multidimensional convolutions of higher-order is computationally challenging due to millions of unknown parameters across dozens of layers. To alleviate this, one approach is to apply low-rank tensor decompositions to convolution kernels in order to compress the network and reduce its number of parameters. Alternatively, new convolutional blocks, such as MobileNet, can be directly designed for efficiency. In this paper, we unify these two approaches by proposing a tensor factorization framework for efficient multidimensional (separable) convolutions of higher-order. Interestingly, the proposed framework enables a novel higher-order transduction, allowing to train a network on a given domain (e.g., 2D images or N-dimensional data in general) and using transduction to generalize to higher-order data such as videos (or (N+K)-dimensional data in general), capturing for instance temporal dynamics while preserving the learnt spatial information. We apply the proposed methodology, coined CP-Higher-Order Convolution (HO-CPConv), to spatio-temporal facial emotion analysis. Most existing facial affect models focus on static imagery and discard all temporal information. This is due to the above-mentioned burden of training 3D convolutional nets and the lack of large bodies of video data annotated by experts. We address both issues with our proposed framework. Initial training is first done on static imagery before using transduction to generalize to the temporal domain. We demonstrate superior performance on three challenging large scale affect estimation datasets, AffectNet, SEWA, and AFEW-VA.

Tables

Table 1. Table 1 : Results on the AffectNet dataset

		Valence				Arousal
Network	Acc.	RMSE	SAGR	PCC	CCC	RMSE	SAGR	PCC	CCC
AffectNet baseline [30]	0.58	0.37	0.74	0.66	0.60	0.41	0.65	0.54	0.34
Face-SSD [13]	-	0.44	0.73	0.58	0.57	0.39	0.71	0.50	0.47
VGG-Face+2M imgs [18]	0.60	0.37	0.78	0.66	0.62	0.39	0.75	0.55	0.54
Baseline ResNet-18	0.55	0.35	0.79	0.68	0.66	0.33	0.8	0.58	0.57
Ours	0.59	0.35	0.79	0.71	0.71	0.32	0.8	0.63	0.63

Table 2. Table 2 : Results on the SEWA database

		Valence				Arousal
Case	Network	RMSE	SAGR	PCC	CCC	RMSE	SAGR	PCC	CCC
Static	[24]	-	-	0.32	0.31	-	-	0.18	0.20
	VGG16+TRL [29]	0.33	-	0.50	0.47	0.39	-	0.44	0.39
	ResNet-18	0.37	0.62	0.33	0.29	0.52	0.62	0.28	0.19
	Ours	0.33	0.65	0.64	0.6	0.39	0.75	0.48	0.44
Temporal	ResNet-3D	0.37	0.59	0.47	0.41	0.41	0.69	0.29	0.21
	ResNet-(2+1)D	0.35	0.63	0.59	0.49	0.41	0.63	0.39	0.31
	Ours – scratch	0.33	0.63	0.62	0.54	0.40	0.72	0.42	0.32
	Ours – transduction	0.24	0.69	0.84	0.75	0.32	0.80	0.60	0.52

Table 3. Table 3 : Results on the AFEW-VA database

		Valence				Arousal
Case	Network	RMSE	SAGR	PCC	CCC	RMSE	SAGR	PCC	CCC
Static	RF Hybrid DCT [23]	0.27	-	0.407	-	0.23	-	0.45	-
	ResNet50+TRL [29]	0.40	-	0.33	0.33	0.41	-	0.42	0.4
	ResNet-18	0.43	0.42	0.05	0.03	0.41	0.68	0.06	0.05
	Ours	0.24	0.64	0.55	0.55	0.24	0.77	0.57	0.52
Temporal	Baseline ResNet-18-3D	0.26	0.56	0.19	0.17	0.22	0.77	0.33	0.29
	ResNet-18-(2+1)D	0.31	0.50	0.17	0.16	0.29	0.73	0.33	0.20
	AffWildNet [19]	-	-	0.51	0.52	-	-	0.575	0.556
	Ours – scratch	0.28	0.53	0.12	0.11	0.19	0.75	0.23	0.15
	Ours – transduction	0.20	0.67	0.64	0.57	0.21	0.79	0.62	0.56

Table 4. Table 4 : Number of parameters optimized to train the temporal model

Network	Total # parameters	# parameters removed with LASSO	# parameters optimized for video
ResNet18-(2+1)D	31M	-	31M
ResNet-18-3D	33M	-	33M
Ours $\left[\lambda =0.01\right]$	11M	0.7	0.24
Ours $\left[\lambda =0.05\right]$	11M	1.3M	0.24

Table 5. Table 5: Results on the LSEMSW database

Method	Accuracy
ResNet 34 [11]	28.39 %
Ours	34.55%

Table 6. Table 6 : Results on the CIFAR-10 dataset

Network	# parameters	Accuracy (%)
MobileNet-v2	$2.30$M	$94$
Ours	$2.29$M	$94$

Table 7. Table 7 : Results on the 20BN-Jester Dataset

	#conv	Accuracy (%)
Network	parameters	Top-1	Top-5
3D-ConvNet	$2.9$M	$83.2$	$97.0$
HO-CP ConvNet (Ours)	$1.2$M	$83.8$	$97.4$
HO-CP ConvNet-S (Ours)	$1.2$M	$85.4$	$98.6$

Equations

$$\Phi (\mathcal{X}{)}_{t,y,x}=\mathcal{X}\star \mathcal{W}=\sum _{k=0}^C{\mathcal{W}}_{t,k,y,x}{\mathcal{X}}_{k,y,x}=\mathcal{X}{\times }_0\mathbf{W}$$

$$\mathcal{F}_{t,y,x}=\leavevmode\color[rgb]{0,0,1}\sum_{k=1}^{C}\leavevmode\color[rgb]{1,0,0}\sum_{j=1}^{H}\leavevmode\color[rgb]{0,0.88,0}\sum_{i=1}^{W}\mathcal{W}(t,\leavevmode\color[rgb]{0,0,1}k,\leavevmode\color[rgb]{1,0,0}j,\leavevmode\color[rgb]{0,0.88,0}i)\mathcal{X}(\leavevmode\color[rgb]{0,0,1}k,\leavevmode\color[rgb]{1,0,0}j+y,\leavevmode\color[rgb]{0,0.88,0}i+x)$$

$$\mathcal{W}_{t,s,j,i}={\color[rgb]{0,0,0}{\sum_{r=0}^{R-1}}}\mathbf{U}^{(T)}_{t,r}\mathbf{U}^{(C)}_{s,r}\mathbf{U}^{(H)}_{j,r}\mathbf{U}^{(W)}_{i,r}$$

$$\mathcal{F}_{t,y,x}=\underbrace{\sum_{r=0}^{R-1}\mathbf{U}^{(T)}_{t,r}\underbrace{\left[\leavevmode\color[rgb]{0,0.88,0}\sum_{i=1}^{W}\mathbf{U}^{(W)}_{\leavevmode\color[rgb]{0,0.88,0}i,r}\underbrace{\left(\leavevmode\color[rgb]{1,0,0}\sum_{j=1}^{H}\mathbf{U}^{(H)}_{\leavevmode\color[rgb]{1,0,0}j,r}\underbrace{\left[\leavevmode\color[rgb]{0,0,1}\sum_{k=1}^{C}\mathbf{U}^{(C)}_{\leavevmode\color[rgb]{0,0,1}k,r}\mathcal{X}(\leavevmode\color[rgb]{0,0,1}k,\leavevmode\color[rgb]{1,0,0}j+y,\leavevmode\color[rgb]{0,0.88,0}i+x)\right]}_{\leavevmode\color[rgb]{0,0,1}1\times 1\text{ conv}}\right)}_{\leavevmode\color[rgb]{1,0,0}\text{ depthwise conv}}\right]}_{\leavevmode\color[rgb]{0,0.88,0}\text{ depthwise conv}}}_{1\times 1\text{ convolution}}$$

$$\mathcal{W}(t,s,j,i)=\sum _{r_0=0}^{R_0-1}\sum _{r_1=0}^{R_1-1}\sum _{r_2=0}^{R_2-1}\sum _{r_3=0}^{R_3-1}{\mathcal{G}}_{r_0,r_1,r_2,r_3}{\mathbf{U}}_{t,r_0}^{(T)}{\mathbf{U}}_{s,r_1}^{(C)}{\mathbf{U}}_{j,r_2}^{(H)}{\mathbf{U}}_{i,r_3}^{(W)}$$

$${\mathcal{F}}_{t,y,x}=\sum _{k=1}^C\sum _{j=1}^H\sum _{i=1}^W\sum _{r_0=0}^{R_0-1}\sum _{r_1=0}^{R_1-1}\sum _{r_2=0}^{R_2-1}\sum _{r_3=0}^{R_3-1}{\mathcal{G}}_{r_0,r_1,r_2,r_3}{\mathbf{U}}_{t,r_0}^{(T)}{\mathbf{U}}_{k,r_1}^{(C)}{\mathbf{U}}_{j,r_2}^{(H)}{\mathbf{U}}_{i,r_3}^{(W)}{\mathcal{X}}_{k,j+y,i+x}$$

$${\mathcal{F}}_{t,y,x}=\sum _{k=1}^C\sum _{j=1}^H\sum _{i=1}^W\sum _{r_0=0}^{R_0-1}\sum _{r_1=0}^{R_1-1}{\mathcal{H}}_{r_0,r_1,j,i}{\mathbf{U}}_{t,r_0}^{(T)}{\mathbf{U}}_{k,r_1}^{(C)}{\mathcal{X}}_{k,j+y,i+x}$$

$$\displaystyle\mathcal{F}_{t,y,x}=\underbrace{\leavevmode\color[rgb]{0,0.88,0}\sum_{r_{0}=0}^{R_{0}-1}\mathbf{U}^{(T)}_{t,\leavevmode\color[rgb]{0,0.88,0}r_{0}}\left[\underbrace{\leavevmode\color[rgb]{1,0,0}\sum_{j=1}^{H}\sum_{i=1}^{W}\sum_{r_{1}=0}^{R_{1}-1}\mathcal{H}_{\leavevmode\color[rgb]{0,0.88,0}r_{0},\leavevmode\color[rgb]{1,0,0}r_{1},\leavevmode\color[rgb]{1,0,0}j,\leavevmode\color[rgb]{1,0,0}i}\left[\underbrace{\leavevmode\color[rgb]{0,0,1}\sum_{k=1}^{C}\mathbf{U}^{(C)}_{\leavevmode\color[rgb]{0,0,1}k,\leavevmode\color[rgb]{1,0,0}r_{1}}\mathcal{X}(\leavevmode\color[rgb]{0,0,1}k,\leavevmode\color[rgb]{1,0,0}j+y,\leavevmode\color[rgb]{1,0,0}i+x)}_{\leavevmode\color[rgb]{0,0,1}1\times 1\text{ conv}}\right]}_{\leavevmode\color[rgb]{1,0,0}H\times W\text{ conv}}\right]}_{\leavevmode\color[rgb]{0,0.88,0}1\times 1\text{ conv}}$$

$$\mathcal{F}=\left(\left(\mathcal{X}{\times }_0{\mathbf{U}}^{(C)}\right)\star \mathcal{G}\right){\times }_0{\mathbf{U}}^{(T)}$$

$$\mathcal{W}_{t,\leavevmode\color[rgb]{0,0,1}s,\leavevmode\color[rgb]{1,0,0}j,i}={\color[rgb]{0,0,0}{\sum_{\leavevmode\color[rgb]{0,0.88,0}r=0}^{R-1}}}\mathbf{U}^{(T)}_{t,\leavevmode\color[rgb]{0,0.88,0}r}\mathbf{U}^{(C)}_{\leavevmode\color[rgb]{0,0,1}s,\leavevmode\color[rgb]{0,0.88,0}r}\mathbf{U}^{(S)}_{\leavevmode\color[rgb]{1,0,0}j,i,\leavevmode\color[rgb]{0,0.88,0}r}$$

$$\begin{split}\Phi(\mathcal{X})_{t,i_{0},\cdots,i_{N-1}}&=\sum_{r=0}^{R}\sum_{s=0}^{C}\sum_{i_{0}=0}^{K_{0}}\cdots\\ &\cdots\sum_{i_{N-1}=0}^{K_{N-1}}\lambda_{r}\bigl{[}\mathbf{U}^{(T)}_{t,r}\mathbf{U}^{(C)}_{s,r}\mathbf{U}^{(K_{0})}_{i_{0},r}\cdots\mathbf{U}^{(K_{N-1})}_{i_{N-1},r}\mathcal{X}_{s,i_{0},\cdots,i_{N-1}}\bigr{]}\end{split}$$

$$\mathcal{F}=\left(\rho \left(\mathcal{X}{\times }_0{\mathbf{U}}^{(T)}\right)\right){\times }_0\left(\mathrm{diag}(\mathbf{\lambda }){\mathbf{U}}^{(C)}\right)$$

$$\rho (\mathcal{X})=\left(\mathcal{X}{\star }_1{\mathbf{U}}^{(K_0)}{\star }_2{\mathbf{U}}^{(K_1)}\star \cdots {\star }_{N+1}{\mathbf{U}}^{(K_{N-1})}\right)$$

$$\mathcal{F}=\rho \left(\Psi \left(\mathcal{X}{\times }_0{\mathbf{U}}^{(T)}\right)\right){\times }_0\left(\mathrm{diag}(\mathbf{\lambda }){\mathbf{U}}^{(C)}\right)$$

$$\mathcal{F}=\hat{\rho }\left(\Psi \left(\mathcal{X}{\times }_0{\mathbf{U}}^{(T)}\right)\right){\times }_0\left(\mathrm{diag}(\mathbf{\lambda }){\mathbf{U}}^{(C)}\right),$$

Full text

We apply the proposed methodology, coined CP-Higher-Order Convolution (HO-CPConv), to spatio-temporal facial emotion analysis. Most existing facial affect models focus on static imagery and discard all temporal information. This is due to the above-mentioned burden of training 3D convolutional nets and the lack of large bodies of video data annotated by experts. We address both issues with our proposed framework. Initial training is first done on static imagery before using transduction to generalize to the temporal domain. We demonstrate superior performance on three challenging large scale affect estimation datasets, AffectNet, SEWA, and AFEW-VA.

1 Introduction

With the unprecedented success of deep convolutional neural networks came the quest for training always deeper networks. However, while deeper neural networks give better performance when trained appropriately, that depth also translates into memory and computationally heavy models, typically with tens of millions of parameters. This is especially true when training training higher-order convolutional nets –e.g. third order (3D) on videos. However, such models are necessary to perform predictions in the spatio-temporal domain and are crucial in many applications, including action recognition and emotion recognition.

In this paper, we depart from conventional approaches and propose a novel factorized multidimensional convolutional block that achieves superior performance through efficient use of the structure in the data. In addition, our model can first be trained on the image domain and extended seamlessly to transfer performance to the temporal domain. This novel transduction method is made possible by the structure of our proposed block. In addition, it allows one to drastically decrease the number of parameters, while improving performance and computational efficiency and it can be applied to already existing spatio-temporal network architectures such as a ResNet3D [47]. Our method leverages a CP tensor decomposition, in order to separately learn the disentangled spatial and temporal information of 3D convolutions. This improves accuracy by a large margin while reducing the number of parameters of spatio-temporal architectures and greatly facilitating training on video data.

In summary, we make the following contributions:

• •

We show that many of deep nets architectural improvements, such as MobileNet or ResNet’s Bottleneck blocks, are in fact drawn from the same larger family of tensor decomposition methods (Section 3). We propose a general framework unifying tensor decompositions and efficient architectures, showing how these efficient architectures can be derived by applying tensor decomposition to convolutional kernels (Section 3.4)

• •

Using this framework, we propose factorized higher-order convolutional neural networks, that leverage efficient general multi-dimensional convolutions. These achieve the same performance with a fraction of the parameters and floating point operations (Section 4).

• •

Finally, we propose a novel mechanism called higher-order transduction which can be employed to convert our model, trained in $N$ dimensions, to $N+K$ dimensions.

• •

We show that our factorized higher-order networks outperform existing works on static affect estimation on the AffectNet, SEWA and AFEW-VA datasets.

• •

Using transduction on the static models, we also demonstrate state-of-the-art results for continuous facial emotion analysis from video on both SEWA and AFEW-VA datasets.

2 Background and related work

Multidimensional convolutions

arise in several mathematical models across different fields. They are a cornerstone of Convolutional Neural Networks (CNNs) [26, 28] enabling them to effectively learn from high-dimensional data by mitigating the curse of dimensionality [34]. However, CNNs are computationally demanding, with the cost of convolutions being dominant during both training and inference. As a result, there is an increasing interest in improving the efficiency of multidimensional convolutions.

Several efficient implementations of convolutions have been proposed. For instance, 2D convolution can be efficiently implemented as matrix multiplication by converting the convolution kernel to a Toeplitz matrix. However, this procedure requires replicating the kernel values multiple times across different matrix columns in the Toeplitz matrix, thus increasing the memory requirements. Implementing convolutions via the im2col approach is also memory intensive due to the space required for building the column matrix. These memory requirements may be prohibitive for mobile or embedded devices, hindering the deployment of CNNs in resource-limited platforms.

In general, most existing attempts at efficient convolutions are isolated and there currently is no unified framework to study them. In particular we are interested in two different branches of work, which we review next. Firstly, approaches that leverage tensor methods for efficient convolutions, either to compress or reformulate them for speed. Secondly, approaches that directly formulate efficient neural architecture, e.g., using separable convolutions.

Tensor methods for efficient deep networks

The properties of tensor methods [14, 41, 17] make them a prime choice for deep learning. Beside theoretical study of the properties of deep neural networks [3], they have been especially studied in the context of reparametrizing existing layers [49]. One goal of such reparametrization is parameter space savings [7]. [31] for instance proposed to reshape the weight matrix of fully-connected layers into high-order tensors with a Tensor-Train (TT) [32] structure. In a follow-up work [4], the same strategy is also applied to convolutional layers. Fully connected layers and flattening layers can be removed altogether and replaced with tensor regression layers [21]. These express outputs through a low-rank multi-linear mapping from a high-order activation tensor to an output tensor of arbitrary order. Parameter space saving can also be obtained, while maintaining multi-linear structure, by applying tensor contraction [20].

Another advantage of tensor reparametrization is computational speed-up. In particular, a tensor decomposition is an efficient way of obtaining separable filters from convolutional kernels. These separable convolutions were proposed in computer vision by [36] in the context of filter banks. [12] first applied this concept to deep learning and proposed leveraging redundancies across channels using separable convolutions. [1, 27] proposed to apply CP decomposition directly to the (4–dimensional) kernels of pretrained 2D convolutional layers, making them separable. The incurred loss in performance is compensated by fine-tuning.

Efficient rewriting of convolutions can also be obtained using Tucker decompositions instead of CP to decompose the convolutional layers of a pre-trained network [15]. This allows rewriting the convolution as a $1\times 1$ convolution, followed by regular convolution with a smaller kernel and another $1\times 1$ convolution. In this case, the spatial dimensions of the convolutional kernel are left untouched and only the input and output channels are compressed. Again, the loss in performance is compensated by fine-tuning the whole network. Finally, [46] propose to remove redundancy in convolutional layers and express these as the composition of two convolutional layers with less parameters. Each 2D filter is approximated by a sum of rank–$1$ matrices. Thanks to this restricted setting, a closed-form solution can be readily obtained with SVD. Here, we unify the above works and propose factorized higher-order (separable) convolutions.

Efficient neural networks

While concepts such as separable convolutions have been studied since the early successes of deep learning using tensor decompositions, they have only relatively recently been “rediscovered” and proposed as standalone end-to-end trainable efficient neural network architectures. The first attempts in the direction of neural network architecture optimization were proposed early in the ground-breaking VGG network [42] where the large convolutional kernels used in AlexNet [25] were replaced with a series of smaller ones that have an equivalent receptive field size: i.e. a convolution with a $5\times 5$ kernel can be replaced by two consecutive convolutions of size $3\times 3$. In parallel, the idea of decomposing larger kernels into a series of smaller ones is explored in the multiple iterations of the Inception block [43, 44, 45] where a convolutional layer with a $7\times 7$ kernel is approximated with two $7\times 1$ and $1\times 7$ kernels. [8] introduced the so-called bottleneck module that reduces the number of channels on which the convolutional layer with higher kernel size ($3\times 3$) operate on by projecting back and forth the features using two convolutional layers with $1\times 1$ filters. [48] expands upon this by replacing the $3\times 3$ convolution with a grouped convolutional layer that can further reduce the complexity of the model while increasing representational power at the same time. Recently, [10] introduced the MobileNet architecture where they proposed to replace the $3\times 3$ convolutions with a depth-wise separable module: a depth-wise $3\times 3$ convolution (the number of groups is equal to the number of channels) followed by a $1\times 1$ convolutional layer that aggregates the information. These type of structures were shown to offer a good balance between the performance offered and the computational cost they incur. [39] goes one step further and incorporates the idea of using separable convolutions in an inverted bottleneck module. The proposed module uses $1\times 1$ layers to expand and then contract the channels (hence inverted bottleneck) while using separable convolutions for the $3\times 3$ convolutional layer.

Facial affect analysis

is the first step towards better human-computer interactions. Early research focused on detecting discrete emotions such as happiness and sadness, based on the hypothesis that these are universal. However, this categorization of human affect is limited and does not cover the wide emotional spectrum displayed by humans on a daily basis. Psychologists have since moved towards more fine grained dimensional measures of affect [35, 38]. The goal is to estimate continuous levels of valence –how positive or negative an emotional display is– and arousal –how exciting or calming is the emotional experience. This task is the subject of most of the recent research in affect estimation [37, 19, 23, 50] and is the focus of this paper.

Valence and arousal are dimensional measures of affect that vary in time. It is these changes that are important for accurate human-affect estimation. Capturing temporal dynamics of emotions is therefore crucial and requires video rather than static analysis. However, spatio-temporal models are difficult to train due to their large number of parameters, requiring very large amounts of annotated videos to be trained successfully. Unfortunately, the quality and quantity of available video data and annotation collected in naturalistic conditions is low [40]. As a result, most work in this area of affect estimation in-the-wild focuses on affect estimation from static imagery [23, 30]. Estimation from videos is then done on a frame-by-frame basis. Here, we tackle both issues and train spatio-temporal networks that outperform existing methods for affect estimation.

3 Convolutions in a tensor framework

In this section, we explore the relationship between tensor methods and deep neural networks’ convolutional layers. Without loss of generality, we omit the batch size in all the following formulas.

Mathematical background and notation

We denote $1^{\text{st}}$–order tensors (vectors) as $\mathbf{\bm{v}}$, $2^{\text{nd}}$–order tensor (matrices) as $\mathbf{M}$ and tensors of order $\geq 3$ as $\mathcal{T}$. We denote a regular convolution of $\mathcal{X}$ with $\mathbf{W}$ as $\mathcal{X}\star_{n}\mathbf{W}$. For $1$–D convolutions, we write the convolution of a tensor $\mathcal{X}\in\mathbb{R}^{I_{0},\cdots,I_{N}}$ with a vector $\mathbf{\bm{v}}\in\mathbb{R}^{K}$ along the $n^{\text{th}}$–mode as $\mathcal{X}\star_{n}\mathbf{\bm{v}}$. In practice, as done in current deep learning frameworks [33], we use cross-correlation, which differs from a convolution by a flip of the kernel. This does not impact the results since the weights are learned end-to-end. In other words, $\left(\mathcal{X}\star_{n}\mathbf{\bm{v}}\right)_{i_{0},\cdots,i_{N}}=\sum_{k=1}^{K}\mathbf{\bm{v}}_{k}\mathcal{X}_{i_{0},\cdots,i_{n-1},i_{n}+k,i_{n+1},\cdots,I_{N}}$.

3.1 $1\times 1$ convolutions and tensor contraction

We show that $1\times 1$ convolutions are equivalent to a tensor contraction with the kernel of the convolution along the channels dimension. Let’s consider a $1\times 1$ convolution $\Phi$, defined by kernel $\mathcal{W}\in\mathbb{R}^{T\times C\times 1\times 1}$ and applied to an activation tensor $\mathcal{X}\in\mathbb{R}^{C\times H\times W}$. We denote the squeezed version of $\mathcal{W}$ along the first mode as $\mathbf{W}\in\mathbb{R}^{T\times C}$.

The tensor contraction of a tensor $\mathcal{T}\in\mathbb{R}^{I_{0}\times I_{1}\times\cdots\times I_{N}}$ with matrix $\mathbf{M}\in\mathbb{R}^{J\times I_{n}}$, along the $n^{\text{th}}$–mode ($n\in[{0}\mathrel{{.}\,{.}}\nobreak{N}]$), known as n–mode product, is defined as $\mathcal{P}=\mathcal{T}\times_{n}\mathcal{M}$, with: $\mathcal{P}_{i_{0},\cdots,i_{N}}=\sum_{k=0}^{I_{n}}\mathcal{T}_{i_{0},\cdots,i_{n-1},k,i_{n+1},\cdots,i_{N}}\mathbf{M}_{i_{n},k}$

By plugging this into the expression of $\Phi(\mathcal{X})$, we readily observe that the $1\times 1$ convolution is equivalent with an n-mode product between $\mathcal{X}$ and the matrix $\mathbf{W}$:

[TABLE]

3.2 Kruskal convolutions

Here we show how separable convolutions can be obtained by applying CP decomposition to the kernel of a regular convolution [27]. We consider a convolution defined by its kernel weight tensor $\mathcal{W}\in\mathbb{R}^{T\times C\times K_{H}\times K_{W}}$, applied on an input of size $\mathbb{R}^{C\times H\times W}$. Let $\mathcal{X}\in\mathbb{R}^{C\times H\times W}$ be an arbitrary activation tensor. If we define the resulting feature map as $\mathcal{F}=\mathcal{X}\star\mathcal{W}$, we have:

[TABLE]

Assuming a low-rank Kruskal structure on the kernel $\mathcal{W}$ (which can be readily obtained by applying CP decomposition), we can write:

[TABLE]

By plugging 2 into 1 and re-arranging the terms, we get:

[TABLE]

This allows to replace the original convolution by a series of efficient depthwise separable convolutions [27], figure 2.

3.3 Tucker convolutions

As previously, we consider the convolution $\mathcal{F}=\mathcal{X}\star\mathcal{W}$. However, instead of a Kruskal structure, we now assume a low-rank Tucker structure on the kernel $\mathcal{W}$ (which can be readily obtained by applying Tucker decomposition) and yields an efficient formulation [15]. We can write:

[TABLE]

Plugging back into a convolution, we get:

[TABLE]

We can further absorb the factors along the spacial dimensions into the core by writing $\mathcal{H}=\mathcal{G}\times_{2}\mathbf{U}^{(H)}_{j,r_{2}}\times_{3}\mathbf{U}^{(W)}_{i,r_{3}}.$

In that case, the expression above simplifies to:

[TABLE]

In other words, this is equivalence to first transforming the number of channels, then applying a (small) convolution before returning from the rank to the target number of channels. This can be seen by rearranging the terms from Equation 3:

[TABLE]

In short, this simplifies to the following expression, also illustrated in Figure 3:

[TABLE]

3.4 Efficient architectures in a tensor framework

While tensor decompositions have been explored in the field of mathematics for decades and in the context of deep learning for years, they are regularly rediscovered and re-introduced in different forms. Here, we revisit popular deep neural network architectures under the lens of tensor factorization. Specifically, we show how these blocks can be obtained from a regular convolution by applying tensor decomposition to its kernel. In practice, batch-normalisation layers and non-linearities are inserted in between the intermediary convolution to facilitate learning from scratch.

ResNet Bottleneck block

[9] introduced a block, coined Bottleneck block in their seminal work on deep residual networks. It consists in a series of a $1\times 1$ convolution, to reduce the number of channels, a smaller regular ($3\times 3$) convolution, and another $1\times 1$ convolution to restore the rank to the desired number of output channels. Based on the equivalence derived in Section 3.3, it is straightforward to see this as applying Tucker decomposition to the kernel of a regular convolution.

ResNext and Xception

ResNext [48] builds on this bottleneck architecture, which, as we have shown, is equivalent to applying Tucker decomposition to the convolutional kernel. In order to reduce the rank further, the output is expressed as a sum of such bottlenecks, with a lower-rank. This can be reformulated efficiently using grouped-convolution [48]. In parallel, a similar approach was proposed by [2], but without $1\times 1$ convolution following the grouped depthwise convolution.

MobileNet v1

MobileNet v1 [10] uses building blocks made of a depthwise separable convolutions (spatial part of the convolution) followed by a $1\times 1$ convolution to adjust the number of output channels. This can be readily obtained from a CP decomposition (Section 3.2) as follows: first we write the convolutional weight tensor as detailed in Equation 2, with a rank equal to the number of input channels, i.e. $R=C$. The first depthwise-separable convolution can be obtained by combining the two spatial $1$D convolutions $\mathbf{U}^{(H)}$ and $\mathbf{U}^{(W)}$. This results into a single spatial factor \leavevmode\color[rgb]{0,0,1}\mathbf{U}^{(S)}\in\mathbb{R}^{H\times W\times R}, such that $\mathbf{U}^{(S)}_{j,i,r}=\mathbf{U}^{(H)}_{j,r}\mathbf{U}^{(W)}_{i,r}$. The $1\times 1$ convolution is then given by the matrix-product of the remaining factors, \leavevmode\color[rgb]{1,0,0}\mathbf{U}^{(F)}=\mathbf{U}^{(T)}\left(\mathbf{U}^{(C)}\right)^{\top}\in\mathbb{R}^{T\times C}. This is illustrated in Figure 4.

MobileNet v2

MobileNet v2 [39] employs a similar approach by grouping the spatial factors into one spatial factor $\mathbf{U}^{(S)}\in\mathbb{R}^{H\times W\times R}$, as explained previously for the case of MobileNet. However, the other factors are left untouched. The rank of the decomposition, in this case, corresponds, for each layer, to the expansion factor $\times$ the number of input channels. This results in two $1\times 1$ convolutions and a $3\times 3$ depthwise separable convolution. Finally, the kernel weight tensor (displayed graphically in Figure 5) is expressed as:

[TABLE]

In practice, MobileNet-v2 also includes batch-normalisation layers and non-linearities as well as a skip connection to facilitate learning.

4 Factorized higher-order convolutions

We propose to generalize the framework introduced above to convolutions of any arbitrary order. Specifically, we express, in the general case, separable ND-convolutions as a series of tensor contractions and 1D convolutions. We show how this is derived from a CP convolution on the N–dimensional kernel. We then detail how to expand our proposed factorized higher-order convolutions, trained in N-dimensions to $(N+1)$ dimensions.

Efficient N-D convolutions via higher-order factorization

In particular, here, we consider an $N+1^{\text{th}}$–order input activation tensor $\mathcal{X}\in\mathbb{R}^{C\times D_{0}\times\cdots\times D_{N-1}}$ corresponding to $N$ dimensions with $C$ channels. We define a general, high order separable convolution $\Phi$ defined by a kernel $\mathcal{W}\in\mathbb{R}^{T\times C\times K_{0}\times\cdots\times K_{N-1}}$, and expressed as a Kruskal tensor, i.e. $\mathcal{W}=\llbracket{}{\mathbf{\bm{\lambda}};\,\,\mathbf{U}^{(T)},\mathbf{U}^{(C)},\mathbf{U}^{(K_{0})},\cdots,\mathbf{U}^{(K_{N-1})}}\rrbracket{}$. We can then write:

[TABLE]

By rearranging the terms, this expression can be rewritten as:

[TABLE]

where $\rho$ applies the 1D spatial convolutions:

[TABLE]

Tensor decompositions (and, in particular, decomposed convolutions) are notoriously hard to train end-to-end [12, 27, 46]. As a result, most existing approach rely on first training an uncompressed network, then decomposing the convolutional kernels before replacing the convolution with their efficient rewriting and fine-tuning to recover lost performance. However, this approach is not suitable for higher-order convolutions where it might not be practical to train the full N-D convolution. It is possible to facilitate training from scratch by absorbing the magnitude of the factors into the weight vector $\mathbf{\bm{\lambda}}$. We can also add add non-linearities $\Psi$ (e.g. batch normalisation combined with RELU), leading to the following expression, resulting in an efficient higher-order CP convolution:

[TABLE]

Skip connection can also be added by introducing an additional factor $\mathbf{U}^{(S)}\in\mathbb{R}^{T\times C}$ and using $\mathcal{F}^{\prime}=\mathcal{X}+\left(\mathcal{F}\times_{0}\mathbf{U}^{(S)}\right)$.

This formulation is significantly more efficient than that of a regular convolution. Let’s consider an N-dimensional convolution, with $C$ input channels and $T$ output channels, i.e. a weight of size $\mathcal{W}\in\mathbb{R}^{C\times T\times I_{0}\times\cdots\times I_{N-1}}$. Then a regular 3D convolution has $C\times T\times\left(\prod_{k=0}^{N-1}I_{k}\right)$ parameters. By contrast, our HO-CP convolution with rank $R$ has only $R\left(C+T+\sum_{k=0}^{N-1}I_{k}\right)+1$ parameters. The $+1$ term accounts for the weights $\mathbf{\bm{\lambda}}$. For instance, for a 3D convolution with a cubic kernel (of size $K\times K\times K$, a regular 3D convolution would have $CTK^{3}$ parameters, versus only $R(C+T+3K)$ for our proposed factorized version.

This reduction in the number of parameters translates into much more efficient operation in terms of floating point operations (FLOPs). We show, in Figure 6, a visualisation of the number of Giga FLOPs (GFLOPs, with 1GFLOP = $1e^{9}$ FLOPs), for both a regular 3D convolution and our proposed approach, for an input of size $32\times 32\times 16$, varying the number of input and output channels, with a kernel size of $3\times 3\times 3$.

Higher-Order Transduction

Here, we introduce transduction, which allows to first train an $N$–dimensional convolution and expand it to $(N+K)$ dimensions, $K>0$.

Thanks to the efficient formulation introduced in Equation 7, we now have an effective way to go from $N$ dimensions to $N+1$. We place ourselves in the same setting as for Equation 7, where we have a regular N-D convolution with spatial dimensions and extend the model to $N+1$ dimensions. To do so, we introduce a new factor $\mathbf{U}^{(K_{N+1})}\in\mathbb{R}^{(K^{N+1}\times R)}$ corresponding to the new $N+1^{\text{th}}$ dimension. The final formulation is then :

[TABLE]

with $\hat{\rho}(\mathcal{X})=\rho(\mathcal{X})\star_{N+1}\mathbf{U}^{(K_{N+1})}$.

Note that only the new factor needs to be trained, e.g. transduction can be done by simply training $K_{N+1}\times R$ additional parameters.

Automatic Rank Selection

Our proposed factorized higher-order convolutions introduce a new additional parameter, corresponding to the rank of the factorization. This can be efficiently incorporated in the formulation by introducing a vector of weights, represented by $\mathbf{\bm{\lambda}}$ in Equation 8. This allows us to automatically tune the rank of each of the layers by introducing an additional Lasso term in the loss function, e.g., an $\ell_{1}$ regularization on $\mathbf{\bm{\lambda}}$. Let $\mathbf{\bm{\lambda}}_{l}$ be the vector of weights for each layer $l\in[{0}\mathrel{{.}\,{.}}\nobreak{L-1}]$ of our neural network, associated with a loss $\mathcal{L}$. The overall loss with regularization will become $\mathcal{L}_{\text{reg}}=\mathcal{L}+\gamma\sum_{l=0}^{L-1}|\mathbf{\bm{\lambda}}_{l}|$, where $\gamma$ controls the amount of sparsity in the weights.

5 Experimental setting

Datasets for In-The-Wild Affect Estimation

We validate the performance of our approach on established large scale datasets for continuous affect estimation in-the-wild.

AffectNet [30] is a large static dataset of human faces and labelled in terms of facial landmarks, emotion categories as well as valence and arousal values. It contains more than a million images, including $450,000$ manually labelled by twelve expert annotators.

AFEW-VA [23] is composed of video clips taken from feature films and accurately annotated per frame in terms of continuous levels of valence and arousal. Besides $68$ accurate facial landmarks are also provided for each frame.

SEWA [24] is the largest video dataset for affect estimation in-the-wild. It contains over $2000$ minutes of audio and video data annotated in terms of facial landmarks, valence and arousal values. It contains 398 subjects from six different cultures, is gender balanced and uniformly spans the age range 18 to 65.

Implementation details

We implemented all models using PyTorch [33] and TensorLy [22]. In all cases, we divided the dataset in subject independent training, validation and testing sets. For our factorized higher-order convolutional network, we further removed the flattening and fully-connected layers and replace them with a single tensor regression layer [21] in order to fully preserve the structure in the activations. For training, we employed an Adam optimizer [16] and validated the learning rate in the range $[10^{-5};0.01]$, the beta parameters in the range $[0.0;0.999]$ and the weight decay in the range $[0.0;0.01]$ using a randomized grid search. We also decreased the learning rate by a factor of $10$ every $15$ epochs. The regularization parameter $\gamma$ was validated in the range $[10^{-4};1.0]$ on AffectNet and consequently set to $0.01$ for all other experiments.

For our baseline we use both a 3D ResNet and a ResNet 2+1D [47], both with a ResNet-18 backbone. For our method, we use the same ResNet-18 architecture but replace each of the convolutional layers with our proposed higher-order factorized convolution. We initialised the rank so that the number of parameters would be the same as the original convolution. When performing transduction, the added temporal factors to the CP convolutions are initialized to a constant value of one. In a first step, these factors are optimized while keeping the remaining parameters fixed. The whole network is then fine-tuned. This avoids the transducted factors to pollute what has already been learnt in the static case. The full process is summarized in Figure 1.

Performance metrics and loss function

In all cases, we report performance for the RMSE, SAGR, PCC, and CCC which are common metrics employed in affect estimation. Let $\mathbf{\bm{y}}$ be a ground-truth signal and $\mathbf{\bm{\hat{y}}}$ the associated prediction by the model.

The RMSE is the well known Root Mean Square Error: $\text{RMSE}(\mathbf{\bm{y}},\hat{\mathbf{\bm{y}}})=\sqrt{\mathbb{E}((\mathbf{\bm{y}}-\hat{\mathbf{\bm{y}}})^{2})}.$

The SAGR assesses whether the sign of the two signals agree: $SAGR(Y,\hat{\mathbf{\bm{y}}})=\frac{1}{n}\sum^{n}_{i=1}\delta(\text{sign}(\mathbf{\bm{y}}_{i}),\text{sign}(\hat{\mathbf{\bm{y}}}_{i})).$

The PCC is the Pearson product-moment correlation coefficient and measures how correlated the two signals are: $PCC(\mathbf{\bm{y}},\hat{\mathbf{\bm{y}}})=\frac{\mathbb{E}(\mathbf{\bm{y}}-\mu_{\mathbf{\bm{y}}})(\hat{\mathbf{\bm{y}}}-\mu_{\hat{\mathbf{\bm{y}}}})}{\sigma_{\mathbf{\bm{y}}}\sigma_{\hat{\mathbf{\bm{y}}}}}.$

The CCC is Lin’s Concordance Correlation Coefficient and assesses the correlation of the two signals but also how close the two signals are: $CCC(\mathbf{\bm{y}},\hat{\mathbf{\bm{y}}})=\frac{2\sigma_{\mathbf{\bm{y}}}\sigma_{\hat{\mathbf{\bm{y}}}}\text{PCC}(\mathbf{\bm{y}},\hat{\mathbf{\bm{y}}})}{\sigma^{2}_{\mathbf{\bm{y}}}+\sigma^{2}_{\hat{\mathbf{\bm{y}}}}+(\mu_{\mathbf{\bm{y}}}-\mu_{\hat{\mathbf{\bm{y}}}})^{2}}.$

The goal for continuous affect estimation is typically to maximize the correlations coefficients PCC and CCC. However minimizing the RMSE also helps maximizing the correlations as it gives a lower error in each individual prediction. Our regression loss function reflects this by incorporating three terms: $\mathcal{L}=\frac{1}{\alpha+\beta+\gamma}(\alpha\mathcal{L}_{RMSE}+\beta\mathcal{L}_{PCC}+\gamma\mathcal{L}_{CCC})$, with $\mathcal{L}_{RMSE}=\text{RMSE}_{\text{valence}}+\text{RMSE}_{\text{arousal}}$,

$\mathcal{L}_{PCC}=1-\frac{\text{PCC}_{\text{valence}}+\text{PCC}_{\text{arousal}}}{2}$ and $\mathcal{L}_{CCC}=1-\frac{\text{CCC}_{\text{valence}}+\text{CCC}_{\text{arousal}}}{2}.$ The coefficients $\alpha$, $\beta$ and $\gamma$ are shake-shake regularization coefficients [5] sampled randomly in the range $[0;1]$ following a uniform distribution. These ensures none of the terms are ignored during optimization. On AffectNet, where discrete classes of emotions are available, we jointly perform a regression of the valence and arousal values and a classification of the emotional class by adding a cross entropy to the loss function.

6 Performance evaluation

In this section, we report the performance of our methods for facial affect estimation in the wild. First, we report results on static images. We then show how the higher-order transduction allows us to extend these models to the temporal domain. In all cases, we compare with the state-of-the-art and show superior results.

Static affect analysis in-the-wild with factorized CNNs

First we show the performance of our models trained and tested on individual (static) imagery. We train our method on AffectNet, which is the largest database but consists of static images only. There, our method outperforms all other works by a large margin (Table 1). We observe similar results on SEWA (Table 2) and AFEW-VA (Table 3). In the supplementary document, we also report results on LSEMSW [11] and CIFAR10 [25].

Temporal prediction via higher-order transduction

We then apply transduction as described in the method section to convert the static model from SEWA (Table 2, temporal case) and AFEW-VA (Table 3, temporal case) to the temporal domain, where we simply train the added temporal factors. This approach allows to efficiently train temporal models on even small datasets. Our method outperforms other approaches, despite having only $11$ million parameters, compared to $33$ million for the corresponding 3D ResNet18, and $31$ million parameters for a (2+1)D ResNet-18. Interestingly, in all cases, we notice that valence is better predicted than arousal, which, is in line with finding by psychologists that humans are better at estimating valence from visual data [38, 6].

Using the automatic rank selection procedure detailed in section 4, we let the model learn end-to-end the rank of each of the factorized higher-order convolutions. We found that on average, 8 to 15% of the parameters can be set to zero for optimal performance. In practice, about 1 million (of the 11 million) parameters were set to zero by the Lasso regularization. An in-depth study of the effect of the automatic rank selection is provided in the supplementary document.

7 Conclusion

We established the link between tensor factorizations and efficient convolutions, in a unified framework. Based on this, we proposed a factorized higher-order (N-dimensional) convolutional block. This results in efficient models that outperform traditional networks, while being more computationally and memory efficient. We also introduced a higher-order transduction algorithm for converting the expressive power of trained $N$–dimensional convolutions to any $N+K$ dimensions. We then applied our approach to continuous facial affect estimation in naturalistic conditions. Using transduction, we transferred the performance of the models trained on static images to the temporal domain and reported state-of-the-art results in both cases.

APPENDIX

Here, we give additional details on the method, as well as results on other tasks (image classification on CIFAR-10, and gesture estimation from videos).

Dimensional model of affect

Discrete emotional classes are too coarse to summarize the full range of emotions displayed by humans on a daily basis. This is the reason why finer, dimensional affect models, such as the valence and arousal are now favoured by psychologists [38].In this circumplex, which can be seen in Figure 7, the valence level corresponds to how positive or negative an emotion is, while the arousal level explains how calming or exciting the emotion is.

A visualization of the prediction of valence and arousal of our model can be seen in Figure 8, along with some representative frames.

Automatic rank selection

Using the automatic rank selection procedure detailed in the method section, we let the model learn end-to-end the rank of each of the factorized higher-order convolutions.

In Figure 9, we show the number of parameters set to zero by the network for a regularization parameter of $0.01$ and $0.05$. The lasso regularization is an easy way to automatically tune the rank. We found that on average 8 to 15% of the parameters can be set to zero for optimal performance. In practice, about 1 million parameters were removed thanks to this regularization.

In Table 4 we report the number of parameters of spatio-temporal baselines and compare it to our CP factorized model. Besides having less parameters, our approach has the advantage of having a very low number of temporal parameters which facilitates the training on spatio-temporal data once it has been pretrained on static data.

Results on LSEMSW

LSEMSW [11] is the Large-scale Subtle Emotions and Mental States in the Wild database. It contains more than $175,000$ static images annotated in terms of subtle emotions and cognitive states such as helpless, suspicious, arrogant, etc. We report results on LSEMSW in table. 5.

Results on CIFAR-10

While in the paper we focus on affect estimation, we report here results on a traditional image classification dataset, CIFAR 10.

CIFAR-10 [25] is a dataset for image classification composed of $10$ classes with $6,000$ images which, divided into $5000$ images per class for training and $1000$ images per class for testing, on which we report the results.

We used a MobileNet-v2 as our baseline. For our approach, we simply replaced the full MobileNet-v2 blocks with ours (which, in the 2D case, differs from MobileNet-v2 by the use of two separable convolutions along the spatial dimensions instead of a single 2D kernel). We kept all the parameters the same for all experiments to allow for fair comparison and reported performance averaged across $3$ runs. The standard deviation was $0.033$ for MobileNet-v2 and $0.036$ for our approach. We optimized the loss using stochastic gradient descent with a mini-batch size of $128$, starting with a learning rate of $0.1$, decreased by a factor of $10$ after $150$ and $250$ epochs, for a total of $400$ epochs, with a momentum of $0.9$. For MobileNet-v2, we used a weight decay of $4e^{-5}$ and $1e^{-4}$ for our approach. Optimization was done on a single NVIDIA 1080Ti GPU.

We compared our method with a MobileNet-v2 with a comparable number of parameters, Table 6. Unsurprisingly, both approach yield similar results since, in the 2D case, the two networks architectures are similar. It is worth noting that our method has marginally less parameters than MobileNet-v2, for the same number of channels, even though that network is already optimized for efficiency.

Results on gesture estimation

In this section, we report additional results on the Jester dataset. We compare a network that employs regular convolutional blocks to the same network that uses our proposed higher-order factorized convolutional blocks.

20BN-Jester v1 is a dataset111Dataset available at https://www.twentybn.com/datasets/jester/v1. composed of $148,092$ videos, each representing one of $27$ hand gestures (e.g. swiping left, thumb up, etc). Each video contains a person performing one of gestures in front of a web camera. Out of the $148,092$ videos $118,562$ are used for training, $14,787$ for validation on which we report the results.

For the 20BN-Jester dataset, we used a convolutional column composed of $4$ convolutional blocks with kernel size $3\times 3\times 3$, with respective input and output of channels: $(3,64),(64,128),(128,256)$ and $(256,256)$, followed by two fully-connected layers to reduce the dimensionality to $512$ first, and finally to the number of classes. Between each convolution we added a batch-normalisation layer, non-linearity (ELU) and $2\times 2\times 2$ max-pooling. The full architecture is graphically represented in Figure 10 of the architecture detailed in the paper, for clarity.

For our approach, we used the same setting but replaced the 3D convolutions with our proposed block and used, for each layer, $6\times n_{\text{input-channels}}$ for the rank of the HO-CP convolution. The dataset was processed by batches of $32$ sequences of RGB images, with a temporal resolution of $18$ frames and a size of $84\times 84$. The loss is optimized by mini-batches of $32$ samples using stochastic gradient descent, with a starting learning-rate of $0.001$, decreased by a factor of $10$ on plateau, a weight decay of $1e^{-5}$ and momentum of $0.9$. All optimization was done on $2$ NVIDIA 1080Ti GPUs.

Results for 3D convolutional networks For the 3D case, we test our Higher-Order CP convolution with a regular 3D convolution in a simple neural network architecture, in the same setting, in order to be able to compare them. Our approach is more computationally efficient and gets better performance as shown in Table 7. In particular, the basic version without skip connection and with RELU (emphHO-CP ConvNet) has $1.7$ million less parameters in the convolutional layers compared to the regular 3D network, and yet, converges to better Top-1 and Top-5 accuracy. The version with skip-connection and PReLU (HO-CP ConvNet-S) beats all approaches.

Algorithm for our CP convolutions

We summarize our efficient higher-order factorized convolution in algorithm 1.