Robust Emotion Recognition from Low Quality and Low Bit Rate Video: A Deep Learning Approach

TL;DR

This paper introduces a deep learning framework that jointly performs super-resolution and emotion recognition on low bit rate videos, maintaining high accuracy despite significant downsampling.

Contribution

A novel max-mix training strategy enables a single robust model for diverse downsampling factors, improving emotion recognition from low quality videos.

Findings

Significantly improved recognition accuracy on AVEC 2016 benchmark.

Enhanced rate-distortion performance compared to baseline methods.

Robustness across a wide range of downsampling factors.

Abstract

Emotion recognition from facial expressions is tremendously useful, especially when coupled with smart devices and wireless multimedia applications. However, the inadequate network bandwidth often limits the spatial resolution of the transmitted video, which will heavily degrade the recognition reliability. We develop a novel framework to achieve robust emotion recognition from low bit rate video. While video frames are downsampled at the encoder side, the decoder is embedded with a deep network model for joint super-resolution (SR) and recognition. Notably, we propose a novel max-mix training strategy, leading to a single "One-for-All" model that is remarkably robust to a vast range of downsampling factors. That makes our framework well adapted for the varied bandwidths in real transmission scenarios, without hampering scalability or efficiency. The proposed framework is evaluated on the AVEC 2016 benchmark, and demonstrates significantly improved stand-alone recognition performance, as well as rate-distortion (R-D) performance, than either directly recognizing from LR frames, or separating SR and recognition.