Action-Affect Classification and Morphing using Multi-Task Representation Learning

TL;DR

This paper introduces a multi-task learning model based on Conditional Restricted Boltzmann Machines for analyzing and generating body affect in sequences, improving classification accuracy and generative capabilities.

Contribution

It proposes a novel Multi-Task Conditional Restricted Boltzmann Machine model that enhances affect and action classification and generation from body movement data.

Findings

Superior classification performance over state-of-the-art methods

Effective generation of body affect sequences

Validated on two public datasets

Abstract

Most recent work focused on affect from facial expressions, and not as much on body. This work focuses on body affect analysis. Affect does not occur in isolation. Humans usually couple affect with an action in natural interactions; for example, a person could be talking and smiling. Recognizing body affect in sequences requires efficient algorithms to capture both the micro movements that differentiate between happy and sad and the macro variations between different actions. We depart from traditional approaches for time-series data analytics by proposing a multi-task learning model that learns a shared representation that is well-suited for action-affect classification as well as generation. For this paper we choose Conditional Restricted Boltzmann Machines to be our building block. We propose a new model that enhances the CRBM model with a factored multi-task component to become Multi-Task Conditional Restricted Boltzmann Machines (MTCRBMs). We evaluate our approach on two publicly available datasets, the Body Affect dataset and the Tower Game dataset, and show superior classification performance improvement over the state-of-the-art, as well as the generative abilities of our model.