TranSTYLer: Multimodal Behavioral Style Transfer for Facial and Body Gestures Generation

TL;DR

This paper introduces TranSTYLer, a multimodal transformer model that transfers behavioral expressivity styles across modalities like speech, gestures, and facial expressions, without requiring style labels, and outperforms existing methods in style transfer tasks.

Contribution

The paper presents a style-content disentanglement approach in a multimodal transformer that enables style transfer without style labels and generalizes to unseen styles.

Findings

Outperforms state-of-the-art in style transfer accuracy

Effective style transfer for both seen and unseen styles

Proposes a methodology to evaluate style and content preservation

Abstract

This paper addresses the challenge of transferring the behavior expressivity style of a virtual agent to another one while preserving behaviors shape as they carry communicative meaning. Behavior expressivity style is viewed here as the qualitative properties of behaviors. We propose TranSTYLer, a multimodal transformer based model that synthesizes the multimodal behaviors of a source speaker with the style of a target speaker. We assume that behavior expressivity style is encoded across various modalities of communication, including text, speech, body gestures, and facial expressions. The model employs a style and content disentanglement schema to ensure that the transferred style does not interfere with the meaning conveyed by the source behaviors. Our approach eliminates the need for style labels and allows the generalization to styles that have not been seen during the training phase. We train our model on the PATS corpus, which we extended to include dialog acts and 2D facial landmarks. Objective and subjective evaluations show that our model outperforms state of the art models in style transfer for both seen and unseen styles during training. To tackle the issues of style and content leakage that may arise, we propose a methodology to assess the degree to which behavior and gestures associated with the target style are successfully transferred, while ensuring the preservation of the ones related to the source content.