Altering Facial Expression Based on Textual Emotion

TL;DR

This paper presents a method combining GANs and LSTM to alter facial expressions in images based on textual emotion, enabling dynamic expression changes driven by text input.

Contribution

It introduces a novel pipeline that integrates emotion extraction from text with facial expression synthesis using GANs, extending existing image editing techniques.

Findings

Successfully altered facial expressions based on textual emotion.

Demonstrated a prototype application for dynamic profile picture editing.

Extended StarGAN with text-driven emotion control.

Abstract

Faces and their expressions are one of the potent subjects for digital images. Detecting emotions from images is an ancient task in the field of computer vision; however, performing its reverse -- synthesizing facial expressions from images -- is quite new. Such operations of regenerating images with different facial expressions, or altering an existing expression in an image require the Generative Adversarial Network (GAN). In this paper, we aim to change the facial expression in an image using GAN, where the input image with an initial expression (i.e., happy) is altered to a different expression (i.e., disgusted) for the same person. We used StarGAN techniques on a modified version of the MUG dataset to accomplish this objective. Moreover, we extended our work further by remodeling facial expressions in an image indicated by the emotion from a given text. As a result, we applied a Long Short-Term Memory (LSTM) method to extract emotion from the text and forwarded it to our expression-altering module. As a demonstration of our working pipeline, we also create an application prototype of a blog that regenerates the profile picture with different expressions based on the user's textual emotion.