Image Sentiment Transfer

TL;DR

This paper introduces a novel framework for transferring sentiment at the object level in images, utilizing object detection, multiple references, and a specialized GAN to achieve more nuanced sentiment manipulation.

Contribution

It presents a new object-level sentiment transfer method with a Sentiment-aware GAN and a content disentanglement strategy, advancing beyond global style transfer techniques.

Findings

Effective object-level sentiment transfer demonstrated on VSO dataset.

Outperforms existing global transfer methods in qualitative and quantitative evaluations.

Framework enables nuanced sentiment manipulation for individual objects.

Abstract

In this work, we introduce an important but still unexplored research task -- image sentiment transfer. Compared with other related tasks that have been well-studied, such as image-to-image translation and image style transfer, transferring the sentiment of an image is more challenging. Given an input image, the rule to transfer the sentiment of each contained object can be completely different, making existing approaches that perform global image transfer by a single reference image inadequate to achieve satisfactory performance. In this paper, we propose an effective and flexible framework that performs image sentiment transfer at the object level. It first detects the objects and extracts their pixel-level masks, and then performs object-level sentiment transfer guided by multiple reference images for the corresponding objects. For the core object-level sentiment transfer, we propose a novel Sentiment-aware GAN (SentiGAN). Both global image-level and local object-level supervisions are imposed to train SentiGAN. More importantly, an effective content disentanglement loss cooperating with a content alignment step is applied to better disentangle the residual sentiment-related information of the input image. Extensive quantitative and qualitative experiments are performed on the object-oriented VSO dataset we create, demonstrating the effectiveness of the proposed framework.