Training on Art Composition Attributes to Influence CycleGAN Art Generation

TL;DR

This paper introduces a method to guide CycleGAN image translation by integrating an Art Composition Attributes Network trained on art evaluation rules, enhancing control over generated art images.

Contribution

The paper presents a novel approach of using an auxiliary neural network trained on art composition attributes to influence CycleGAN outputs.

Findings

ACAN effectively encodes art composition attributes.

Incorporating ACAN improves control over CycleGAN translation.

Method leverages art domain knowledge for better image synthesis.

Abstract

I consider how to influence CycleGAN, image-to-image translation, by using additional constraints from a neural network trained on art composition attributes. I show how I trained the the Art Composition Attributes Network (ACAN) by incorporating domain knowledge based on the rules of art evaluation and the result of applying each art composition attribute to apple2orange image translation.

Full text

1 Introduction

The standard adversarial and cyclical losses of a CycleGAN [1] were augmented with additional loss terms from a convolutional neural network trained with art composition attributes. During training of the CycleGAN, the user specifies values for each of the art composition attributes. For instance, if a target contrast value of 10 is specified, the generator should output images with more contrast than if the target contrast value is 1.

1.1 Art Composition Attributes

Eight art composition attributes were selected: variety of texture, variety of shape, variety of size, variety of color, contrast, repetition, primary color, and color harmony. Five hundred images from the WikiArt dataset [2] were labeled with these attributes. Figures 1, 2, 3 and 4 are examples of low and high values for variety of texture and contrast.

2 ACAN

Training consisted of fine-tuning a ResNet50 [3] pretrained on the ImageNet dataset. ResNet50 is a fifty-layer deep residual network with 16 residual blocks. Global Average Pooling (GAP) is applied to the ReLU output from each of the sixteen ResNet block activations, called rectified convolution maps [4]. The sixteen GAP outputs were concatenated and L2 normalization was applied to create a merge layer. From the merge layer, there are eight outputs, one for each of the attributes.

3 CycleGAN and ACAN

In addition to the standard CycleGAN losses (Adversarial, Cycle-Consistency, and Identity) the ACAN losses are a series of eight losses generated when the translated image is passed through the ACAN with eight target values. The difference between these target values and the values output by the network are the attribute losses.

4 Results

Below is a sampling of the results of running the CycleGAN training with ACAN on the apple2orange dataset. Even with a small training set size of 500 images, the ACAN is able to learn and generate apples with the eight art compositional attributes.

The most surprising result of this project is the painterly effect that the ACAN was able to inject into the CycleGAN generated images as seen in Figures 18 and 19.

Acknowledgments

This project was initially developed as part of the 2018 OpenAI Scholars program. I would like to thank my mentor, Christy Dennison from OpenAI, for her helpful comments along with support from Larissa Schiavo, Joshua Achiam, Jack Clark, and Greg Brockman from OpenAI.