Neural Collage Transfer: Artistic Reconstruction via Material Manipulation

TL;DR

This paper introduces a reinforcement learning-based method for creating artistic collages by intelligently selecting and arranging diverse materials to reconstruct target images, emphasizing aesthetic quality and style fidelity.

Contribution

It presents a novel reinforcement learning framework with a model-based soft actor-critic for collage creation, addressing the lack of data and complexity in the process.

Findings

The method effectively reconstructs target images as collages with high content and style scores.

The approach outperforms pixel-wise generation methods in aesthetic evaluation.

Active material selection improves collage quality and visual appeal.

Abstract

Collage is a creative art form that uses diverse material scraps as a base unit to compose a single image. Although pixel-wise generation techniques can reproduce a target image in collage style, it is not a suitable method due to the solid stroke-by-stroke nature of the collage form. While some previous works for stroke-based rendering produced decent sketches and paintings, collages have received much less attention in research despite their popularity as a style. In this paper, we propose a method for learning to make collages via reinforcement learning without the need for demonstrations or collage artwork data. We design the collage Markov Decision Process (MDP), which allows the agent to handle various materials and propose a model-based soft actor-critic to mitigate the agent's training burden derived from the sophisticated dynamics of collage. Moreover, we devise additional techniques such as active material selection and complexity-based multi-scale collage to handle target images at any size and enhance the results' aesthetics by placing relatively more scraps in areas of high complexity. Experimental results show that the trained agent appropriately selected and pasted materials to regenerate the target image into a collage and obtained a higher evaluation score on content and style than pixel-wise generation methods. Code is available at https://github.com/northadventure/CollageRL.