Resolution enhancement in the recovery of underdrawings via style transfer by generative adversarial deep neural networks

TL;DR

This paper introduces a deep neural network approach using generative adversarial networks to enhance the resolution of style-transferred underdrawings in art x-ray images, improving detail recovery in fine art analysis.

Contribution

It develops a hierarchical, coarse-to-fine neural architecture that significantly increases resolution and incorporates minimal human segmentation data to improve style transfer quality.

Findings

Achieves up to 64-fold increase in pixel resolution.

Qualitative and quantitative improvements with minimal segmentation data.

Successfully applied to historical artworks like Leonardo da Vinci's paintings.

Abstract

We apply generative adversarial convolutional neural networks to the problem of style transfer to underdrawings and ghost-images in x-rays of fine art paintings with a special focus on enhancing their spatial resolution. We build upon a neural architecture developed for the related problem of synthesizing high-resolution photo-realistic image from semantic label maps. Our neural architecture achieves high resolution through a hierarchy of generators and discriminator sub-networks, working throughout a range of spatial resolutions. This coarse-to-fine generator architecture can increase the effective resolution by a factor of eight in each spatial direction, or an overall increase in number of pixels by a factor of 64. We also show that even just a few examples of human-generated image segmentations can greatly improve -- qualitatively and quantitatively -- the generated images. We demonstrate our method on works such as Leonardo's Madonna of the carnation and the underdrawing in his Virgin of the rocks, which pose several special problems in style transfer, including the paucity of representative works from which to learn and transfer style information.