Deep Multi-class Adversarial Specularity Removal

TL;DR

This paper introduces a novel CNN-based method for removing specular highlights from single images by generating diffuse components, utilizing a multi-class discriminator within a GAN framework to improve accuracy and generalization.

Contribution

The paper presents a new multi-class discriminator GAN architecture for specularity removal and a synthetic dataset to enhance model generalization.

Findings

Outperforms existing methods in consistency across synthetic and real images.

Effectively isolates diffuse components from specular highlights.

Demonstrates robustness and improved accuracy in specularity removal.

Abstract

We propose a novel learning approach, in the form of a fully-convolutional neural network (CNN), which automatically and consistently removes specular highlights from a single image by generating its diffuse component. To train the generative network, we define an adversarial loss on a discriminative network as in the GAN framework and combined it with a content loss. In contrast to existing GAN approaches, we implemented the discriminator to be a multi-class classifier instead of a binary one, to find more constraining features. This helps the network pinpoint the diffuse manifold by providing two more gradient terms. We also rendered a synthetic dataset designed to help the network generalize well. We show that our model performs well across various synthetic and real images and outperforms the state-of-the-art in consistency.