Improved Image Generation via Sparse Modeling

TL;DR

This paper interprets GAN generators as convolutional sparse coding models, and shows that enforcing sparsity improves image synthesis quality, providing a new understanding and design principle for generative models.

Contribution

It introduces a sparsity-based perspective on generators, explicitly enforces sparsity regularization, and demonstrates improved image synthesis and inverse problem solutions.

Findings

Enhanced image quality with sparsity regularization

Applicable to GANs and Deep Image Prior methods

Provides a new theoretical understanding of generator mechanisms

Abstract

The interest of the deep learning community in image synthesis has grown massively in recent years. Nowadays, deep generative methods, and especially Generative Adversarial Networks (GANs), are leading to state-of-the-art performance, capable of synthesizing images that appear realistic. While the efforts for improving the quality of the generated images are extensive, most attempts still consider the generator part as an uncorroborated "black-box". In this paper, we aim to provide a better understanding and design of the image generation process. We interpret existing generators as implicitly relying on sparsity-inspired models. More specifically, we show that generators can be viewed as manifestations of the Convolutional Sparse Coding (CSC) and its Multi-Layered version (ML-CSC) synthesis processes. We leverage this observation by explicitly enforcing a sparsifying regularization on appropriately chosen activation layers in the generator, and demonstrate that this leads to improved image synthesis. Furthermore, we show that the same rationale and benefits apply to generators serving inverse problems, demonstrated on the Deep Image Prior (DIP) method.