Style Memory: Making a Classifier Network Generative

TL;DR

This paper introduces a neural network with a style memory that enables both accurate classification and input reconstruction, enhancing generative capabilities and robustness against ambiguous or adversarial inputs.

Contribution

The authors propose a novel style memory mechanism integrated into classifier networks, allowing simultaneous classification and reconstruction, and explore its implications for bidirectional information flow.

Findings

The network can reconstruct inputs accurately when classification is correct.

Style memory captures stylistic features related to digits and letters.

Bidirectional architecture may improve robustness against adversarial attacks.

Abstract

Deep networks have shown great performance in classification tasks. However, the parameters learned by the classifier networks usually discard stylistic information of the input, in favour of information strictly relevant to classification. We introduce a network that has the capacity to do both classification and reconstruction by adding a "style memory" to the output layer of the network. We also show how to train such a neural network as a deep multi-layer autoencoder, jointly minimizing both classification and reconstruction losses. The generative capacity of our network demonstrates that the combination of style-memory neurons with the classifier neurons yield good reconstructions of the inputs when the classification is correct. We further investigate the nature of the style memory, and how it relates to composing digits and letters. Finally, we propose that this architecture enables the bidirectional flow of information used in predictive coding, and that such bidirectional networks can help mitigate against being fooled by ambiguous or adversarial input.