Understanding VAEs in Fisher-Shannon Plane

TL;DR

This paper explores the relationship between Fisher information and Shannon entropy in VAEs, introducing a Fisher auto-encoder that balances these information measures to improve reconstruction and latent code quality.

Contribution

It investigates VAEs in the Fisher-Shannon plane, revealing their intrinsic connection to these information quantities and proposing a new Fisher auto-encoder variant.

Findings

Fisher auto-encoder improves reconstruction accuracy

Balances Fisher and Shannon information effectively

Reduces non-informative latent codes

Abstract

In information theory, Fisher information and Shannon information (entropy) are respectively used to quantify the uncertainty associated with the distribution modeling and the uncertainty in specifying the outcome of given variables. These two quantities are complementary and are jointly applied to information behavior analysis in most cases. The uncertainty property in information asserts a fundamental trade-off between Fisher information and Shannon information, which enlightens us the relationship between the encoder and the decoder in variational auto-encoders (VAEs). In this paper, we investigate VAEs in the Fisher-Shannon plane and demonstrate that the representation learning and the log-likelihood estimation are intrinsically related to these two information quantities. Through extensive qualitative and quantitative experiments, we provide with a better comprehension of VAEs in tasks such as high-resolution reconstruction, and representation learning in the perspective of Fisher information and Shannon information. We further propose a variant of VAEs, termed as Fisher auto-encoder (FAE), for practical needs to balance Fisher information and Shannon information. Our experimental results have demonstrated its promise in improving the reconstruction accuracy and avoiding the non-informative latent code as occurred in previous works.