Generative Flows with Invertible Attentions

TL;DR

This paper introduces invertible attention mechanisms for flow-based generative models, enabling efficient long-range dependency modeling and improving image synthesis performance compared to existing methods.

Contribution

It proposes two novel invertible attention modules, map-based and transformer-based, with a masked scheme for tractable Jacobian computation in generative flows.

Findings

Achieved better image synthesis quality than state-of-the-art models.

Demonstrated efficient long-range dependency modeling in flow-based generative models.

Validated effectiveness on multiple image synthesis tasks.

Abstract

Flow-based generative models have shown an excellent ability to explicitly learn the probability density function of data via a sequence of invertible transformations. Yet, learning attentions in generative flows remains understudied, while it has made breakthroughs in other domains. To fill the gap, this paper introduces two types of invertible attention mechanisms, i.e., map-based and transformer-based attentions, for both unconditional and conditional generative flows. The key idea is to exploit a masked scheme of these two attentions to learn long-range data dependencies in the context of generative flows. The masked scheme allows for invertible attention modules with tractable Jacobian determinants, enabling its seamless integration at any positions of the flow-based models. The proposed attention mechanisms lead to more efficient generative flows, due to their capability of modeling the long-term data dependencies. Evaluation on multiple image synthesis tasks shows that the proposed attention flows result in efficient models and compare favorably against the state-of-the-art unconditional and conditional generative flows.