Invertible Memory Flow Networks

TL;DR

The paper introduces Invertible Memory Flow Networks, a novel approach for long sequence compression that decomposes the task into simpler pairwise merges, enabling efficient and scalable sequence modeling.

Contribution

It proposes a new invertible network architecture that decomposes sequence compression into pairwise merges using a binary tree structure, improving scalability and efficiency.

Findings

Validated on long MNIST sequences

Achieved effective high-dimensional data compression

Demonstrated constant-cost inference with distilled models

Abstract

Long sequence neural memory remains a challenging problem. RNNs and their variants suffer from vanishing gradients, and Transformers suffer from quadratic scaling. Furthermore, compressing long sequences into a finite fixed representation remains an intractable problem due to the difficult optimization landscape. Invertible Memory Flow Networks (IMFN) make long sequence compression tractable through factorization: instead of learning end-to-end compression, we decompose the problem into pairwise merges using a binary tree of "sweeper" modules. Rather than learning to compress long sequences, each sweeper learns a much simpler 2-to-1 compression task, achieving O(log N) depth with sublinear error accumulation in sequence length. For online inference, we distilled into a constant-cost recurrent student achieving O(1) sequential steps. Empirical results validate IMFN on long MNIST sequences and UCF-101 videos, demonstrating compression of high-dimensional data over long sequences.