FlowNIB: An Information Bottleneck Analysis of Bidirectional vs. Unidirectional Language Models

TL;DR

This paper uses the Information Bottleneck principle and a new method, FlowNIB, to explain why bidirectional language models outperform unidirectional ones in understanding tasks, showing they retain more task-relevant information.

Contribution

The paper introduces FlowNIB, a scalable method for estimating mutual information during training, and provides a theoretical framework demonstrating bidirectional models retain more information and are more expressive.

Findings

Bidirectional models retain more mutual information.

Bidirectional models have higher effective dimensionality.

FlowNIB effectively analyzes information flow during training.

Abstract

Bidirectional language models have better context understanding and perform better than unidirectional models on natural language understanding tasks, yet the theoretical reasons behind this advantage remain unclear. In this work, we investigate this disparity through the lens of the Information Bottleneck (IB) principle, which formalizes a trade-off between compressing input information and preserving task-relevant content. We propose FlowNIB, a dynamic and scalable method for estimating mutual information during training that addresses key limitations of classical IB approaches, including computational intractability and fixed trade-off schedules. Theoretically, we show that bidirectional models retain more mutual information and exhibit higher effective dimensionality than unidirectional models. To support this, we present a generalized framework for measuring representational complexity and prove that bidirectional representations are strictly more informative under mild conditions. We further validate our findings through extensive experiments across multiple models and tasks using FlowNIB, revealing how information is encoded and compressed throughout training. Together, our work provides a principled explanation for the effectiveness of bidirectional architectures and introduces a practical tool for analyzing information flow in deep language models.