Context Tokens are Anchors: Understanding the Repetition Curse in dMLLMs from an Information Flow Perspective

TL;DR

This paper investigates the causes of repetitive text in diffusion-based multimodal large language models by analyzing information flow, and proposes a method called CoTA to mitigate this issue and improve model performance.

Contribution

The paper introduces an information flow perspective to understand repetition in dMLLMs and presents CoTA, a novel plug-and-play approach to reduce repetition and enhance decoding quality.

Findings

Repetition is linked to disruptions in context token information flow.

Deeper layers show convergence of context token entropy, indicating prediction certainty.

CoTA effectively reduces repetition and improves task performance.

Abstract

Recent diffusion-based Multimodal Large Language Models (dMLLMs) suffer from high inference latency and therefore rely on caching techniques to accelerate decoding. However, the application of cache mechanisms often introduces undesirable repetitive text generation, a phenomenon we term the \textbf{Repeat Curse}. To better investigate underlying mechanism behind this issue, we analyze repetition generation through the lens of information flow. Our work reveals three key findings: (1) context tokens aggregate semantic information as anchors and guide the final predictions; (2) as information propagates across layers, the entropy of context tokens converges in deeper layers, reflecting the model's growing prediction certainty; (3) Repetition is typically linked to disruptions in the information flow of context tokens and to the inability of their entropy to converge in deeper layers. Based on these insights, we present \textbf{CoTA}, a plug-and-play method for mitigating repetition. CoTA enhances the attention of context tokens to preserve intrinsic information flow patterns, while introducing a penalty term to the confidence score during decoding to avoid outputs driven by uncertain context tokens. With extensive experiments, CoTA demonstrates significant effectiveness in alleviating repetition and achieves consistent performance improvements on general tasks. Code is available at https://github.com/ErikZ719/CoTA