Decentralized Autoregressive Generation

TL;DR

This paper introduces a theoretical framework for decentralizing autoregressive generation, demonstrating that decentralized and centralized training can be equivalent for multimodal language models through experiments.

Contribution

It proposes the Decentralized Discrete Flow Matching objective and empirically shows the equivalence of decentralized and centralized training methods.

Findings

Decentralized autoregressive models can match centralized training performance.

Theoretical analysis supports the use of expert flows in probability generation.

Experimental results validate the proposed framework across benchmarks.

Abstract

We present a theoretical analysis of decentralization of autoregressive generation. We define the Decentralized Discrete Flow Matching objective, by expressing probability generating velocity as a linear combination of expert flows. We also conduct experiments demonstrating the equivalence between decentralized and centralized training settings for multimodal language models across diverse set of benchmarks. Specifically, we compare two distinct paradigms: LLaVA and InternVL 2.5-1B, which uses a fixed CLIP vision encoder and performs full-parameter fine-tuning (ViT+MLP+LLM) during the instruction tuning stage.