Where Should Diffusion Enter a Language Model? Geometry-Guided Hidden-State Replacement

TL;DR

This paper introduces DiHAL, a hybrid diffusion-transformer model that uses geometry-guided layer selection to improve language model denoising by replacing lower transformer layers with diffusion, avoiding direct token recovery.

Contribution

DiHAL proposes a novel method for integrating diffusion into pretrained transformers using geometry-based layer scoring and hidden-state reconstruction, improving diffusion effectiveness in language models.

Findings

Geometry score predicts effective diffusion layers.

Hidden-state recovery outperforms continuous diffusion baselines.

Shallow diffusion insertion improves language model denoising.

Abstract

Continuous diffusion language models lag behind autoregressive transformers, partly because diffusion is applied in spaces poorly suited to language denoising and token recovery. We propose DiHAL, a geometry-guided diffusion-transformer hybrid that asks where diffusion should enter a pretrained transformer. DiHAL scores layers with geometry-based proxies, selects a diffusion-friendly hidden-state interface, and replaces the lower transformer prefix with a diffusion bridge while retaining the upper layers and original LM head. By reconstructing the selected-layer hidden state rather than tokens, DiHAL avoids direct continuous-to-discrete recovery. Experiments on 8B-scale backbones show that the geometry score predicts effective shallow insertion layers under a fixed bridge-training protocol and that hidden-state recovery improves over continuous diffusion baselines in a diagnostic comparison matching the diffusion/recovery training budget. These results suggest that hidden-state geometry helps identify where diffusion-based replacement is feasible inside pretrained language models.