LACE: Lattice Attention for Cross-thread Exploration

TL;DR

LACE introduces a novel framework enabling large language models to coordinate multiple reasoning paths through cross-thread attention, significantly improving reasoning accuracy by facilitating interaction and error correction among parallel trajectories.

Contribution

The paper presents a new architecture and training pipeline that allows parallel reasoning paths to communicate and correct each other during inference.

Findings

LACE improves reasoning accuracy by over 7 points.

Cross-thread attention enables models to share insights and correct errors.

Unified exploration outperforms standard parallel search.

Abstract

Current large language models reason in isolation. Although it is common to sample multiple reasoning paths in parallel, these trajectories do not interact, and often fail in the same redundant ways. We introduce LACE, a framework that transforms reasoning from a collection of independent trials into a coordinated, parallel process. By repurposing the model architecture to enable cross-thread attention, LACE allows concurrent reasoning paths to share intermediate insights and correct one another during inference. A central challenge is the absence of natural training data that exhibits such collaborative behavior. We address this gap with a synthetic data pipeline that explicitly teaches models to communicate and error-correct across threads. Experiments show that this unified exploration substantially outperforms standard parallel search, improving reasoning accuracy by over 7 points. Our results suggest that large language models can be more effective when parallel reasoning paths are allowed to interact.