Graph-of-Causal Evolution: Challenging Chain-of-Model for Reasoning

TL;DR

This paper introduces Graph-of-Causal Evolution (GoCE), a novel method that enhances transformer models by explicitly modeling long-range causal dependencies and enabling self-evolution, outperforming traditional chain-of-model approaches.

Contribution

The work proposes a new graph-based causal structure learning method for transformers, improving long-range dependency capture and adaptive self-evolution capabilities.

Findings

GoCE improves transformer ability to capture long-range causal dependencies.

GoCE enhances the self-evolution capability of transformer models.

Experimental results show GoCE surpasses baseline models on multiple datasets.

Abstract

In view of the problem that each subchain in the chain-of-model (CoM) relies only on the information of the previous subchain and may lose long-range dependencies due to the causal mask blocking the global context flow between multi-level subchains, this work proposes a graph of causal evolution (GoCE). Its core principle is to map the implicit token representation into a differentiable and sparse causal adjacency matrix, then permeate causal constraints through each layer of calculation using causal-masked attention and causal-MoE. By combining intervention consistency loss test and self-evolution gate, the dynamic balance between causal structure learning and adaptive updating of transformer architecture is realized. The researcher built experimental environments in sandboxes built with Claude Sonnet 4, o4-mini-high, and DeepSeek R1 respectively with the transformer variant architecture introduced in GoCE. It is evaluated on publicly available datasets including CLUTRR, CLADDER, EX-FEVER, and CausalQA and compared with the baseline LLMs. The finding proves that GoCE strengthens the transformer's ability to capture long-range causal dependencies, while the ability to self-evolve is improved. It not only surpasses the design of CoM in terms of design principles, but also provides experience for future research on causal learning and continuous adaptive improvement.