SMAGDi: Socratic Multi Agent Interaction Graph Distillation for Efficient High Accuracy Reasoning

TL;DR

SMAGDi is a novel distillation framework that compresses multi-agent debate systems into smaller, efficient models by modeling interaction graphs, achieving high accuracy retention on reasoning benchmarks.

Contribution

Introduces SMAGDi, a graph-based distillation method that captures debate dynamics for efficient high-accuracy reasoning in smaller models.

Findings

Retains 88% of multi-agent system accuracy in a 6B student model.

Outperforms prior distillation methods like MAGDi and KD.

Enables real-world deployment of high-accuracy reasoning models.

Abstract

Multi-agent systems (MAS) often achieve higher reasoning accuracy than single models, but their reliance on repeated debates across agents makes them computationally expensive. We introduce SMAGDi, a distillation framework that transfers the debate dynamics of a five-agent Llama-based MAS into a compact Socratic decomposer-solver student. SMAGDi represents debate traces as directed interaction graphs, where nodes encode intermediate reasoning steps with correctness labels and edges capture continuity and cross-agent influence. The student is trained with a composite objective combining language modeling, graph-based supervision, contrastive reasoning, and embedding alignment to preserve both fluency and structured reasoning. On StrategyQA and MMLU, SMAGDi compresses a 40B multi-agent system into a 6B student while retaining 88% of its accuracy, substantially outperforming prior distillation methods such as MAGDi, standard KD, and fine-tuned baselines. These results highlight that explicitly modeling interaction graphs and Socratic decomposition enable small models to inherit the accuracy benefits of multi-agent debate while remaining efficient enough for real-world deployment.