Structured Agent Distillation for Large Language Model

TL;DR

This paper introduces Structured Agent Distillation, a method to compress large language model agents into smaller, efficient models while maintaining reasoning and action quality, demonstrated across multiple benchmarks.

Contribution

It proposes a span-segmented, structure-aware distillation framework that improves the fidelity of smaller models in replicating large LLM decision processes.

Findings

Outperforms token-level and imitation baselines in experiments.

Achieves significant compression with minimal performance loss.

Span-level alignment is crucial for efficient agent deployment.

Abstract

Large language models (LLMs) exhibit strong capabilities as decision-making agents by interleaving reasoning and actions, as seen in ReAct-style frameworks. Yet, their practical deployment is constrained by high inference costs and large model sizes. We propose Structured Agent Distillation, a framework that compresses large LLM-based agents into smaller student models while preserving both reasoning fidelity and action consistency. Unlike standard token-level distillation, our method segments trajectories into {[REASON]} and {[ACT]} spans, applying segment-specific losses to align each component with the teacher's behavior. This structure-aware supervision enables compact agents to better replicate the teacher's decision process. Experiments on ALFWorld, HotPotQA-ReAct, and WebShop show that our approach consistently outperforms token-level and imitation learning baselines, achieving significant compression with minimal performance drop. Scaling and ablation results further highlight the importance of span-level alignment for efficient and deployable agents.