ClawTrace: Cost-Aware Tracing for LLM Agent Skill Distillation

TL;DR

ClawTrace is a platform that records detailed per-step costs in LLM agent sessions, enabling cost-aware skill distillation and effective pruning of unnecessary expensive steps.

Contribution

The paper introduces ClawTrace, a novel tracing system that captures per-step costs and redundancy flags, facilitating cost-aware skill distillation and pruning in LLM agents.

Findings

Cost attribution and pruning reduce quality regressions in skill distillation.

Prune rules transfer across benchmarks, significantly lowering costs.

Preserve rules can cause regressions when applied to new tasks.

Abstract

Skill-distillation pipelines learn reusable rules from LLM agent trajectories, but they lack a key signal: how much each step costs. Without per-step cost, a pipeline cannot distinguish adding a missing step to fix a bug from removing an expensive step that never affected the outcome. We introduce ClawTrace, an agent tracing platform that records every LLM call, tool use, and sub-agent spawn during an agent session and compiles each session into a TraceCard: a compact YAML summary with per-step USD cost, token counts, and redundancy flags. Built on ClawTrace, CostCraft is a distillation pipeline that reads TraceCards and produces three types of skill patches. Preserve patches keep behaviors that led to success. Prune patches remove expensive steps that did not matter, each backed by a counterfactual argument against a named high-cost step. Repair patches fix failures grounded in oracle evidence. Ablations on 30 held-out SpreadsheetBench tasks show that both cost attribution and prune patches independently reduce quality regressions. When the same skill is applied to 30 unrelated SkillsBench tasks, an unexpected asymmetry emerges: prune rules transferred across benchmarks and cut median cost by 32%, while preserve rules, trained on benchmark-specific conventions, caused regressions on new task types. We release ClawTrace and TraceCards as open infrastructure for cost-aware agent research.