What Do Agents Learn from Trajectory-SFT: Semantics or Interfaces?

TL;DR

This paper introduces PIPE, a protocol augmentation to diagnose whether language model agents rely on semantics or interface-specific patterns, revealing that trajectory-SFT often leads to interface shortcutting, especially under minimal interface rewrites.

Contribution

The paper proposes PIPE for diagnosing interface reliance and introduces Interface Reliance (IR), highlighting the impact of trajectory-SFT on interface shortcutting and environment-dependent training dynamics.

Findings

Trajectory-SFT amplifies interface shortcutting.

Agents degrade under minimal interface rewrites.

Interface reliance varies with environment and training dynamics.

Abstract

Large language models are increasingly evaluated as interactive agents, yet standard agent benchmarks conflate two qualitatively distinct sources of success: semantic tool-use and interface-specific interaction pattern memorization. Because both mechanisms can yield identical task success on the original interface, benchmark scores alone are not identifiable evidence of environment-invariant capability. We propose PIPE, a protocol-level evaluation augmentation for diagnosing interface reliance by minimally rewriting environment interfaces while preserving task semantics and execution behavior. Across 16 environments from AgentBench and AgentGym and a range of open-source and API-based agents, PIPE reveals that trajectory-SFT substantially amplifies interface shortcutting: trained agents degrade sharply under minimal interface rewrites, while non-trajectory-trained models remain largely stable. We further introduce Interface Reliance (IR), a counterbalanced alias-based metric that quantifies preference for training-time interfaces, and show that interface shortcutting exhibits environment-dependent, non-monotonic training dynamics that remain invisible under standard evaluation. Our code is available at https://anonymous.4open.science/r/What-Do-Agents-Learn-from-Trajectory-SFT-Semantics-or-Interfaces--0831/.