Agent Mentor: Framing Agent Knowledge through Semantic Trajectory Analysis

TL;DR

This paper presents an open-source analytics pipeline that monitors and adaptively corrects AI agent behavior by analyzing semantic features in execution logs, improving performance amid prompt ambiguity.

Contribution

It introduces the Agent Mentor library that systematically identifies semantic issues and injects corrective instructions to enhance agent performance.

Findings

Consistent accuracy improvements across diverse agent configurations.

Effective in settings with high prompt ambiguity.

Automates agent behavior correction through semantic analysis.

Abstract

AI agent development relies heavily on natural language prompting to define agents' tasks, knowledge, and goals. These prompts are interpreted by Large Language Models (LLMs), which govern agent behavior. Consequently, agentic performance is susceptible to variability arising from imprecise or ambiguous prompt formulations. Identifying and correcting such issues requires examining not only the agent's code, but also the internal system prompts generated throughout its execution lifecycle, as reflected in execution logs. In this work, we introduce an analytics pipeline implemented as part of the Agent Mentor open-source library that monitors and incrementally adapts the system prompts defining another agent's behavior. The pipeline improves performance by systematically injecting corrective instructions into the agent's knowledge. We describe its underlying mechanism, with particular emphasis on identifying semantic features associated with undesired behaviors and using them to derive corrective statements. We evaluate the proposed pipeline across three exemplar agent configurations and benchmark tasks using repeated execution runs to assess effectiveness. These experiments provide an initial exploration of automating such a mentoring pipeline within future agentic governance frameworks. Overall, the approach demonstrates consistent and measurable accuracy improvements across diverse configurations, particularly in settings dominated by specification ambiguity. For reproducibility, we released our code as open source under the Agent Mentor library.