Unsupervised Cycle Detection in Agentic Applications

TL;DR

This paper introduces an unsupervised framework combining structural and semantic analysis to detect hidden execution cycles in agentic applications powered by large language models, improving resource efficiency detection.

Contribution

The paper presents a novel hybrid cycle detection method that outperforms individual structural and semantic approaches in identifying costly hidden cycles.

Findings

Achieved an F1 score of 0.72 on stock market application trajectories.

Significantly outperformed structural (F1: 0.08) and semantic (F1: 0.28) methods.

Demonstrated the effectiveness of combining structural and semantic analysis for cycle detection.

Abstract

Agentic applications powered by Large Language Models exhibit non-deterministic behaviors that can form hidden execution cycles, silently consuming resources without triggering explicit errors. Traditional observability platforms fail to detect these costly inefficiencies. We present an unsupervised cycle detection framework that combines structural and semantic analysis. Our approach first applies computationally efficient temporal call stack analysis to identify explicit loops and then leverages semantic similarity analysis to uncover subtle cycles characterized by redundant content generation. Evaluated on 1575 trajectories from a LangGraph-based stock market application, our hybrid approach achieves an F1 score of 0.72 (precision: 0.62, recall: 0.86), significantly outperforming individual structural (F1: 0.08) and semantic methods (F1: 0.28). While these results are encouraging, there remains substantial scope for improvement, and future work is needed to refine the approach and address its current limitations.