What Challenges Do Developers Face in AI Agent Systems? An Empirical Study on Stack Overflow & GitHub Issues

TL;DR

This study investigates the main challenges faced by developers in AI agent systems by analyzing discussions on Stack Overflow and GitHub issues, revealing key areas of difficulty and maintenance burdens.

Contribution

It provides a comprehensive taxonomy of developer challenges in AI agent systems based on empirical analysis of developer discussions and issue reports.

Findings

Seven Stack Overflow topics with 28 subtopics identified.

Thirteen GitHub issue themes categorized into five major families.

Retrieval and orchestration challenges are complex and persist as ongoing issues.

Abstract

AI Agents have rapidly gained prominence in both research and industry as systems that extend large language models with planning, tool use, memory, and goal-directed action. Despite this progress, the development and maintenance of Agent systems present recurring engineering difficulties that are not yet well characterized in developer-facing evidence. To address this gap, this study analyzes developer discussions on Stack Overflow and failure reports from GitHub issue trackers associated with widely used Agent frameworks. For Stack Overflow, an Agent-focused corpus is constructed through tag expansion and filtering, latent themes are derived using LDA-MALLET, and topics are manually validated and labeled. For GitHub, a taxonomy of issue themes is developed to capture deployment-time failures and maintenance burdens. Analysis across both platforms identifies seven Stack Overflow topics (comprising 28 subtopics) and thirteen GitHub issue topics, which are synthesized into five overarching families of major Agent challenges: (1) environment, platforms, and dependency management; (2) retrieval, embeddings, and Agent memory; (3) orchestration and execution control; (4) interaction contracts between models and tools; and (5) runtime reliability and operational robustness. Topic popularity and difficulty are quantified, revealing that widely discussed issues, such as installation and prompting, are often resolved more quickly, whereas retrieval- and orchestration-related challenges are less visible, more complex, and tend to persist as ongoing maintenance burdens on GitHub.