In-IDE Toolkit for Developers of AI-Based Features
Yaroslav Sokolov, Yury Khudyakov, Lenar Sharipov, Andrei Gasparian, Parth Tiwary, Artem Trofimov

TL;DR
The paper introduces an IDE plugin that integrates AI model evaluation and debugging tools directly into the development environment, making AI feature development more accessible and disciplined for software engineers.
Contribution
It presents a novel IDE-native workflow for AI evaluation and debugging, including trace capture, inspection, and dataset management, tailored for non-ML specialists.
Findings
Early signals of promising adoption and usage patterns.
The plugin lowers barriers to AI observability within IDEs.
Initial telemetry suggests increased developer engagement.
Abstract
AI-enabled features built on LLMs and agentic workflows are difficult to test, debug, and reproduce, especially for product-focused software engineers without a machine learning background. We present the AI Toolkit plugin for JetBrains IDEs, which brings tracing and evaluation directly into the Run/Debug loop. A mixed methods study with practitioners presents three consistent needs: (1) make evaluation regular and repeatable, (2) expose traces at the moment of execution, and (3) minimize setup and context switching. Guided by these needs, the AI Toolkit introduces an IDE-native workflow: run-triggered trace capture; immediate, hierarchical inspection; one-click "Add to Dataset" from traces; and unit-test-like evaluations with pluggable metrics. The first release in PyCharm shows promising early signals - strong conversion when promoted at Run, sustained usage among those who capture…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
