Agentic Jackal: Live Execution and Semantic Value Grounding for Text-to-JQL
Vishnu Murali, Anmol Gulati, Elias Lumer, Kevin Frank, Sindy Campagna, Vamse Kumar Subbiah

TL;DR
This paper introduces Jackal, a large-scale benchmark for translating natural language to Jira Query Language (JQL), and proposes Agentic Jackal, an agent that improves translation accuracy by executing queries live and resolving ambiguities.
Contribution
The paper presents the first execution-based benchmark for NL to JQL translation and introduces an agent that significantly enhances LLM performance through live query execution and semantic retrieval.
Findings
Single-pass LLMs average 43.4% accuracy on JQL translation.
Agentic approach improves 7 of 9 models, with a 9.0% relative gain.
Categorical-value accuracy increases from 48.7% to 71.7% with the agent.
Abstract
Translating natural language into Jira Query Language (JQL) requires resolving ambiguous field references, instance-specific categorical values, and complex Boolean predicates. Single-pass LLMs cannot discover which categorical values (e.g., component names or fix versions) actually exist in a given Jira instance, nor can they verify generated queries against a live data source, limiting accuracy on paraphrased or ambiguous requests. No open, execution-based benchmark exists for mapping natural language to JQL. We introduce Jackal, the first large-scale, execution-based text-to-JQL benchmark comprising 100,000 validated NL-JQL pairs on a live Jira instance with over 200,000 issues. To establish baselines on Jackal, we propose Agentic Jackal, a tool-augmented agent that equips LLMs with live query execution via the Jira MCP server and JiraAnchor, a semantic retrieval tool that resolves…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
