KnowCoder-A1: Incentivizing Agentic Reasoning Capability with Outcome Supervision for KBQA

TL;DR

KnowCoder-A1 is an LLM designed for autonomous agentic reasoning in KBQA, trained with outcome-only supervision and curriculum reinforcement learning, achieving superior zero-shot performance with less data.

Contribution

It introduces outcome-only supervision and multi-stage curriculum RL to enhance agentic reasoning in LLMs for KBQA tasks, reducing reliance on process supervision.

Findings

Outperforms prior methods on three KBQA datasets.

Achieves up to 11.1% relative improvement on GrailQA zero-shot subset.

Uses only one-twelfth of the training data compared to baselines.

Abstract

Knowledge Base Question Answering (KBQA) aims to answer natural-language questions over a structured Knowledge Base (KB). Recent work improves KBQA by adopting an agentic reasoning paradigm, in which Large Language Models (LLMs) iteratively decompose a question, generate its corresponding logical queries, and interact with the KB to derive the answer. However, these methods typically fine-tune LLMs on reasoning trajectories synthesized via process supervision, which offers weak incentives for exploration and thus fails to strengthen the agentic reasoning ability. In this paper, we propose KnowCoder-A1, an LLM that can autonomously perform agentic reasoning on KBs to obtain answers. To incentivize autonomous exploration, KnowCoder-A1 trains the LLM under outcome-only supervision via a multi-stage curriculum reinforcement learning with an easy-to-hard curriculum. To establish foundational agentic capabilities, KnowCoder-A1 first fine-tunes the LLM on a small set of high-quality trajectories obtained through outcome-based rejection sampling. Then, to alleviate the reward sparsity inherent in outcome-only supervision, it applies multi-stage curriculum RL with reward schedules that progress from easy to hard. Trained with outcome-only supervision, KnowCoder-A1 exhibits powerful reasoning behaviors and consistently outperforms prior approaches across three mainstream datasets. Notably, on the zero-shot subset of GrailQA, KnowCoder-A1 achieves up to an 11.1% relative improvement while using only one-twelfth of the training data, demonstrating strong agentic reasoning capabilities.