Code-Style In-Context Learning for Knowledge-Based Question Answering

TL;DR

This paper introduces a code-style in-context learning approach for KBQA that transforms logical form generation into code generation, significantly reducing formatting errors and achieving state-of-the-art results in few-shot settings.

Contribution

The paper proposes a novel code-style ICL method for KBQA that improves logical form generation accuracy and reduces formatting errors in large language models.

Findings

Achieved new SOTA results on WebQSP, GrailQA, and GraphQ datasets.

Significantly reduced logical form formatting errors.

Effective in few-shot learning scenarios.

Abstract

Current methods for Knowledge-Based Question Answering (KBQA) usually rely on complex training techniques and model frameworks, leading to many limitations in practical applications. Recently, the emergence of In-Context Learning (ICL) capabilities in Large Language Models (LLMs) provides a simple and training-free semantic parsing paradigm for KBQA: Given a small number of questions and their labeled logical forms as demo examples, LLMs can understand the task intent and generate the logic form for a new question. However, current powerful LLMs have little exposure to logic forms during pre-training, resulting in a high format error rate. To solve this problem, we propose a code-style in-context learning method for KBQA, which converts the generation process of unfamiliar logical form into the more familiar code generation process for LLMs. Experimental results on three mainstream datasets show that our method dramatically mitigated the formatting error problem in generating logic forms while realizing a new SOTA on WebQSP, GrailQA, and GraphQ under the few-shot setting. The code and supplementary files are released at https://github.com/Arthurizijar/KB-Coder .