KnowCoder: Coding Structured Knowledge into LLMs for Universal Information Extraction

TL;DR

KnowCoder introduces a unified code-style schema representation and a two-phase learning framework for large language models to improve universal information extraction, demonstrating significant gains in few-shot, zero-shot, and supervised settings.

Contribution

The paper presents a novel code-style schema representation and a large, comprehensive schema library, along with a two-phase training framework, enabling LLMs to better understand and follow schemas for UIE.

Findings

Achieves 49.8% relative F1 improvement over LLaMA2 in few-shot settings.

Attains up to 12.5% and 21.9% improvements over SOTA in zero-shot and low-resource settings.

Enhances performance by 7.5% in supervised learning with human-annotated datasets.

Abstract

In this paper, we propose KnowCoder, a Large Language Model (LLM) to conduct Universal Information Extraction (UIE) via code generation. KnowCoder aims to develop a kind of unified schema representation that LLMs can easily understand and an effective learning framework that encourages LLMs to follow schemas and extract structured knowledge accurately. To achieve these, KnowCoder introduces a code-style schema representation method to uniformly transform different schemas into Python classes, with which complex schema information, such as constraints among tasks in UIE, can be captured in an LLM-friendly manner. We further construct a code-style schema library covering over $\textbf{30,000}$ types of knowledge, which is the largest one for UIE, to the best of our knowledge. To ease the learning process of LLMs, KnowCoder contains a two-phase learning framework that enhances its schema understanding ability via code pretraining and its schema following ability via instruction tuning. After code pretraining on around $1.5$B automatically constructed data, KnowCoder already attains remarkable generalization ability and achieves relative improvements by $\textbf{49.8%}$ F1, compared to LLaMA2, under the few-shot setting. After instruction tuning, KnowCoder further exhibits strong generalization ability on unseen schemas and achieves up to $\textbf{12.5%}$ and $\textbf{21.9%}$, compared to sota baselines, under the zero-shot setting and the low resource setting, respectively. Additionally, based on our unified schema representations, various human-annotated datasets can simultaneously be utilized to refine KnowCoder, which achieves significant improvements up to $\textbf{7.5%}$ under the supervised setting.