AtomR: Atomic Operator-Empowered Large Language Models for Heterogeneous Knowledge Reasoning

TL;DR

AtomR introduces atomic knowledge operators and a fine-grained reasoning framework for large language models, significantly improving heterogeneous knowledge reasoning accuracy on challenging benchmarks.

Contribution

The paper proposes AtomR, a novel framework with atomic operators for precise heterogeneous knowledge reasoning in LLMs, addressing prior limitations in reasoning planning and knowledge integration.

Findings

AtomR outperforms baselines with 9.4% and 9.5% F1 score improvements.

Introduces BlendQA, a new benchmark for heterogeneous knowledge reasoning.

Demonstrates effective question decomposition and knowledge manipulation.

Abstract

Despite the outstanding capabilities of large language models (LLMs), knowledge-intensive reasoning still remains a challenging task due to LLMs' limitations in compositional reasoning and the hallucination problem. A prevalent solution is to employ chain-of-thought (CoT) with retrieval-augmented generation (RAG), which first formulates a reasoning plan by decomposing complex questions into simpler sub-questions, and then applies iterative RAG at each sub-question. However, prior works exhibit two crucial problems: inadequate reasoning planning and poor incorporation of heterogeneous knowledge. In this paper, we introduce AtomR, a framework for LLMs to conduct accurate heterogeneous knowledge reasoning at the atomic level. Inspired by how knowledge graph query languages model compositional reasoning through combining predefined operations, we propose three atomic knowledge operators, a unified set of operators for LLMs to retrieve and manipulate knowledge from heterogeneous sources. First, in the reasoning planning stage, AtomR decomposes a complex question into a reasoning tree where each leaf node corresponds to an atomic knowledge operator, achieving question decomposition that is highly fine-grained and orthogonal. Subsequently, in the reasoning execution stage, AtomR executes each atomic knowledge operator, which flexibly selects, retrieves, and operates atomic level knowledge from heterogeneous sources. We also introduce BlendQA, a challenging benchmark specially tailored for heterogeneous knowledge reasoning. Experiments on three single-source and two multi-source datasets show that AtomR outperforms state-of-the-art baselines by a large margin, with F1 score improvements of 9.4% on 2WikiMultihop and 9.5% on BlendQA. We release our code and datasets.