RiTeK: A Dataset for Large Language Models Complex Reasoning over Textual Knowledge Graphs in Medicine

TL;DR

RiTeK is a new dataset designed to evaluate and improve large language models' ability to perform complex reasoning over medical textual knowledge graphs, addressing current limitations in medical data and retrieval methods.

Contribution

The paper introduces RiTeK, a comprehensive medical TKG dataset with synthesized queries and a benchmark for evaluating retrieval systems in medical reasoning tasks.

Findings

Existing retrieval methods perform poorly on RiTeK benchmark.

RiTeK covers diverse topological structures and complex textual descriptions.

Expert evaluation confirms the quality of synthesized queries.

Abstract

Answering complex real-world questions in the medical domain often requires accurate retrieval from medical Textual Knowledge Graphs (medical TKGs), as the relational path information from TKGs could enhance the inference ability of Large Language Models (LLMs). However, the main bottlenecks lie in the scarcity of existing medical TKGs, the limited expressiveness of their topological structures, and the lack of comprehensive evaluations of current retrievers for medical TKGs. To address these challenges, we first develop a Dataset1 for LLMs Complex Reasoning over medical Textual Knowledge Graphs (RiTeK), covering a broad range of topological structures. Specifically, we synthesize realistic user queries integrating diverse topological structures, relational information, and complex textual descriptions. We conduct a rigorous medical expert evaluation process to assess and validate the quality of our synthesized queries. RiTeK also serves as a comprehensive benchmark dataset for evaluating the capabilities of retrieval systems built upon LLMs. By assessing 11 representative retrievers on this benchmark, we observe that existing methods struggle to perform well, revealing notable limitations in current LLM-driven retrieval approaches. These findings highlight the pressing need for more effective retrieval systems tailored for semi-structured data in the medical domain.