Are Large Language Models a Good Replacement of Taxonomies?

TL;DR

This paper introduces TaxoGlimpse, a new benchmark to evaluate how well large language models understand and handle various taxonomies, revealing their limitations especially in specialized domains and detailed levels.

Contribution

The paper presents TaxoGlimpse, a comprehensive benchmark for assessing LLMs across diverse taxonomies and levels, highlighting their current shortcomings in specialized knowledge.

Findings

LLMs perform poorly on specialized taxonomies.

LLMs struggle with leaf-level entities.

Benchmark reveals gaps in LLM knowledge

Abstract

Large language models (LLMs) demonstrate an impressive ability to internalize knowledge and answer natural language questions. Although previous studies validate that LLMs perform well on general knowledge while presenting poor performance on long-tail nuanced knowledge, the community is still doubtful about whether the traditional knowledge graphs should be replaced by LLMs. In this paper, we ask if the schema of knowledge graph (i.e., taxonomy) is made obsolete by LLMs. Intuitively, LLMs should perform well on common taxonomies and at taxonomy levels that are common to people. Unfortunately, there lacks a comprehensive benchmark that evaluates the LLMs over a wide range of taxonomies from common to specialized domains and at levels from root to leaf so that we can draw a confident conclusion. To narrow the research gap, we constructed a novel taxonomy hierarchical structure discovery benchmark named TaxoGlimpse to evaluate the performance of LLMs over taxonomies. TaxoGlimpse covers ten representative taxonomies from common to specialized domains with in-depth experiments of different levels of entities in this taxonomy from root to leaf. Our comprehensive experiments of eighteen state-of-the-art LLMs under three prompting settings validate that LLMs can still not well capture the knowledge of specialized taxonomies and leaf-level entities.