TaxoAlign: Scholarly Taxonomy Generation Using Language Models

TL;DR

TaxoAlign is a novel method that automatically generates scholarly taxonomies, aligning closely with human-created structures, and is validated through a new benchmark and comprehensive evaluation framework.

Contribution

We introduce TaxoAlign, a three-phase instruction-guided approach for scholarly taxonomy creation, along with the CS-TaxoBench benchmark and an automated evaluation framework.

Findings

TaxoAlign outperforms baseline methods on most metrics.

The benchmark includes 460 human-written and 80 curated taxonomies.

Automated and human evaluations confirm TaxoAlign's effectiveness.

Abstract

Taxonomies play a crucial role in helping researchers structure and navigate knowledge in a hierarchical manner. They also form an important part in the creation of comprehensive literature surveys. The existing approaches to automatic survey generation do not compare the structure of the generated surveys with those written by human experts. To address this gap, we present our own method for automated taxonomy creation that can bridge the gap between human-generated and automatically-created taxonomies. For this purpose, we create the CS-TaxoBench benchmark which consists of 460 taxonomies that have been extracted from human-written survey papers. We also include an additional test set of 80 taxonomies curated from conference survey papers. We propose TaxoAlign, a three-phase topic-based instruction-guided method for scholarly taxonomy generation. Additionally, we propose a stringent automated evaluation framework that measures the structural alignment and semantic coherence of automatically generated taxonomies in comparison to those created by human experts. We evaluate our method and various baselines on CS-TaxoBench, using both automated evaluation metrics and human evaluation studies. The results show that TaxoAlign consistently surpasses the baselines on nearly all metrics. The code and data can be found at https://github.com/AvishekLahiri/TaxoAlign.