RaTE: a Reproducible automatic Taxonomy Evaluation by Filling the Gap

TL;DR

RaTE introduces a reproducible, label-free automatic evaluation method for taxonomies using large pre-trained language models, aligning well with human judgments and enabling consistent assessment of taxonomy quality.

Contribution

The paper presents RaTE, a novel automatic taxonomy evaluation method that reduces reliance on manual scoring and correlates strongly with human judgments.

Findings

RaTE correlates well with human judgments

Degrading a taxonomy decreases RaTE score

RaTE provides a reproducible evaluation framework

Abstract

Taxonomies are an essential knowledge representation, yet most studies on automatic taxonomy construction (ATC) resort to manual evaluation to score proposed algorithms. We argue that automatic taxonomy evaluation (ATE) is just as important as taxonomy construction. We propose RaTE, an automatic label-free taxonomy scoring procedure, which relies on a large pre-trained language model. We apply our evaluation procedure to three state-of-the-art ATC algorithms with which we built seven taxonomies from the Yelp domain, and show that 1) RaTE correlates well with human judgments and 2) artificially degrading a taxonomy leads to decreasing RaTE score.