Reference-Free Evaluation of Taxonomies

TL;DR

This paper presents two novel reference-free metrics for evaluating taxonomies without labels, assessing robustness and logical adequacy, and demonstrating their effectiveness in predicting hierarchical classification performance.

Contribution

Introduces two innovative reference-free metrics for taxonomy quality evaluation, addressing limitations of existing metrics and enabling prediction of downstream classification performance.

Findings

Metrics correlate well with F1 against ground truth taxonomies.

Metrics effectively predict downstream hierarchical classification performance.

Proposed methods evaluate robustness and logical adequacy of taxonomies.

Abstract

We introduce two reference-free metrics for quality evaluation of taxonomies in the absence of labels. The first metric evaluates robustness by calculating the correlation between semantic and taxonomic similarity, addressing error types not considered by existing metrics. The second uses Natural Language Inference to assess logical adequacy. Both metrics are tested on five taxonomies and are shown to correlate well with F1 against ground truth taxonomies. We further demonstrate that our metrics can predict downstream performance in hierarchical classification when used with label hierarchies.