Constructing Taxonomies from Pretrained Language Models

TL;DR

This paper introduces a novel method for building taxonomic trees from pretrained language models by predicting parent-child relations and optimizing these into trees, achieving significant improvements on WordNet subtrees across multiple languages.

Contribution

The paper presents a new two-module approach for constructing taxonomies from language models, incorporating web glosses and extending results to multiple languages.

Findings

Achieves 66.7 ancestor F1 on WordNet subtrees, a 20% improvement.

Effectively extends to nine languages using multilingual WordNet.

Incorporates web glosses to enhance taxonomy construction.

Abstract

We present a method for constructing taxonomic trees (e.g., WordNet) using pretrained language models. Our approach is composed of two modules, one that predicts parenthood relations and another that reconciles those predictions into trees. The parenthood prediction module produces likelihood scores for each potential parent-child pair, creating a graph of parent-child relation scores. The tree reconciliation module treats the task as a graph optimization problem and outputs the maximum spanning tree of this graph. We train our model on subtrees sampled from WordNet, and test on non-overlapping WordNet subtrees. We show that incorporating web-retrieved glosses can further improve performance. On the task of constructing subtrees of English WordNet, the model achieves 66.7 ancestor F1, a 20.0% relative increase over the previous best published result on this task. In addition, we convert the original English dataset into nine other languages using Open Multilingual WordNet and extend our results across these languages.