Instilling Type Knowledge in Language Models via Multi-Task QA

TL;DR

This paper presents a novel multi-task question-answering approach to embed fine-grained entity type knowledge into language models using a large-scale dataset derived from Wikipedia and Wikidata, improving zero-shot entity understanding.

Contribution

The authors introduce the WikiWiki dataset and a training method that enhances language models' ability to recognize and infer entity types in a fine-grained manner.

Findings

State-of-the-art zero-shot dialog state tracking performance

Accurate inference of entity types in Wikipedia articles

Discovery of new useful entity types by models

Abstract

Understanding human language often necessitates understanding entities and their place in a taxonomy of knowledge -- their types. Previous methods to learn entity types rely on training classifiers on datasets with coarse, noisy, and incomplete labels. We introduce a method to instill fine-grained type knowledge in language models with text-to-text pre-training on type-centric questions leveraging knowledge base documents and knowledge graphs. We create the WikiWiki dataset: entities and passages from 10M Wikipedia articles linked to the Wikidata knowledge graph with 41K types. Models trained on WikiWiki achieve state-of-the-art performance in zero-shot dialog state tracking benchmarks, accurately infer entity types in Wikipedia articles, and can discover new types deemed useful by human judges.