From Dataset Recycling to Multi-Property Extraction and Beyond

TL;DR

This paper advances information extraction by evaluating Transformer models on WikiReading, introducing a new dataset for multi-property extraction, and providing detailed analysis tools to improve model understanding.

Contribution

It introduces WikiReading Recycled, a new dataset that addresses previous limitations, and explores multi-property extraction with enhanced evaluation methods.

Findings

Dual-source Transformer model outperforms previous state-of-the-art

WikiReading Recycled dataset improves data quality and diversity

Diagnostic subsets enable detailed performance analysis

Abstract

This paper investigates various Transformer architectures on the WikiReading Information Extraction and Machine Reading Comprehension dataset. The proposed dual-source model outperforms the current state-of-the-art by a large margin. Next, we introduce WikiReading Recycled-a newly developed public dataset and the task of multiple property extraction. It uses the same data as WikiReading but does not inherit its predecessor's identified disadvantages. In addition, we provide a human-annotated test set with diagnostic subsets for a detailed analysis of model performance.