TIER-A: Denoising Learning Framework for Information Extraction

TL;DR

This paper introduces TIER-A, a co-regularization framework that uses temperature calibration and entropy regularization to reduce overfitting in neural information extraction models trained on noisy datasets.

Contribution

The paper proposes a novel joint-training framework leveraging information entropy regularization to combat overconfidence and overfitting in neural models for information extraction.

Findings

Effective in reducing overfitting on noisy datasets

Improves extraction accuracy on TACRED and CoNLL03

Validates the entropy-based overfitting hypothesis

Abstract

With the development of deep neural language models, great progress has been made in information extraction recently. However, deep learning models often overfit on noisy data points, leading to poor performance. In this work, we examine the role of information entropy in the overfitting process and draw a key insight that overfitting is a process of overconfidence and entropy decreasing. Motivated by such properties, we propose a simple yet effective co-regularization joint-training framework TIER-A, Aggregation Joint-training Framework with Temperature Calibration and Information Entropy Regularization. Our framework consists of several neural models with identical structures. These models are jointly trained and we avoid overfitting by introducing temperature and information entropy regularization. Extensive experiments on two widely-used but noisy datasets, TACRED and CoNLL03, demonstrate the correctness of our assumption and the effectiveness of our framework.