Better than BERT but Worse than Baseline

TL;DR

This paper evaluates BERT's performance on the ADI task, showing that while reranking improves results, BERT still underperforms compared to a rule-based baseline, highlighting areas for further improvement.

Contribution

It introduces new reranking features, charmatch and freq, to analyze BERT's limitations in abbreviation and expansion identification tasks.

Findings

BERT with reranking outperforms BERT without reranking.

BERT still underperforms compared to the rule-based baseline.

New features reveal specific areas where BERT is lacking.

Abstract

This paper compares BERT-SQuAD and Ab3P on the Abbreviation Definition Identification (ADI) task. ADI inputs a text and outputs short forms (abbreviations/acronyms) and long forms (expansions). BERT with reranking improves over BERT without reranking but fails to reach the Ab3P rule-based baseline. What is BERT missing? Reranking introduces two new features: charmatch and freq. The first feature identifies opportunities to take advantage of character constraints in acronyms and the second feature identifies opportunities to take advantage of frequency constraints across documents.