OntoGUM: Evaluating Contextualized SOTA Coreference Resolution on 12 More Genres
Yilun Zhu, Sameer Pradhan, Amir Zeldes

TL;DR
This paper introduces OntoGUM, a new diverse coreference dataset based on GUM, and evaluates the generalizability of state-of-the-art coreference models across 12 genres, revealing significant out-of-domain performance degradation.
Contribution
It provides the first large, human-annotated coreference dataset aligned with OntoNotes guidelines for multiple genres and evaluates model robustness across diverse domains.
Findings
Neural coreference models degrade 15-20% out-of-domain performance.
OntoGUM dataset enables comprehensive cross-genre evaluation.
Existing models show limited generalizability beyond OntoNotes.
Abstract
SOTA coreference resolution produces increasingly impressive scores on the OntoNotes benchmark. However lack of comparable data following the same scheme for more genres makes it difficult to evaluate generalizability to open domain data. This paper provides a dataset and comprehensive evaluation showing that the latest neural LM based end-to-end systems degrade very substantially out of domain. We make an OntoNotes-like coreference dataset called OntoGUM publicly available, converted from GUM, an English corpus covering 12 genres, using deterministic rules, which we evaluate. Thanks to the rich syntactic and discourse annotations in GUM, we are able to create the largest human-annotated coreference corpus following the OntoNotes guidelines, and the first to be evaluated for consistency with the OntoNotes scheme. Out-of-domain evaluation across 12 genres shows nearly 15-20% degradation…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsNatural Language Processing Techniques · Topic Modeling · Text Readability and Simplification
