Exploring Non-Verbal Predicates in Semantic Role Labeling: Challenges and Opportunities

TL;DR

This paper highlights the limitations of current SRL benchmarks in capturing non-verbal predicates, introduces a new dataset and challenge set to address this gap, and demonstrates the need for improved models to handle diverse predicate types.

Contribution

It presents a new PropBank dataset covering multiple predicate types and a challenge set to evaluate and improve SRL systems on non-verbal predicates.

Findings

Standard benchmarks underestimate non-verbal predicate performance.

State-of-the-art SRL systems struggle with cross-predicate transfer.

The new datasets reveal significant gaps in current SRL capabilities.

Abstract

Although we have witnessed impressive progress in Semantic Role Labeling (SRL), most of the research in the area is carried out assuming that the majority of predicates are verbs. Conversely, predicates can also be expressed using other parts of speech, e.g., nouns and adjectives. However, non-verbal predicates appear in the benchmarks we commonly use to measure progress in SRL less frequently than in some real-world settings -- newspaper headlines, dialogues, and tweets, among others. In this paper, we put forward a new PropBank dataset which boasts wide coverage of multiple predicate types. Thanks to it, we demonstrate empirically that standard benchmarks do not provide an accurate picture of the current situation in SRL and that state-of-the-art systems are still incapable of transferring knowledge across different predicate types. Having observed these issues, we also present a novel, manually-annotated challenge set designed to give equal importance to verbal, nominal, and adjectival predicate-argument structures. We use such dataset to investigate whether we can leverage different linguistic resources to promote knowledge transfer. In conclusion, we claim that SRL is far from "solved", and its integration with other semantic tasks might enable significant improvements in the future, especially for the long tail of non-verbal predicates, thereby facilitating further research on SRL for non-verbal predicates.