Syntax Helps ELMo Understand Semantics: Is Syntax Still Relevant in a Deep Neural Architecture for SRL?

TL;DR

This paper investigates whether explicit syntactic information remains beneficial for semantic role labeling when using advanced contextual embeddings like ELMo, finding that syntax still provides advantages, especially out-of-domain.

Contribution

It demonstrates that incorporating syntactic structures into neural SRL models with ELMo embeddings improves performance, highlighting the continued relevance of linguistic structure.

Findings

Syntactic models outperform syntax-free models with ELMo on in-domain data.

Syntactic models have a larger advantage on out-of-domain data.

ELMo reduces but does not eliminate the importance of syntax in SRL.

Abstract

Do unsupervised methods for learning rich, contextualized token representations obviate the need for explicit modeling of linguistic structure in neural network models for semantic role labeling (SRL)? We address this question by incorporating the massively successful ELMo embeddings (Peters et al., 2018) into LISA (Strubell et al., 2018), a strong, linguistically-informed neural network architecture for SRL. In experiments on the CoNLL-2005 shared task we find that though ELMo out-performs typical word embeddings, beginning to close the gap in F1 between LISA with predicted and gold syntactic parses, syntactically-informed models still out-perform syntax-free models when both use ELMo, especially on out-of-domain data. Our results suggest that linguistic structures are indeed still relevant in this golden age of deep learning for NLP.