Quirk or Palmer: A Comparative Study of Modal Verb Frameworks with Annotated Datasets

TL;DR

This paper introduces the Moverb dataset with annotated modal verb senses based on two frameworks, demonstrating the challenge of disambiguation and providing a resource for future NLP research.

Contribution

It presents a new dataset with annotations for modal verb senses using Quirk and Palmer frameworks, and evaluates RoBERTa classifiers on this data.

Findings

Similar inter-annotator agreement for both frameworks

Achieved F1 scores of 82.2 and 78.3 with RoBERTa classifiers

Modal sense disambiguation remains a challenging task

Abstract

Modal verbs, such as "can", "may", and "must", are commonly used in daily communication to convey the speaker's perspective related to the likelihood and/or mode of the proposition. They can differ greatly in meaning depending on how they're used and the context of a sentence (e.g. "They 'must' help each other out." vs. "They 'must' have helped each other out.") Despite their practical importance in natural language understanding, linguists have yet to agree on a single, prominent framework for the categorization of modal verb senses. This lack of agreement stems from high degrees of flexibility and polysemy from the modal verbs, making it more difficult for researchers to incorporate insights from this family of words into their work. This work presents Moverb dataset, which consists of 27,240 annotations of modal verb senses over 4,540 utterances containing one or more sentences from social conversations. Each utterance is annotated by three annotators using two different theoretical frameworks (i.e., Quirk and Palmer) of modal verb senses. We observe that both frameworks have similar inter-annotator agreements, despite having different numbers of sense types (8 for Quirk and 3 for Palmer). With the RoBERTa-based classifiers fine-tuned on \dataset, we achieve F1 scores of 82.2 and 78.3 on Quirk and Palmer, respectively, showing that modal verb sense disambiguation is not a trivial task. Our dataset will be publicly available with our final version.