SimVerb-3500: A Large-Scale Evaluation Set of Verb Similarity

TL;DR

SimVerb-3500 is a comprehensive human-annotated dataset of 3,500 verb pairs designed to evaluate and improve the understanding of verb similarity in distributional semantics models.

Contribution

It introduces a large-scale, detailed verb similarity dataset covering all VerbNet classes, enabling more robust evaluation and analysis of semantic models.

Findings

Provides extensive human ratings for verb pairs

Facilitates analysis of syntactic and semantic influences on verb meaning

Supports development of better verb representation methods

Abstract

Verbs play a critical role in the meaning of sentences, but these ubiquitous words have received little attention in recent distributional semantics research. We introduce SimVerb-3500, an evaluation resource that provides human ratings for the similarity of 3,500 verb pairs. SimVerb-3500 covers all normed verb types from the USF free-association database, providing at least three examples for every VerbNet class. This broad coverage facilitates detailed analyses of how syntactic and semantic phenomena together influence human understanding of verb meaning. Further, with significantly larger development and test sets than existing benchmarks, SimVerb-3500 enables more robust evaluation of representation learning architectures and promotes the development of methods tailored to verbs. We hope that SimVerb-3500 will enable a richer understanding of the diversity and complexity of verb semantics and guide the development of systems that can effectively represent and interpret this meaning.