Are LLMs Models of Distributional Semantics? A Case Study on Quantifiers

TL;DR

This study investigates whether large language models truly embody distributional semantics by examining their understanding of quantifiers, revealing they perform better on exact than vague quantifiers, challenging existing assumptions.

Contribution

It provides a systematic evaluation of LLMs' ability to handle different types of quantifiers, questioning the alignment of LLMs with distributional semantics theories.

Findings

LLMs align more closely with human judgments on exact quantifiers.

LLMs perform better on precise quantifiers than vague ones.

Results challenge assumptions about LLMs as models of distributional semantics.

Abstract

Distributional semantics is the linguistic theory that a word's meaning can be derived from its distribution in natural language (i.e., its use). Language models are commonly viewed as an implementation of distributional semantics, as they are optimized to capture the statistical features of natural language. It is often argued that distributional semantics models should excel at capturing graded/vague meaning based on linguistic conventions, but struggle with truth-conditional reasoning and symbolic processing. We evaluate this claim with a case study on vague (e.g. "many") and exact (e.g. "more than half") quantifiers. Contrary to expectations, we find that, across a broad range of models of various types, LLMs align more closely with human judgements on exact quantifiers versus vague ones. These findings call for a re-evaluation of the assumptions underpinning what distributional semantics models are, as well as what they can capture.