Is artificial intelligence still intelligence? LLMs generalize to novel adjective-noun pairs, but don't mimic the full human distribution

TL;DR

This paper investigates the ability of large language models to generalize to novel adjective-noun pairs, showing they can mimic human judgments in context but still have limitations in out-of-context inferences.

Contribution

The study introduces methods to evaluate LLMs' understanding of adjective-noun combinations and reveals their partial success in mimicking human-like judgments.

Findings

Largest models generalize well in context

LLMs show human-like judgment distribution in 75% of cases

Room for improvement in out-of-context inference accuracy

Abstract

Inferences from adjective-noun combinations like "Is artificial intelligence still intelligence?" provide a good test bed for LLMs' understanding of meaning and compositional generalization capability, since there are many combinations which are novel to both humans and LLMs but nevertheless elicit convergent human judgments. We study a range of LLMs and find that the largest models we tested are able to draw human-like inferences when the inference is determined by context and can generalize to unseen adjective-noun combinations. We also propose three methods to evaluate LLMs on these inferences out of context, where there is a distribution of human-like answers rather than a single correct answer. We find that LLMs show a human-like distribution on at most 75\% of our dataset, which is promising but still leaves room for improvement.