Benchmark for Assessing Olfactory Perception of Large Language Models

TL;DR

This paper introduces the OP benchmark to evaluate large language models' ability to reason about olfactory concepts, revealing current models mainly rely on lexical associations and highlighting significant gaps in olfactory reasoning.

Contribution

The OP benchmark is the first comprehensive dataset for assessing LLMs' olfactory reasoning across multiple tasks and representations, providing insights into their capabilities and limitations.

Findings

Compound-name prompts outperform SMILES prompts in olfactory tasks.

Best model achieves 64.4% accuracy, indicating emerging but limited olfactory reasoning.

Multilingual aggregation improves prediction accuracy, AUROC = 0.86.

Abstract

Here we introduce the Olfactory Perception (OP) benchmark, designed to assess the capability of large language models (LLMs) to reason about smell. The benchmark contains 1,010 questions across eight task categories spanning odor classification, odor primary descriptor identification, intensity and pleasantness judgments, multi-descriptor prediction, mixture similarity, olfactory receptor activation, and smell identification from real-world odor sources. Each question is presented in two prompt formats, compound names and isomeric SMILES, to evaluate the effect of molecular representations. Evaluating 21 model configurations across major model families, we find that compound-name prompts consistently outperform isomeric SMILES, with gains ranging from +2.4 to +18.9 percentage points (mean approx +7 points), suggesting current LLMs access olfactory knowledge primarily through lexical associations rather than structural molecular reasoning. The best-performing model reaches 64.4\% overall accuracy, which highlights both emerging capabilities and substantial remaining gaps in olfactory reasoning. We further evaluate a subset of the OP across 21 languages and find that aggregating predictions across languages improves olfactory prediction, with AUROC = 0.86 for the best performing language ensemble model. LLMs should be able to handle olfactory and not just visual or aural information.