New York Smells: A Large Multimodal Dataset for Olfaction

TL;DR

This paper introduces 'New York Smells,' a large multimodal dataset pairing images with olfactory signals from natural environments, enabling new research in machine olfaction and cross-modal perception.

Contribution

The paper presents a novel, extensive dataset of 7,000 paired smell-image samples from diverse objects and environments, facilitating multimodal olfactory research.

Findings

Visual data aids in cross-modal olfactory representation learning

Learned olfactory representations outperform hand-crafted features

The dataset enables new tasks like smell-to-image retrieval and scene recognition

Abstract

While olfaction is central to how animals perceive the world, this rich chemical sensory modality remains largely inaccessible to machines. One key bottleneck is the lack of diverse, multimodal olfactory training data collected in natural settings. We present New York Smells, a large dataset of paired image and olfactory signals captured ``in the wild.'' Our dataset contains 7,000 smell-image pairs from 3,500 distinct objects across indoor and outdoor environments, with approximately 70$\times$ more objects than existing olfactory datasets. Our benchmark has three tasks: cross-modal smell-to-image retrieval, recognizing scenes, objects, and materials from smell alone, and fine-grained discrimination between grass species. Through experiments on our dataset, we find that visual data enables cross-modal olfactory representation learning, and that our learned olfactory representations outperform widely-used hand-crafted features.