Quantifying the human visual exposome with vision language models

TL;DR

This paper introduces a scalable method combining vision language models and large language models to quantify the visual environment's impact on mental health, surpassing traditional proxies.

Contribution

It presents a novel, scalable approach to quantify visual context and its association with mental health using advanced AI models and real-world imagery.

Findings

VLM-derived greenness estimates predict affect and stress.

Over 33% of visual context ratings correlate with mental health measures.

The pipeline enables high-throughput decoding of visual environment effects.

Abstract

The visual environment is a fundamental yet unquantified determinant of mental health. While the concept of the environmental exposome is well established, current methods rely on coarse geospatial proxies or biased self reports, failing to capture the first person visual context of daily life. We addressed this gap by coupling ecological momentary assessment with vision language models (VLMs) to quantify the semantic richness of human visual experience. Across 2674 participant generated photographs, VLM derived estimates of greenness robustly predicted momentary affect and chronic stress, consistent with established benchmarks. We then developed a semi autonomous large language model (LLM) based pipeline that mined over seven million scientific publications to extract nearly 1000 environmental features empirically linked to mental health. When applied to real world imagery, up to 33 percent of VLM extracted context ratings significantly correlated with affect and stress. These findings establish a scalable objective paradigm for visual exposomics, enabling high throughput decoding of how the visible world is associated with mental health.