TCEval: Using Thermal Comfort to Assess Cognitive and Perceptual Abilities of AI

TL;DR

TCEval introduces a novel framework using thermal comfort scenarios to evaluate AI's cognitive abilities, focusing on reasoning, causal understanding, and decision-making in real-world contexts.

Contribution

This work pioneers the use of thermal comfort as a paradigm for assessing AI cognition, providing a new benchmark that emphasizes embodied perception and adaptive decision-making.

Findings

LLMs show limited exact alignment with human thermal comfort feedback.

Directional consistency in LLM responses improves with a 1 PMV tolerance.

LLMs perform near-random in discrete thermal comfort classification.

Abstract

A critical gap exists in LLM task-specific benchmarks. Thermal comfort, a sophisticated interplay of environmental factors and personal perceptions involving sensory integration and adaptive decision-making, serves as an ideal paradigm for evaluating real-world cognitive capabilities of AI systems. To address this, we propose TCEval, the first evaluation framework that assesses three core cognitive capacities of AI, cross-modal reasoning, causal association, and adaptive decision-making, by leveraging thermal comfort scenarios and large language model (LLM) agents. The methodology involves initializing LLM agents with virtual personality attributes, guiding them to generate clothing insulation selections and thermal comfort feedback, and validating outputs against the ASHRAE Global Database and Chinese Thermal Comfort Database. Experiments on four LLMs show that while agent feedback has limited exact alignment with humans, directional consistency improves significantly with a 1 PMV tolerance. Statistical tests reveal that LLM-generated PMV distributions diverge markedly from human data, and agents perform near-randomly in discrete thermal comfort classification. These results confirm the feasibility of TCEval as an ecologically valid Cognitive Turing Test for AI, demonstrating that current LLMs possess foundational cross-modal reasoning ability but lack precise causal understanding of the nonlinear relationships between variables in thermal comfort. TCEval complements traditional benchmarks, shifting AI evaluation focus from abstract task proficiency to embodied, context-aware perception and decision-making, offering valuable insights for advancing AI in human-centric applications like smart buildings.