Experimental study on surveillance video-based indoor occupancy measurement with occupant-centric control

TL;DR

This study evaluates LLM-enhanced vision-based indoor occupancy measurement methods and their impact on occupant-centric HVAC control, demonstrating improved accuracy and energy savings in smart buildings.

Contribution

It introduces a novel LLM-based refinement pipeline for occupancy measurement and demonstrates its effectiveness in HVAC energy savings.

Findings

Tracking-based methods improve temporal stability.

LLM-based refinement reduces false unoccupied predictions.

Achieved 17.94% HVAC energy savings with the proposed framework.

Abstract

Accurate occupancy information is essential for closed-loop occupant-centric control (OCC) in smart buildings. However, existing vision-based occupancy measurement methods often struggle to provide stable and accurate measurements in real indoor environments, and their implications for downstream HVAC control remain insufficiently studied. To achieve Net Zero emissions by 2050, this paper presents an experimental study of large language models (LLMs)-enhanced vision-based indoor occupancy measurement and its impact on OCC-enabled HVAC operation. Detection-only, tracking-based, and LLM-based refinement pipelines are compared under identical conditions using real surveillance data collected from a research laboratory in China, with frame-level manual ground-truth annotations. Results show that tracking-based methods improve temporal stability over detection-only measurement, while LLM-based refinement further improves occupancy measurement performance and reduces false unoccupied prediction. The best-performing pipeline, YOLOv8+DeepSeek, achieves an accuracy of 0.8824 and an F1-score of 0.9320. This pipeline is then integrated into an HVAC supervisory model predictive control framework in OpenStudio-EnergyPlus. Experimental results demonstrate that the proposed framework can support more efficient OCC operation, achieving a substantial HVAC energy-saving potential of 17.94%. These findings provide an effective methodology and practical foundation for future research in AI-enhanced smart building operations.