The Use of Multimodal Large Language Models to Detect Objects from Thermal Images: Transportation Applications

TL;DR

This paper explores the use of Multimodal Large Language Models like GPT-4 and Gemini to detect and classify objects in thermal images for transportation safety, demonstrating their potential in autonomous systems.

Contribution

It evaluates the capability of MLLMs to understand thermal images, detect objects, and verify scene consistency across modalities, advancing multimodal perception in ITS applications.

Findings

GPT-4 and Gemini effectively detect objects in thermal images.

MAPE for pedestrian classification ranged from 70.39% to 81.48%.

MLLMs can identify thermal images for ITS applications.

Abstract

The integration of thermal imaging data with Multimodal Large Language Models (MLLMs) constitutes an exciting opportunity for improving the safety and functionality of autonomous driving systems and many Intelligent Transportation Systems (ITS) applications. This study investigates whether MLLMs can understand complex images from RGB and thermal cameras and detect objects directly. Our goals were to 1) assess the ability of the MLLM to learn from information from various sets, 2) detect objects and identify elements in thermal cameras, 3) determine whether two independent modality images show the same scene, and 4) learn all objects using different modalities. The findings showed that both GPT-4 and Gemini were effective in detecting and classifying objects in thermal images. Similarly, the Mean Absolute Percentage Error (MAPE) for pedestrian classification was 70.39% and 81.48%, respectively. Moreover, the MAPE for bike, car, and motorcycle detection were 78.4%, 55.81%, and 96.15%, respectively. Gemini produced MAPE of 66.53%, 59.35% and 78.18% respectively. This finding further demonstrates that MLLM can identify thermal images and can be employed in advanced imaging automation technologies for ITS applications.