LLMs Can Check Their Own Results to Mitigate Hallucinations in Traffic Understanding Tasks

TL;DR

This paper evaluates the effectiveness of SelfCheckGPT in detecting and filtering hallucinations in traffic-related image captions generated by state-of-the-art LLMs, demonstrating its potential to improve reliability in automotive perception tasks.

Contribution

The study systematically assesses SelfCheckGPT's ability to identify hallucinations in LLM-generated traffic image captions across multiple datasets and models, highlighting its practical utility.

Findings

GPT-4o outperforms LLaVA in generating faithful captions.

Dataset type does not significantly impact caption quality or hallucination detection.

Models perform better on daytime images than on dawn, dusk, or night images.

Abstract

Today's Large Language Models (LLMs) have showcased exemplary capabilities, ranging from simple text generation to advanced image processing. Such models are currently being explored for in-vehicle services such as supporting perception tasks in Advanced Driver Assistance Systems (ADAS) or Autonomous Driving (AD) systems, given the LLMs' capabilities to process multi-modal data. However, LLMs often generate nonsensical or unfaithful information, known as ``hallucinations'': a notable issue that needs to be mitigated. In this paper, we systematically explore the adoption of SelfCheckGPT to spot hallucinations by three state-of-the-art LLMs (GPT-4o, LLaVA, and Llama3) when analysing visual automotive data from two sources: Waymo Open Dataset, from the US, and PREPER CITY dataset, from Sweden. Our results show that GPT-4o is better at generating faithful image captions than LLaVA, whereas the former demonstrated leniency in mislabeling non-hallucinated content as hallucinations compared to the latter. Furthermore, the analysis of the performance metrics revealed that the dataset type (Waymo or PREPER CITY) did not significantly affect the quality of the captions or the effectiveness of hallucination detection. However, the models showed better performance rates over images captured during daytime, compared to during dawn, dusk or night. Overall, the results show that SelfCheckGPT and its adaptation can be used to filter hallucinations in generated traffic-related image captions for state-of-the-art LLMs.