Scalable Object Detection in the Car Interior With Vision Foundation Models

TL;DR

The paper introduces ODAL, a distributed framework leveraging foundation models for interior vehicle object detection, overcoming resource constraints, and benchmarks it with a new ODALbench metric.

Contribution

It proposes a novel distributed architecture for interior scene understanding using foundation models and introduces ODALbench for comprehensive performance assessment.

Findings

Fine-tuned ODAL-LLaVA achieves 89% ODAL score, a 71% improvement.

ODAL-LLaVA outperforms GPT-4o by nearly 20% in ODAL score.

Fine-tuning reduces hallucinations and maintains high detection accuracy.

Abstract

AI tasks in the car interior like identifying and localizing externally introduced objects is crucial for response quality of personal assistants. However, computational resources of on-board systems remain highly constrained, restricting the deployment of such solutions directly within the vehicle. To address this limitation, we propose the novel Object Detection and Localization (ODAL) framework for interior scene understanding. Our approach leverages vision foundation models through a distributed architecture, splitting computational tasks between on-board and cloud. This design overcomes the resource constraints of running foundation models directly in the car. To benchmark model performance, we introduce ODALbench, a new metric for comprehensive assessment of detection and localization.Our analysis demonstrates the framework's potential to establish new standards in this domain. We compare the state-of-the-art GPT-4o vision foundation model with the lightweight LLaVA 1.5 7B model and explore how fine-tuning enhances the lightweight models performance. Remarkably, our fine-tuned ODAL-LLaVA model achieves an ODAL$_{score}$ of 89%, representing a 71% improvement over its baseline performance and outperforming GPT-4o by nearly 20%. Furthermore, the fine-tuned model maintains high detection accuracy while significantly reducing hallucinations, achieving an ODAL$_{SNR}$ three times higher than GPT-4o.