Multi-modal Machine Learning for Vehicle Rating Predictions Using Image, Text, and Parametric Data

TL;DR

This paper introduces a multi-modal learning model that combines image, text, and parametric data to improve vehicle rating predictions, outperforming unimodal models and aiding vehicle design optimization.

Contribution

The paper presents a novel multi-modal learning approach that integrates multiple data types for more accurate vehicle ratings, with interpretability analysis for design insights.

Findings

Multi-modal model's explanatory power is 4%-12% higher than unimodal models.

The model predicts five vehicle rating scores with improved accuracy.

SHAP analysis provides insights for vehicle design optimization.

Abstract

Accurate vehicle rating prediction can facilitate designing and configuring good vehicles. This prediction allows vehicle designers and manufacturers to optimize and improve their designs in a timely manner, enhance their product performance, and effectively attract consumers. However, most of the existing data-driven methods rely on data from a single mode, e.g., text, image, or parametric data, which results in a limited and incomplete exploration of the available information. These methods lack comprehensive analyses and exploration of data from multiple modes, which probably leads to inaccurate conclusions and hinders progress in this field. To overcome this limitation, we propose a multi-modal learning model for more comprehensive and accurate vehicle rating predictions. Specifically, the model simultaneously learns features from the parametric specifications, text descriptions, and images of vehicles to predict five vehicle rating scores, including the total score, critics score, performance score, safety score, and interior score. We compare the multi-modal learning model to the corresponding unimodal models and find that the multi-modal model's explanatory power is 4% - 12% higher than that of the unimodal models. On this basis, we conduct sensitivity analyses using SHAP to interpret our model and provide design and optimization directions to designers and manufacturers. Our study underscores the importance of the data-driven multi-modal learning approach for vehicle design, evaluation, and optimization. We have made the code publicly available at http://decode.mit.edu/projects/vehicleratings/.