Knowledge-based model validation using a custom metric

TL;DR

This paper introduces a custom metric trained on expert opinions to assess vehicle model accuracy, providing a more direct measure of model sufficiency than traditional metrics.

Contribution

The paper presents a novel approach using a regression-based custom metric trained on expert assessments to evaluate vehicle model quality.

Findings

The custom metric accurately predicts expert judgments on model sufficiency.

Traditional metrics are insufficient to determine if a model is accurate enough.

The approach offers a confidence measure for model validation results.

Abstract

Vehicle models have a long history of research and as of today are able to model the involved physics in a reasonable manner. However, each new vehicle has its new characteristics or parameters. The identification of these is the main task of an engineer. To validate whether the correct parameter set has been chosen is a tedious task and often can only be performed by experts. Metrics known commonly used in literature are able to compare different results under certain aspects. However, they fail to answer the question: Are the models accurate enough? In this article, we propose the usage of a custom metric trained on the knowledge of experts to tackle this problem. Our approach involves three main steps: first, the formalized collection of subject matter experts' opinion on the question: Having seen the measurement and simulation time series in comparison, is the model quality sufficient? From this step, we obtain a data set that is able to quantify the sufficiency of a simulation result based on a comparison to corresponding experimental data. In a second step, we compute common model metrics on the measurement and simulation time series and use these model metrics as features to a regression model. Third, we fit a regression model to the experts' opinions. This regression model, i.e., our custom metric, can than predict the sufficiency of a new simulation result and gives a confidence on this prediction.