Application of the representative measure approach to assess the reliability of decision trees in dealing with unseen vehicle collision data

TL;DR

This paper evaluates the $\\varepsilon$-representativeness method for assessing dataset similarity and reliability of decision trees, demonstrating theoretical guarantees and experimental validation on vehicle collision data.

Contribution

It provides a theoretical guarantee linking dataset similarity via $\\varepsilon$-representativeness to decision tree prediction similarity, and extends analysis to XGBoost on unseen vehicle collision data.

Findings

Theoretical guarantee of prediction similarity under $\\varepsilon$-representativeness.

Significant correlation between $\\varepsilon$-representativeness and feature importance ordering.

Experimental validation on vehicle collision data with XGBoost.

Abstract

Machine learning algorithms are fundamental components of novel data-informed Artificial Intelligence architecture. In this domain, the imperative role of representative datasets is a cornerstone in shaping the trajectory of artificial intelligence (AI) development. Representative datasets are needed to train machine learning components properly. Proper training has multiple impacts: it reduces the final model's complexity, power, and uncertainties. In this paper, we investigate the reliability of the $\varepsilon$-representativeness method to assess the dataset similarity from a theoretical perspective for decision trees. We decided to focus on the family of decision trees because it includes a wide variety of models known to be explainable. Thus, in this paper, we provide a result guaranteeing that if two datasets are related by $\varepsilon$-representativeness, i.e., both of them have points closer than $\varepsilon$, then the predictions by the classic decision tree are similar. Experimentally, we have also tested that $\varepsilon$-representativeness presents a significant correlation with the ordering of the feature importance. Moreover, we extend the results experimentally in the context of unseen vehicle collision data for XGboost, a machine-learning component widely adopted for dealing with tabular data.