FIB: A Method for Evaluation of Feature Impact Balance in Multi-Dimensional Data

TL;DR

This paper introduces the FIB score, a metric to evaluate the balance of feature contributions in errors across multi-dimensional data, aiding model assessment and selection.

Contribution

The paper proposes the FIB score, a novel metric to quantify feature impact balance in error vectors, with applications demonstrated on AutoEncoders and Variational AutoEncoders.

Findings

FIB score ranges from 0 to 1, indicating imbalance to balance in feature contributions.

FIB varies during training, reflecting changes in feature impact.

FIB supports model selection in both single and multi-output tasks.

Abstract

Errors might not have the same consequences depending on the task at hand. Nevertheless, there is limited research investigating the impact of imbalance in the contribution of different features in an error vector. Therefore, we propose the Feature Impact Balance (FIB) score. It measures whether there is a balanced impact of features in the discrepancies between two vectors. We designed the FIB score to lie in [0, 1]. Scores close to 0 indicate that a small number of features contribute to most of the error, and scores close to 1 indicate that most features contribute to the error equally. We experimentally study the FIB on different datasets, using AutoEncoders and Variational AutoEncoders. We show how the feature impact balance varies during training and showcase its usability to support model selection for single output and multi-output tasks.