Model-Agnostic Dynamic Feature Selection with Uncertainty Quantification

TL;DR

This paper introduces a model-agnostic dynamic feature selection framework that quantifies uncertainty, enabling resource-efficient decision-making compatible with pre-trained models and highlighting the importance of uncertainty-aware strategies.

Contribution

It formalizes new uncertainty sources in dynamic feature selection and proposes a model-agnostic, efficient subset reparametrization approach compatible with existing classifiers.

Findings

Competitive accuracy on tabular and image datasets

Uncertainty sources persist across existing DFS methods

Highlights the need for uncertainty-aware DFS strategies

Abstract

Dynamic feature selection (DFS) addresses budget constraints in decision-making by sequentially acquiring features for each instance, making it appealing for resource-limited scenarios. However, existing DFS methods require models specifically designed for the sequential acquisition setting, limiting compatibility with models already deployed in practice. Furthermore, they provide limited uncertainty quantification, undermining trust in high-stakes decisions. In this work, we show that DFS introduces new uncertainty sources compared to the static setting. We formalise how model adaptation to feature subsets induces epistemic uncertainty, how standard imputation strategies bias aleatoric uncertainty estimation, and why predictive confidence fails to discriminate between good and bad selection policies. We also propose a model-agnostic DFS framework compatible with pre-trained classifiers, including interpretable-by-design models, through efficient subset reparametrization strategies. Empirical evaluation on tabular and image datasets demonstrates competitive accuracy against state-of-the-art greedy and reinforcement learning-based DFS methods with both neural and rule-based classifiers. We further show that the identified uncertainty sources persist across most existing approaches, highlighting the need for uncertainty-aware DFS.