Hypernetworks for Dynamic Feature Selection

TL;DR

This paper introduces Hyper-DFS, a hypernetwork-based method for dynamic feature selection that generates task-specific classifiers on demand, outperforming existing approaches in various benchmarks.

Contribution

The paper proposes Hyper-DFS, a novel hypernetwork approach that reduces structural complexity and improves generalization in dynamic feature selection tasks.

Findings

Hyper-DFS outperforms state-of-the-art methods on synthetic and real tabular data.

It is competitive or superior on all tested image datasets.

Hyper-DFS exhibits stronger zero-shot generalization to unseen feature subsets.

Abstract

Dynamic feature selection (DFS) is a machine learning framework in which features are acquired sequentially for individual samples under budget constraints. The exponential growth in the number of possible feature acquisition paths forces a DFS model to balance fitting specific scenarios against maintaining general performance, even when the feature space is moderate in size. In this paper, we study the structural limitations of existing DFS approaches to achieve an optimal solution. Then, we propose \textsc{Hyper-DFS}, a hypernetwork-based DFS approach that generates feature subset-specific classifier parameters on demand. We show that the use of hypernetworks compared to mask-embedding methods results in a smaller structural complexity bound. We also use a Set Transformer encoding to create a smooth conditioning space for the hypernetwork, so that functionally similar tasks are also geometrically close. In our benchmarks, \textsc{Hyper-DFS} outperforms all state-of-the-art approaches on synthetic and real-life tabular data. It is also competitive or superior across all image datasets tested, and shows substantially stronger zero-shot generalisation to feature subsets never seen during training than existing DFS approaches.