AFABench: A Generic Framework for Benchmarking Active Feature Acquisition

TL;DR

AFABench is a comprehensive benchmarking framework for Active Feature Acquisition, enabling systematic evaluation of various strategies on diverse datasets and fostering future research in cost-effective feature selection.

Contribution

This paper introduces AFABench, the first standardized benchmark platform for evaluating AFA methods across multiple datasets and strategies, including a novel synthetic dataset for testing lookahead capabilities.

Findings

Revealed trade-offs between static, myopic, and reinforcement learning AFA methods.

Demonstrated the effectiveness of different strategies on synthetic and real-world datasets.

Provided insights into the limitations of myopic approaches using the CUBE-NM dataset.

Abstract

In many real-world scenarios, acquiring all features of a data instance can be expensive or impractical due to monetary cost, latency, or privacy concerns. Active Feature Acquisition (AFA) addresses this challenge by dynamically selecting a subset of informative features for each data instance, trading predictive performance against acquisition cost. While numerous methods have been proposed for AFA, ranging from myopic information-theoretic strategies to non-myopic reinforcement learning approaches, fair and systematic evaluation of these methods has been hindered by a lack of standardized benchmarks. In this paper, we introduce AFABench, the first benchmark framework for AFA. Our benchmark includes a diverse set of synthetic and real-world datasets, supports a wide range of acquisition policies, and provides a modular design that enables easy integration of new methods and tasks. We implement and evaluate representative algorithms from all major categories, including static, myopic, and reinforcement learning-based approaches. To test the lookahead capabilities of AFA policies, we introduce a novel synthetic dataset, CUBE-NM, designed to expose the limitations of myopic selection. Our results highlight key trade-offs between different AFA strategies and provide actionable insights for future research. The benchmark code is available at: https://github.com/Linusaronsson/AFA-Benchmark.