GRAIL: A Benchmark for GRaph ActIve Learning in Dynamic Sensing Environments

TL;DR

GRAIL introduces a comprehensive benchmarking framework for graph active learning in dynamic environments, emphasizing metrics like diversity and user burden, and revealing trade-offs in real-world sensor data applications.

Contribution

This paper presents GRAIL, the first benchmark for evaluating graph active learning strategies in dynamic settings with new metrics and extensive experiments.

Findings

Existing methods often neglect user-centric metrics.

Trade-offs exist between prediction accuracy and user burden.

Dynamic environments require balancing node importance and diversity.

Abstract

Graph-based Active Learning (AL) leverages the structure of graphs to efficiently prioritize label queries, reducing labeling costs and user burden in applications like health monitoring, human behavior analysis, and sensor networks. By identifying strategically positioned nodes, graph AL minimizes data collection demands while maintaining model performance, making it a valuable tool for dynamic environments. Despite its potential, existing graph AL methods are often evaluated on static graph datasets and primarily focus on prediction accuracy, neglecting user-centric considerations such as sampling diversity, query fairness, and adaptability to dynamic settings. To bridge this gap, we introduce GRAIL, a novel benchmarking framework designed to evaluate graph AL strategies in dynamic, real-world environments. GRAIL introduces novel metrics to assess sustained effectiveness, diversity, and user burden, enabling a comprehensive evaluation of AL methods under varying conditions. Extensive experiments on datasets featuring dynamic, real-life human sensor data reveal trade-offs between prediction performance and user burden, highlighting limitations in existing AL strategies. GRAIL demonstrates the importance of balancing node importance, query diversity, and network topology, providing an evaluation mechanism for graph AL solutions in dynamic environments.