Conceptually Diverse Base Model Selection for Meta-Learners in Concept Drifting Data Streams

TL;DR

This paper introduces a novel method for selecting diverse base models in meta-learning for concept drifting data streams by estimating conceptual similarity via Principal Angles, improving efficiency and performance.

Contribution

It proposes two new approaches using conceptual similarity for base model selection, reducing computational overhead while maintaining predictive accuracy in online transfer learning.

Findings

Conceptual similarity thresholding reduces computational cost.

Both proposed methods achieve comparable predictive performance.

Conceptual clustering performs well without parameter tuning.

Abstract

Meta-learners and ensembles aim to combine a set of relevant yet diverse base models to improve predictive performance. However, determining an appropriate set of base models is challenging, especially in online environments where the underlying distribution of data can change over time. In this paper, we present a novel approach for estimating the conceptual similarity of base models, which is calculated using the Principal Angles (PAs) between their underlying subspaces. We propose two methods that use conceptual similarity as a metric to obtain a relevant yet diverse subset of base models: (i) parameterised threshold culling and (ii) parameterless conceptual clustering. We evaluate these methods against thresholding using common ensemble pruning metrics, namely predictive performance and Mutual Information (MI), in the context of online Transfer Learning (TL), using both synthetic and real-world data. Our results show that conceptual similarity thresholding has a reduced computational overhead, and yet yields comparable predictive performance to thresholding using predictive performance and MI. Furthermore, conceptual clustering achieves similar predictive performances without requiring parameterisation, and achieves this with lower computational overhead than thresholding using predictive performance and MI when the number of base models becomes large.