PACE: Prune-And-Compress Ensemble Models

TL;DR

PACE is a framework that combines pruning and compression to optimize ensemble models, improving performance and interpretability while maintaining control over faithfulness to the original ensemble.

Contribution

It introduces a two-phase strategy that actively generates diverse learners before pruning, offering a novel, theoretically grounded approach to ensemble model compression.

Findings

Outperforms prior pruning and compression methods.

Provides principled control of faithfulness guarantees.

Enhances ensemble diversity before pruning.

Abstract

Ensemble models achieve state-of-the-art performance on prediction tasks, but usually require aggregating a large number of weak learners. This can hinder deployment, interpretability, and downstream tasks such as robustness verification. Remedies to this issue fall into two main camps: pruning, which discards redundant learners, and compression, which generates new ones from scratch. We introduce PACE, a framework that interleaves these paradigms in a two-phase strategy. First, new learners are actively generated via a theoretically grounded procedure to enhance the diversity of the initial ensemble. When no more relevant learners can be found, a second phase of pruning is performed on this enriched ensemble. During both operations, PACE allows fine control on the faithfulness to the original ensemble. Experiments show that our method outperforms prior pruning and compression methods while offering principled control of faithfulness guarantees.