CARL: Causality-guided Architecture Representation Learning for an Interpretable Performance Predictor

TL;DR

CARL introduces a causality-guided approach to architecture representation learning that improves the generalization and interpretability of neural architecture performance predictors, significantly enhancing NAS efficiency.

Contribution

The paper presents a novel causality-guided method that separates critical and redundant features in architecture representations for better performance prediction.

Findings

Achieves 97.67% top-1 accuracy on CIFAR-10 with DARTS.

Demonstrates state-of-the-art accuracy across five NAS search spaces.

Provides improved interpretability of architecture performance predictors.

Abstract

Performance predictors have emerged as a promising method to accelerate the evaluation stage of neural architecture search (NAS). These predictors estimate the performance of unseen architectures by learning from the correlation between a small set of trained architectures and their performance. However, most existing predictors ignore the inherent distribution shift between limited training samples and diverse test samples. Hence, they tend to learn spurious correlations as shortcuts to predictions, leading to poor generalization. To address this, we propose a Causality-guided Architecture Representation Learning (CARL) method aiming to separate critical (causal) and redundant (non-causal) features of architectures for generalizable architecture performance prediction. Specifically, we employ a substructure extractor to split the input architecture into critical and redundant substructures in the latent space. Then, we generate multiple interventional samples by pairing critical representations with diverse redundant representations to prioritize critical features. Extensive experiments on five NAS search spaces demonstrate the state-of-the-art accuracy and superior interpretability of CARL. For instance, CARL achieves 97.67% top-1 accuracy on CIFAR-10 using DARTS.