CSCO: Connectivity Search of Convolutional Operators

TL;DR

This paper introduces CSCO, a new method for discovering effective dense connectivity patterns in convolutional neural networks using a neural predictor, graph isomorphism augmentation, and evolutionary search, leading to improved ImageNet performance.

Contribution

CSCO presents a novel paradigm for efficient connectivity search in ConvNets, reducing reliance on existing motifs and enhancing search effectiveness with new techniques.

Findings

Achieves ~0.6% accuracy improvement on ImageNet over existing methods.

Introduces graph isomorphism data augmentation for better sample efficiency.

Employs Metropolis-Hastings evolutionary search to avoid local optima.

Abstract

Exploring dense connectivity of convolutional operators establishes critical "synapses" to communicate feature vectors from different levels and enriches the set of transformations on Computer Vision applications. Yet, even with heavy-machinery approaches such as Neural Architecture Search (NAS), discovering effective connectivity patterns requires tremendous efforts due to either constrained connectivity design space or a sub-optimal exploration process induced by an unconstrained search space. In this paper, we propose CSCO, a novel paradigm that fabricates effective connectivity of convolutional operators with minimal utilization of existing design motifs and further utilizes the discovered wiring to construct high-performing ConvNets. CSCO guides the exploration via a neural predictor as a surrogate of the ground-truth performance. We introduce Graph Isomorphism as data augmentation to improve sample efficiency and propose a Metropolis-Hastings Evolutionary Search (MH-ES) to evade locally optimal architectures and advance search quality. Results on ImageNet show ~0.6% performance improvement over hand-crafted and NAS-crafted dense connectivity. Our code is publicly available.