Neural Architecture Retrieval
Xiaohuan Pei, Yanxi Li, Minjing Dong, Chang Xu
TL;DR
This paper introduces Neural Architecture Retrieval, a method to efficiently find similar neural network designs by dividing graphs into motifs and using multi-level contrastive learning, supported by a large dataset of 12,000 architectures.
Contribution
It proposes a novel graph division into motifs and multi-level contrastive learning for neural architecture retrieval, addressing limitations of existing graph pre-training strategies.
Findings
Outperforms existing methods in retrieving similar architectures
Effective on both human-designed and synthesized architectures
Built a dataset with 12,000 neural architectures and embeddings
Abstract
With the increasing number of new neural architecture designs and substantial existing neural architectures, it becomes difficult for the researchers to situate their contributions compared with existing neural architectures or establish the connections between their designs and other relevant ones. To discover similar neural architectures in an efficient and automatic manner, we define a new problem Neural Architecture Retrieval which retrieves a set of existing neural architectures which have similar designs to the query neural architecture. Existing graph pre-training strategies cannot address the computational graph in neural architectures due to the graph size and motifs. To fulfill this potential, we propose to divide the graph into motifs which are used to rebuild the macro graph to tackle these issues, and introduce multi-level contrastive learning to achieve accurate graph…
Peer Reviews
Decision·ICLR 2024 poster
1. This work proposes a large new NAS dataset of 12k real-world neural architectures rather than pre-define search space. In my opinion, this thing makes a lot of sense for the NAS field which can explore the diversity of search space species architectures. 2. The idea uses motifs to encode architecture and reduce the graph size is novel and reasonable to encode architecture and capture the connection between structures in one architecture. 3. The two-level pretrain task to train the architect
1. In my opinion, the NAR problem to find similar architectures for the query architecture doesn't seem particularly significant for real-world usage, Can the author point out the need for this problem and more application scenarios? 2. Some retrieval-based papers[1-3] for giving a query dataset to search architectures should be discussed. 3. Another question is that I want to know how much performance can be achieved by retrieving similar models only based on the architecture corresponding to
1. The paper presents a new problem called Neural Architecture Retrieval (NAR). This problem is to find similar neural architectures quickly and easily from a big pool of existing and possible designs. It’s a smart way to make the search for neural architectures simpler and more efficient. 2. The authors propose a novel and easy-to-understand graph representation learning framework that addresses the computational graph in neural architectures. This framework adopts motifs of neural architectur
1. The proposed NAR appears to focus solely on the topological similarity of architectures. However, it is important to note that the similarity between architectures can vary across different tasks or datasets. For instance, certain architectures might yield comparable results in image classification but diverge significantly in performance when applied to other tasks. Could the authors provide additional insights and elaborations on this matter? 2. The authors introduce a motif-level contrast
* The motivation behind the paper is well-articulated and resonates with the ongoing challenges faced by researchers in situating their contributions amidst a plethora of existing neural architectures. The introduction of Neural Architecture Retrieval as a solution to automate the discovery of similar neural architectures is timely and could significantly alleviate the existing bottleneck. * The methods used in the paper sound and well-justified. The logic behind each step of the solution is sou
* Page 5 Section 3.4 Eq 4: The objective of the first stage may have a better way. The encoder encodes the architecture into motifs, and then concatenates the embeddings. Directly sampling the highest-frequency motifs $H$ to represent the main design for large models may more reasonable, especially considering that the concatenation of motif embeddings cannot backpropagate the gradients without two stages. * Page 7 Section 4.3: The details of the evaluation part are lacking. Although it covers
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Graph Neural Networks · Machine Learning in Materials Science · Neural Networks and Applications
MethodsContrastive Learning