NasHD: Efficient ViT Architecture Performance Ranking using Hyperdimensional Computing

TL;DR

NasHD introduces a hyperdimensional computing approach to efficiently rank vision transformer architectures, significantly reducing computation time while maintaining high accuracy in performance prediction.

Contribution

The paper presents NasHD, a novel hyperdimensional computing-based model that accelerates NAS for ViT architectures, outperforming traditional methods in speed and efficiency.

Findings

NasHD can rank nearly 100K ViT models in about 1 minute.

NasHD achieves comparable accuracy to more complex models.

Record encoding scheme enhances performance and efficiency.

Abstract

Neural Architecture Search (NAS) is an automated architecture engineering method for deep learning design automation, which serves as an alternative to the manual and error-prone process of model development, selection, evaluation and performance estimation. However, one major obstacle of NAS is the extremely demanding computation resource requirements and time-consuming iterations particularly when the dataset scales. In this paper, targeting at the emerging vision transformer (ViT), we present NasHD, a hyperdimensional computing based supervised learning model to rank the performance given the architectures and configurations. Different from other learning based methods, NasHD is faster thanks to the high parallel processing of HDC architecture. We also evaluated two HDC encoding schemes: Gram-based and Record-based of NasHD on their performance and efficiency. On the VIMER-UFO benchmark dataset of 8 applications from a diverse range of domains, NasHD Record can rank the performance of nearly 100K vision transformer models with about 1 minute while still achieving comparable results with sophisticated models.