Harmonic-NAS: Hardware-Aware Multimodal Neural Architecture Search on Resource-constrained Devices
Mohamed Imed Eddine Ghebriout, Halima Bouzidi, Smail Niar, Hamza, Ouarnoughi

TL;DR
Harmonic-NAS is a hardware-aware neural architecture search framework that optimizes multimodal neural networks for resource-constrained devices, improving accuracy, latency, and energy efficiency.
Contribution
It introduces a two-tier optimization method for designing multimodal neural networks with hardware considerations, addressing the labor-intensive process of manual design.
Findings
Achieves up to 10.9% accuracy improvement
Reduces latency by 1.91x
Increases energy efficiency by 2.14x
Abstract
The recent surge of interest surrounding Multimodal Neural Networks (MM-NN) is attributed to their ability to effectively process and integrate multiscale information from diverse data sources. MM-NNs extract and fuse features from multiple modalities using adequate unimodal backbones and specific fusion networks. Although this helps strengthen the multimodal information representation, designing such networks is labor-intensive. It requires tuning the architectural parameters of the unimodal backbones, choosing the fusing point, and selecting the operations for fusion. Furthermore, multimodality AI is emerging as a cutting-edge option in Internet of Things (IoT) systems where inference latency and energy consumption are critical metrics in addition to accuracy. In this paper, we propose Harmonic-NAS, a framework for the joint optimization of unimodal backbones and multimodal fusion…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsNeural Networks and Applications · Advanced Neural Network Applications · Advanced Chemical Sensor Technologies
MethodsNeural Architecture Search
