Towards Memory-Efficient Neural Networks via Multi-Level in situ Generation

TL;DR

This paper introduces a novel multi-level in situ generation framework that significantly reduces memory access costs in neural networks, enabling more efficient deployment on resource-limited devices without sacrificing accuracy.

Contribution

It presents the first unified approach leveraging bit-level redundancy and intrinsic correlations in DNN kernels to enable on-the-fly high-resolution parameter recovery with minimal hardware overhead.

Findings

Boosts memory efficiency by 10-20x

Achieves comparable accuracy to state-of-the-art methods

Demonstrates effectiveness on multiple neural network architectures

Abstract

Deep neural networks (DNN) have shown superior performance in a variety of tasks. As they rapidly evolve, their escalating computation and memory demands make it challenging to deploy them on resource-constrained edge devices. Though extensive efficient accelerator designs, from traditional electronics to emerging photonics, have been successfully demonstrated, they are still bottlenecked by expensive memory accesses due to tremendous gaps between the bandwidth/power/latency of electrical memory and computing cores. Previous solutions fail to fully-leverage the ultra-fast computational speed of emerging DNN accelerators to break through the critical memory bound. In this work, we propose a general and unified framework to trade expensive memory transactions with ultra-fast on-chip computations, directly translating to performance improvement. We are the first to jointly explore the intrinsic correlations and bit-level redundancy within DNN kernels and propose a multi-level in situ generation mechanism with mixed-precision bases to achieve on-the-fly recovery of high-resolution parameters with minimum hardware overhead. Extensive experiments demonstrate that our proposed joint method can boost the memory efficiency by 10-20x with comparable accuracy over four state-of-the-art designs, when benchmarked on ResNet-18/DenseNet-121/MobileNetV2/V3 with various tasks.