NanoSD: Edge Efficient Foundation Model for Real Time Image Restoration

TL;DR

NanoSD is a family of optimized diffusion models designed for real-time image restoration on edge devices, balancing accuracy, latency, and size through innovative architectural and training techniques.

Contribution

NanoSD introduces a novel co-design and distillation approach to create lightweight diffusion models that maintain generative priors and enable real-time performance on edge hardware.

Findings

NanoSD achieves real-time inference (20ms) on mobile NPUs.

NanoSD outperforms prior lightweight diffusion models in quality and efficiency.

NanoSD supports multiple image restoration tasks with state-of-the-art results.

Abstract

Latent diffusion models such as Stable Diffusion 1.5 offer strong generative priors that are highly valuable for image restoration, yet their full pipelines remain too computationally heavy for deployment on edge devices. Existing lightweight variants predominantly compress the denoising U-Net or reduce the diffusion trajectory, which disrupts the underlying latent manifold and limits generalization beyond a single task. We introduce NanoSD, a family of Pareto-optimal diffusion foundation models distilled from Stable Diffusion 1.5 through network surgery, feature-wise generative distillation, and structured architectural scaling jointly applied to the U-Net and the VAE encoder-decoder. This full-pipeline co-design preserves the generative prior while producing models that occupy distinct operating points along the accuracy-latency-size frontier (e.g., 130M-315M parameters, achieving real-time inference down to 20ms on mobile-class NPUs). We show that parameter reduction alone does not correlate with hardware efficiency, and we provide an analysis revealing how architectural balance, feature routing, and latent-space preservation jointly shape true on-device latency. When used as a drop-in backbone, NanoSD enables state-of-the-art performance across image super-resolution, image deblurring, face restoration, and monocular depth estimation, outperforming prior lightweight diffusion models in both perceptual quality and practical deployability. NanoSD establishes a general-purpose diffusion foundation model family suitable for real-time visual generation and restoration on edge devices.