PyNet-V2 Mobile: Efficient On-Device Photo Processing With Neural Networks

TL;DR

PyNet-V2 Mobile is a lightweight neural network architecture optimized for real-time RAW image processing on mobile devices, achieving high-quality results within 0.5 to 1.5 seconds by leveraging mobile AI accelerators.

Contribution

The paper introduces PyNET-V2 Mobile, a novel CNN architecture specifically designed for efficient on-device RAW image processing on mobile phones, surpassing traditional pipelines and previous neural network solutions.

Findings

Processes 12MP RAW images in under 1.5 seconds on mobile devices.

Outperforms traditional ISP pipelines in image quality.

Compatible with mobile AI accelerators, reducing latency to 0.5 seconds.

Abstract

The increased importance of mobile photography created a need for fast and performant RAW image processing pipelines capable of producing good visual results in spite of the mobile camera sensor limitations. While deep learning-based approaches can efficiently solve this problem, their computational requirements usually remain too large for high-resolution on-device image processing. To address this limitation, we propose a novel PyNET-V2 Mobile CNN architecture designed specifically for edge devices, being able to process RAW 12MP photos directly on mobile phones under 1.5 second and producing high perceptual photo quality. To train and to evaluate the performance of the proposed solution, we use the real-world Fujifilm UltraISP dataset consisting on thousands of RAW-RGB image pairs captured with a professional medium-format 102MP Fujifilm camera and a popular Sony mobile camera sensor. The results demonstrate that the PyNET-V2 Mobile model can substantially surpass the quality of tradition ISP pipelines, while outperforming the previously introduced neural network-based solutions designed for fast image processing. Furthermore, we show that the proposed architecture is also compatible with the latest mobile AI accelerators such as NPUs or APUs that can be used to further reduce the latency of the model to as little as 0.5 second. The dataset, code and pre-trained models used in this paper are available on the project website: https://github.com/gmalivenko/PyNET-v2