MF-LPR$^2$: Multi-Frame License Plate Image Restoration and Recognition using Optical Flow

TL;DR

MF-LPR$^2$ is a novel multi-frame framework that improves license plate image restoration and recognition by aligning and aggregating neighboring frames with optical flow, outperforming existing models on a new challenging dataset.

Contribution

This paper introduces MF-LPR$^2$, a multi-frame license plate restoration and recognition method that leverages optical flow and spatio-temporal consistency, along with a new dataset for evaluation.

Findings

MF-LPR$^2$ outperforms recent restoration models in PSNR, SSIM, and LPIPS.

MF-LPR$^2$ achieves 86.44% recognition accuracy, surpassing single-frame and multi-frame baselines.

Ablation studies confirm the effectiveness of filtering and refinement algorithms.

Abstract

License plate recognition (LPR) is important for traffic law enforcement, crime investigation, and surveillance. However, license plate areas in dash cam images often suffer from low resolution, motion blur, and glare, which make accurate recognition challenging. Existing generative models that rely on pretrained priors cannot reliably restore such poor-quality images, frequently introducing severe artifacts and distortions. To address this issue, we propose a novel multi-frame license plate restoration and recognition framework, MF-LPR$^2$, which addresses ambiguities in poor-quality images by aligning and aggregating neighboring frames instead of relying on pretrained knowledge. To achieve accurate frame alignment, we employ a state-of-the-art optical flow estimator in conjunction with carefully designed algorithms that detect and correct erroneous optical flow estimations by leveraging the spatio-temporal consistency inherent in license plate image sequences. Our approach enhances both image quality and recognition accuracy while preserving the evidential content of the input images. In addition, we constructed a novel Realistic LPR (RLPR) dataset to evaluate MF-LPR$^2$. The RLPR dataset contains 200 pairs of low-quality license plate image sequences and high-quality pseudo ground-truth images, reflecting the complexities of real-world scenarios. In experiments, MF-LPR$^2$ outperformed eight recent restoration models in terms of PSNR, SSIM, and LPIPS by significant margins. In recognition, MF-LPR$^2$ achieved an accuracy of 86.44%, outperforming both the best single-frame LPR (14.04%) and the multi-frame LPR (82.55%) among the eleven baseline models. The results of ablation studies confirm that our filtering and refinement algorithms significantly contribute to these improvements.