Real-Time Refocusing using an FPGA-based Standard Plenoptic Camera

TL;DR

This paper introduces an FPGA-based hardware architecture for real-time plenoptic image rendering, achieving high throughput and power efficiency suitable for embedded and video applications, outperforming GPU and CPU solutions.

Contribution

The paper presents a novel FPGA implementation of FIR filters for plenoptic rendering, enabling real-time performance with significantly lower power consumption.

Findings

Achieves real-time plenoptic rendering with FPGA hardware.

Provides an order of magnitude performance improvement over GPU.

Offers three orders of magnitude performance gain over CPU.

Abstract

Plenoptic cameras are receiving increasing attention in scientific and commercial applications because they capture the entire structure of light in a scene, enabling optical transforms (such as focusing) to be applied computationally after the fact, rather than once and for all at the time a picture is taken. In many settings, real-time interactive performance is also desired, which in turn requires significant computational power due to the large amount of data required to represent a plenoptic image. Although GPUs have been shown to provide acceptable performance for real-time plenoptic rendering, their cost and power requirements make them prohibitive for embedded uses (such as in-camera). On the other hand, the computation to accomplish plenoptic rendering is well-structured, suggesting the use of specialized hardware. Accordingly, this paper presents an array of switch-driven Finite Impulse Response (FIR) filters, implemented with FPGA to accomplish high-throughput spatial-domain rendering. The proposed architecture provides a power-efficient rendering hardware design suitable for full-video applications as required in broadcasting or cinematography. A benchmark assessment of the proposed hardware implementation shows that real-time performance can readily be achieved, with a one order of magnitude performance improvement over a GPU implementation and three orders of magnitude performance improvement over a general-purpose CPU implementation.