Four-dimensional video imaging via generative deep learning and a diffuser-encoded image sensor

TL;DR

This paper introduces 4DCam, a novel four-dimensional imaging system that captures space, spectrum, polarization, and time information simultaneously using a diffuser-encoded sensor, significantly enhancing material discrimination capabilities.

Contribution

The work presents a passive optical system combining physical encoding and generative decoding to enable real-time, multidimensional imaging beyond conventional cameras.

Findings

Achieved 96% accuracy in textile classification

Achieved 90% accuracy in camouflage detection

Improved material discrimination over hyperspectral imaging

Abstract

Light carries rich information across space, spectrum, polarization, and time, yet conventional cameras capture only a narrow projection of this multidimensional structure. A thin diffuser encodes wavelength-dependent information into single-shot scatterograms, captured by a polarization-resolving CMOS sensor that simultaneously measures four linear polarization states. We use 4DCam to image a live Betta splendens fish, uncovering polarization-dependent color modulations that remain invisible to conventional cameras. We experimentally show that the 4D information encoded in the scatterograms markedly improves material discrimination, achieving 96% accuracy for textile classification and 90% for camouflage detection, compared with 70% and 80%, respectively, using 3D hyperspectral imaging alone. Built entirely from passive optics, 4DCam seamlessly integrates physical encoding, generative decoding, and direct inference, enabling real-time, information-complete optical sensing.