Concealed Face Analysis and Facial Reconstruction via a Multi-Task Approach and Cross-Modal Distillation in Terahertz Imaging

TL;DR

This paper introduces a multi-task learning network to analyze and reconstruct concealed faces in terahertz imaging, improving performance through cross-modal distillation.

Contribution

A novel MTL network with cross-modal distillation for THz facial analysis and reconstruction is proposed.

Findings

The MTL network successfully handles concealed face verification, posture classification, and facial reconstruction.

Cross-modal distillation improves latent space separability while maintaining task performance.

Both THz-only and distilled models achieve high fidelity in face reconstruction.

Abstract

Terahertz (THz) sub-millimeter wave imaging offers unique capabilities for stand-off biometrics through concealment, yet it suffers from severe sparsity, low resolution, and high noise. To address these limitations, we introduce a novel unified Multi-Task Learning (MTL) network centered on a custom shared U-Net-like THz data encoder. This network is designed to simultaneously solve three distinct critical tasks on concealed THz facial data, given a limited dataset of approximately 1400 THz facial images of 20 different identities. The tasks include concealed face verification, facial posture classification, and a generative reconstruction of unconcealed faces from concealed ones. While providing highly successful MTL results as a standalone solution on the very challenging dataset, we further studied the expansion of this architecture via a cross-modal teacher-student approach. During training, a privileged visible-spectrum teacher fuses limited visible features with THz data to guide the THz-only student. This distillation process yields a student network that relies solely on THz inputs at inference. The cross-modal trained student achieves better latent space in terms of inter-class separability compared to the single-modality baseline, but with reduced intra-class compactness, while maintaining a similar success in the task performances. Both THz-only and distilled models preserve high unconcealed face generative fidelity.

Figures