UTA-Sign: Unsupervised Thermal Video Augmentation via Event-Assisted Traffic Signage Sketching

TL;DR

This paper introduces UTA-Sign, an unsupervised method that fuses thermal and event camera data to improve traffic signage perception in low-light conditions for autonomous driving.

Contribution

It presents a novel dual-boosting fusion mechanism combining thermal frames and event signals for enhanced signage sketching and detection accuracy.

Findings

Superior signage sketching quality demonstrated

Improved detection accuracy in low-light scenarios

Effective fusion of thermal and event data

Abstract

The thermal camera excels at perceiving outdoor environments under low-light conditions, making it ideal for applications such as nighttime autonomous driving and unmanned navigation. However, thermal cameras encounter challenges when capturing signage from objects made of similar materials, which can pose safety risks for accurately understanding semantics in autonomous driving systems. In contrast, the neuromorphic vision camera, also known as an event camera, detects changes in light intensity asynchronously and has proven effective in high-speed, low-light traffic environments. Recognizing the complementary characteristics of these two modalities, this paper proposes UTA-Sign, an unsupervised thermal-event video augmentation for traffic signage in low-illumination environments, targeting elements such as license plates and roadblock indicators. To address the signage blind spots of thermal imaging and the non-uniform sampling of event cameras, we developed a dual-boosting mechanism that fuses thermal frames and event signals for consistent signage representation over time. The proposed method utilizes thermal frames to provide accurate motion cues as temporal references for aligning the uneven event signals. At the same time, event signals contribute subtle signage content to the raw thermal frames, enhancing the overall understanding of the environment. The proposed method is validated on datasets collected from real-world scenarios, demonstrating superior quality in traffic signage sketching and improved detection accuracy at the perceptual level.

Tables

Table 1. Table 1: Quantitative comparisons of different methods.

Metrics	Text-IF	SHIP	Diff-IF	SCI	Retinexformer	EvLight	UTA-Sign
NIQE↓	5.724	4.846	5.335	13.245	4.135	7.915	4.117
PI↓	4.654	4.243	4.538	8.288	3.887	5.350	3.287
EN↑	5.549	5.762	5.659	5.379	6.471	7.119	7.344
SD↑	15.547	15.517	15.612	15.647	25.375	52.998	66.080
CNNIQA↑	0.576	0.591	0.665	0.419	0.380	0.417	0.676

Table 2. Table 2: Ablation study of different loss compositions.

Method	NIQE↓	PI↓	EN↑	SD↑	CNNIQA↑
w/o G. Loss	5.385	4.500	7.102	45.501	0.478
w/o P. Loss	5.328	4.328	7.095	44.725	0.513
w/o P&G. Loss	5.390	4.389	7.105	45.321	0.514
UTA-Sign	5.229	4.303	7.116	45.590	0.525

Equations

$$F_{\text{EV}}^0=E_{\text{EV}}^{\ast }(I_{\text{EV}}),F_{\text{IR}}^0=E_{\text{IR}}^{\ast }(I_{\text{IR}}),$$

$$F_{\text{EV}}^k=E_{\text{EV}}^{\ast }(F_{\text{EV}}^{k-1}),F_{\text{IR}}^k=E_{\text{IR}}^{\ast }(F_{\text{IR}}^{k-1}).$$

$$O_{k-1}=D_{\text{cov}}(UP(F_{\text{IR}}^k\odot F_{\text{EV}}^k)\odot O_k),$$

$$M(i,j)=\{\begin{array}{rl}& 1\left\{i,j\right\}\in EV,\\ & 0\text{Otherwise.}\end{array}$$

$$G_{\text{SIS}}=M\times I_{\text{EV}}+(1-M)\times I_{\text{IR}}.$$

$${\mathbf{F}}_{\mathbf{D}}=D(V,D_o(V)),$$

$${\mathbf{F}}_{l+1}^{swin}=D_l(S_l({\mathbf{F}}_l^{swin}),$$

$$S_l({\mathbf{F}}_l^{swin})\equiv {\odot }_{b=1}^B{\mathcal{B}}_{l,b}({\mathbf{F}}_l^{swin}),$$

$$\begin{array}{rl}{F}_{out}^b& ={\text{Att}}_{3d}(\text{LayerNorm}(F_{in}^b)),\\ F_{out}^b& =\text{Roll}(F_{out}^b,\mathbf{s})+F_{out}^b,\\ \text{where}& \{\begin{array}{ll}\mathbf{s}=(0,0,0),& b\text{ is final block}\\ \mathbf{s}=\left(\frac{M_D}{2},\frac{M_H}{2},\frac{M_W}{2}\right),& \text{otherwise.}\end{array}\end{array}$$

$$I_f^{\ast }={\text{Att}}_{2d}(\text{Avg}({\mathbf{F}}_{out}^{swin}))$$

$$P_{\text{EV}}^{t^{\prime }\to t}=P^{\text{IR}\to \text{EV}}{P}_{\text{IR}}^{t^{\prime }\to t}(P^{\text{IR}\to \text{EV}}{)}^{-1}.$$

$$G_{\text{TCC}}(t)=1(\sum _{t^{\prime }=1}^{T}W(P_{\text{EV}}^{t^{\prime }\to t},I_{\text{EV}}^{t^{\prime }})+I_{\text{EV}}^t)$$

$$L_{\text{SIS}}(t)=|{\hat{I}}_f(t),G_{\text{SIS}}(t){|}_1.$$

$$L_{\text{TCC}}(t)=|I_f^{\ast }(t)-G_{\text{TCC}}(t){|}_1.$$

$$Loss=\sum_{t=1}^{T}\big{(}L_{\text{SIS}}(t)+L_{\text{TCC}}(t)+L_{per}(t)+L_{G}(t)\big{)}.$$

Full text

{highlights}

An unsupervised thermal-event augmentation method for traffic signage perception is proposed.

The proposed UTA-Sign leverages dual-modal complementarity for low-light signage sketching.

UTA-Sign delivers high-fidelity imagery and accurate detection under real-world traffic data.

The model extends perception ability and supports all-day driving with thermal cameras.

1 Introduction

A fundamental goal of autonomous driving is to maintain safety under all lighting conditions. To meet this challenge, the perception module combines sensors and navigation systems to gather essential data about the traffic environment. This enables a comprehensive understanding of road conditions, which is crucial for informed decision-making. To achieve high-quality perception, it is essential not only to capture the environmental geometry but also to accurately interpret the details of the signage, such as road signs, obstacle indicators, and license plates. By grasping these key elements, vehicles can develop planning strategies designed specifically for the current traffic situation, resulting in better control and improved safety.

The thermal camera works by detecting the radiation emitted from objects, enabling it to function effectively in total darkness without any need for additional light. Using temperature differences, these cameras clearly outline shapes and remain unaffected by surrounding light, such as ambient illumination or vehicle headlights, which ensures consistent visibility of thermal features. However, while thermal imaging excels in low-light settings and delineates edges, it falls short in capturing finer semantic details–for instance, traffic signs, as shown in Fig. 1(a)—-because of the similarities in material and temperature. This shortcoming can impede a thorough understanding of the surroundings and pose risks to driving safety.

The fusion between thermal and optical sensors [1, 2] has been widely investigated for complex perception environments and tasks. By integrating information from multiple sources, effective fusion solutions improve both accuracy and completeness. Among optical sensors, neuromorphic cameras (events) stand out because they detect changes in light intensity rather than absolute values [3, 4, 5, 6], making them highly sensitive to illumination variations (Fig. 1(b)). This system excels in delivering clear imagery even in dim lighting while effectively reducing motion blur, providing as a reliable method for enhancing thermal videos, bringing to light details about traffic signage that thermal sensors typically miss.

In this paper, we introduce UTA-Sign, an unsupervised thermal-event augmentation framework for signage sketching, recovering missing information in thermal frames, as shown in Fig. 1. The proposed UTA-Sign extends the feasibility of thermal sensors in low-light perception, further supporting applications such as autonomous driving. Specifically, the UTA-Sign employs a dual-modal boosting fusion strategy, in which event signals and thermal frames collaboratively guide the fusion process. The fusion model leverages event signals to enrich the signage information in thermal videos while preserving their original content. Importantly, this approach operates in an unsupervised manner, without the need for costly labeled data, facilitating scalability and real-world deployment.

The framework integrates thermal and event signals in a frame-by-frame manner to create signage sketches. The Signage Information Sketching (SIS) module consists of modality-specific encoders to extract features, which are then fused across multiple scales before being decoded. To improve temporal consistency in thermal videos, the sketches are refined using a Temporal Consistency Correction (TCC) module, which aligns frames through deformable convolutions and captures temporal correlations using shifted window self-attention. The TCC module operates in a recurrent manner, with each output relying not only on the current thermal and event inputs but also on the preceding sketch frames. The model progressively refines the outputs, preserving essential features and ensuring temporally coherent and continuous results.

We independently create pseudo ground truths for the SIS and TCC module training. For the SIS module, we implement a masking mechanism to extract signage information from event signals. Specifically, the raw signals from both modalities act as prompts to segment signage information from the event signals, which is then mapped onto the thermal images. For training the TCC module, we use thermal signals to estimate the motion of the event camera. The aligned event signals, synchronized within a unified time frame, serve as the pseudo ground truth. This approach eliminates the need for feature matching across the temporal uneven event camera, a process that is particularly unreliable in a standalone operation. As a result, this constructed pseudo ground truth enhances the completeness of the semantics and improves the consistency of semantic sketching over time.

To validate the effectiveness of the proposed thermal sketching method, we conducted a series of comprehensive experiments. These experiments evaluated both qualitative visual quality and quantitative performance, and also assessed the capabilities of detecting traffic signs in low-illumination driving environments. Specifically, we compared the accuracy of object detection across three approaches: fusion of Thermal and RGB data, fusion of event camera and RGB data, and direct enhancement of RGB frames in low-light conditions. The experimental results demonstrate that the proposed thermal sketching method is more effective for all-day perception in autonomous driving.

The main contributions of this paper are listed as follows:

• •

This paper introduces a novel model that combines thermal and event cameras for the first time to reconstruct high-quality images of traffic signage in thermal videos, specifically for low-light autonomous driving.

• •

We employ an unsupervised training approach that leverages the strengths of thermal and event cameras for mutual enhancement: the former provides precise motion estimation for spatial-temporal compensation of events, while the latter contributes missing fine texture details in thermal videos.

• •

We evaluated the proposed UTA-Sign comprehensively in real traffic scenarios, in terms of both imaging quality and downstream traffic-sign detection, and it demonstrates the optimal results under low-light driving conditions across various metrics.

2 Related Work

Here we review the studies of low-light perception, including RGB–infrared (covering the thermal infrared spectrum) image fusion and event-based intensity image reconstruction, as well as object detection in low-light scenes.

2.1 RGB-infrared Image Fusion

RGB-infrared image fusion integrates complementary information from both modalities to enhance visibility[7, 8, 9, 10, 11, 12, 13]. Current methods prioritize visual quality, lightweight architectures, and generalization. In terms of visual quality, FusionDN [7] is proposed to balance quality and fidelity. The SwinFusion [8] leverages cross-modal attention to enhance local complementarity and global consistency. SDNet [9] integrates an adaptive decision block that modulates fusion based on local texture richness. CrossFuse [10] introduces a novel cross-attention mechanism to enhance complementary information and generate fused images through a two-stage training strategy. For lightweight optimization, a self-supervised decomposition method [11] improves efficiency and robustness under limited data conditions. DIFNet [12] proposes a parameterized pipeline that decouples extraction, fusion, and reconstruction to improve generalization. U2Fusion [13] introduces adaptive information preservation for multiple fusion tasks.

Harsh illumination hinders semantic extraction from visible images, causing notable degradation. To address the issue, Tang et al. [14] employs a dynamic Transformer to enhance salient structures in low-light scenes. Zhang et al. propose DAAF [15], which decomposes images into frequency-domain components to separate illumination and reflection, thereby mitigating degradation. Leveraging large language models (LLMs), recent works incorporate semantic priors via text guidance to improve fusion quality [16, 17]. Specifically, Text-IF [16] utilizes semantic prompts to restore details in low-light regions. OmniFuse[17] introduces the Language driven model to image fusion task for eliminating degradation to improve the robustness of fusion results.

The aforementioned methods demonstrate strong performance in RGB–infrared fusion, effectively enhancing image clarity and edge definition. However, they still face challenges in generating robust and high-fidelity semantic representations under severe illumination conditions.

2.2 Event-based Intensity-Image Reconstruction

Event cameras respond only to brightness changes, offering high temporal resolution. A variational energy minimization framework [4] is developed to jointly estimate optical flow and reconstruct intensity images. Other approaches [5, 6] focus on continuous-time intensity estimation and real-time image reconstruction. Neural networks [18, 19, 20] are employed for event-assisted image reconstruction. Specifically, the research [18] enhances image quality given the sparse captured events. On the basis, research [19] models the optimal event number to reconstruct the high-quality intensity image. Research [20] develops a multi-task collaborative framework, guiding the fusion results approximates the requirement of multi-tasks. SPETNet [21] is proposed to improve the speed and quality of intensity image reconstruction. Specifically, SPETNet exploits the temporal dimension by formulating the input as a tensor rather than a grid, enabling faster reconstruction.

Extending to color-scale reconstruction, the RGB-event fusion system is proposed to integrate event data with color cues, significantly expanding the application scope of event cameras. Diffusion models (DDPM) [22] are applied to enhance image reconstruction quality. The edge information from the event signal is integrated into the DDPM model for sharp edge generation of the intensity image. Another method focuses on improving visual effects in reconstructed images [23]. Additionally, Guo et al. [24] introduce the event camera super-resolution (EFSR-Net) network and design a coupled response block to improve the resolution and restore detailed textures.

Although event cameras face challenges like excessive noise, they remain a key method for capturing fast texture information. The event-based fusion solution could enhance perception ability and address downstream tasks.

2.3 Object detection in Low-light Scenes

Detection in low-light environments requires enhancing the contrast of key features in images and optimizing algorithms through techniques such as color space decomposition and spectral analysis to improve accuracy. The retinex theory, which models the relationship between object brightness, surface reflection, and ambient light, has inspired several enhancement-based approaches, including RetinexNet [25] and Gaussian-Retinex [26].

Based on the illumination-reflection model, Yao et al. [27] enhance image illumination by converting RGB images to HSV and applying guided filtering for better object detection. Wu et al. [28] improve feature extraction, fusion, and classification to enhance detection under low light conditions. Additionally, Ren et al. [29] combine coordinate attention and MBconv to increase both efficiency and accuracy in low-light detection.

Other low-light detection algorithms utilize the YOLO framework, as seen in works by Hui et al. [30], Peng et al. [31], and Vinoth et al. [32]. Hui et al. [30] effectively learn and perform object detection even with limited or incomplete labeled data by using a pre-enhanced image. Peng et al. [31] introduce NLE-YOLO, which reduces noise and effectively fuses features from various scales, thus improving detection accuracy for objects of different sizes in low-light conditions. Vinoth et al. [32] estimate the depth curve for each pixel, enhancing both detection speed and performance in dark scenarios. They also apply TensorRT optimization to reduce computational complexity and memory usage, enabling real-time detection in practical applications.

Currently, the available low-light detection methods are primarily designed for macroscopic objects, limiting their ability to capture fine-grained textures and semantically rich content. Accurately detecting such intricate details remains a challenging and unresolved issue in the field.

3 Method

In this section, we first provide an overview of the proposed UTA-Sign pipeline. We then detail the signage information sketching (SIS) and temporal consistency correction (TCC) modules, along with the construction of pseudo ground truth for each. The training loss is introduced subsequently. Finally, we briefly describe the preprocessing steps—including camera registration and semantic objective localization—as well as the training configuration.

3.1 Pipeline of UTA-Sign

The pipeline of UTA-Sign is shown in Fig. 2. The cameras are calibrated via the intrinsic and extrinsic parameters of two sensors, thereby enabling precise spatial alignment between the two modalities. Once the cameras are calibrated, the hard-triggered synchronization is employed to capture data simultaneously.

The SIS is designed to map the signage information distinguished from event signals onto corresponding infrared frames, as shown in Fig. 2(a). A hierarchical encoder–decoder architecture is employed to capture multiscale features and generate the signage sketches for infrared frames. The unsupervised learning strategy is adopted, wherein a mask guides supervision: regions within the mask reconstruct content from event signals, while regions outside the mask are supervised using the original infrared images.

The TCC module enhances spatiotemporal visual consistency of the sketches across multiple frames caused by the non-uniform density of event signals, as shown in Fig. 2(b). Specifically, we introduce a deformable CNN to extract the temporally aligned feature across multiple frames, effectively modeling motion in the time domain. Later, the TCC extracts 3D features via shift-window spatiotemporal attention and infers the refined signage content of the target frame with a 2D vision transformer. The TCC module incrementally integrates information from preceding frames into the objective frame, enriching the semantic information while preserving visual consistency. Additionally, gradient loss and perceptual loss are incorporated to enhance the visual fidelity of the reconstructed sketches.

3.2 Signage Information Sketching

The SIS module is designed to generate a coarse sketch of signage content by mapping event signals onto the thermal frames. As illustrated in the left part of Fig. 3, the SIS adopts a dual-modal fusion framework based on a typical encoder–decoder architecture.

To simplify the feature extraction of continuous event signals, the event signals are partitioned into discrete event frames according to the frame rate of the thermal camera. Each event frame is independently paired with the corresponding thermal frame, thus the frame index $t$ is omitted. We denote the $I_{\text{EV}}$ and $I_{\text{EV}}$ as the thermal and event frames of each pair. The event encoder $E^{*}_{\text{EV}}$ and thermal encoder $E^{*}_{\text{IR}}$ individually construct the $K$ layers of feature extraction. The first layer of the feature is denoted as

[TABLE]

and subsequent layers are progressively derived as

[TABLE]

The decoder constructs the coarse-to-fine feature concatenation. Given the event frame $I_{\text{EV}}$ and the thermal frame $I_{\text{IR}}$and each layer $k$ of the decoder is represented as

[TABLE]

where $D_{\text{cov}}(\cdot)$ indicates the transposed convolution, and $UP(\cdot)$ presents the up-sampling to the same size as $O_{k}$. The $\odot$ indicates concatenation between features. The final output $\hat{I}_{f}=O_{0}$ is the final fusion result of SIS.

3.3 SIS Pseudo Ground Truth Construction

The SIS module is designed to preserve both high-level signage details and low-level thermal cues, thereby enhancing the perceptual richness and expressiveness of the output sketches. To this end, a mask $M$ is constructed to distinguish semantic regions within the scene and guide the fusion of event signals and thermal images. Specifically, we assume that $S$ signage region with semantics are present. Then we have the union of the regions, denoted as $EV=EV_{0}\cup EV_{1}\cup...\cup EV_{S}$. The mask $M$ is defined as

[TABLE]

where $i,j$ indicate the indices of pixels. According to the definition of the mask, we construct a pseudo ground truth, denoted as

[TABLE]

For each event–thermal frame pair, the mask and pseudo ground truths are constructed independently, thus we omit the temporal index.

3.4 Temporal Consistency Correction

Since the SIS module primarily focuses on short-term information, it overlooks the temporal consistency and smoothness of the sketches across consecutive frames. The asynchronous nature of event cameras may lead to signal missing and unpredictable noise. As a result, semantic sketches derived from event and thermal image fusion often exhibit inconsistent visual quality, thereby hindering high-fidelity real-world traffic perception. To address this issue, we introduce the TCC module to enhance temporal continuity.

As illustrated in Fig. 3, the input to TCC comprises $N$ frames: the SIS output of the target frame $\hat{I}_{f}(t)$ and its $N-1$ preceding frames. These frames are used to generate the final result $I^{*}_{f}(t)$ at time $t$. For clarity, and without loss of generality, we define $V=\left\{\hat{I}_{f}(t-N+1),\ldots,\hat{I}_{f}(t-1),\hat{I}_{f}(t)\right\}$, and omit the explicit time index $t$ in the following discussion.

The deformable CNN is performed to accommodate complicated object motion, which aligns the similar features described in different frames, effectively enhancing temporal coherence and structural consistency. Unlike standard convolutions with fixed sampling grids, the deformable CNN adaptively samples from irregular positions, enabling more effective capture of structural edges and fine details within the volume. We define the Deformable CNN $D$ associate with an offset estimation convolution network $D_{o}$, the output feature of the deformable CNN is

[TABLE]

where $D_{o}(V)$ denotes learned offsets of sampling positions.

Later, a 4-layer swin-transformer is employed to learn deformation features across multiple feature scales. The input of each layer $l$ is denoted as $\mathbf{F}^{swin}_{l}$, where $\mathbf{F}^{swin}_{1}=\mathbf{F_{D}}$ for the first layer. We denote the swin-transformer module of layer $l$ as $S_{l}$, and the feature downsampling as $d_{l}$. The formulation is presented as

[TABLE]

where $\mathbf{F}^{swin}_{out}(t)=\mathbf{F}^{swin}_{4}$ for the final layer.

The swin-transformer layer $S_{l}$ consists of multiple attention blocks. In layer $l$, we define $B$ attention blocks and the $b$-th is represented as $\mathcal{B}_{l,b}$. The $S_{l}$ is defined as

[TABLE]

where $\odot$ indicates the cascade calculation. In impletation, layers 1, 2, and 4 each contain 2 blocks, whereas layer 3 consists of 6 attention blocks for fully exploring the features.

Each attention block generally consists of a self-attention mechanism and a shift-window strategy, except for the final block, which does not employ window shifting. The self-attention mechanism $\text{Att}_{3d}$ calculates a weighted sum of all elements, where the weights are based on the similarity between the query and the keys. The shifted-window strategy $\text{Roll}(F,\textbf{s})$ performs the long-range feature interaction, where s indicates the range size. We define the input and output features are $F^{b}_{in}$ and $F^{b}_{out}$, respectively. Given the window size $M_{D}\times M_{H}\times M_{W}$, the process of attention block $\mathcal{B}_{l,b}$ includes

[TABLE]

After the four layers, the feature is decoded by a 2D transformer $\text{Att}_{2d}$ for the objective frame, which is presented as

[TABLE]

where $\text{Avg}()$ averages of features to the initial input size.

The pipelined strategy is employed where each frame assimilates complementary cues from preceding neighbors. The preceding frames guide the sketching of the current frame, enabling future online inference for real-world implementation.

3.5 TCC Pseudo Ground Truth Construction

In this subsection, we construct pseudo ground truth with consistent sketches to guide the training of the TCC module. We estimate the camera motion and warp the temporally asynchronous event signals to a determined time instant, thereby completing the semantics. Moreover, during a given time interval, by warping the asynchronously captured event signals to respective instants, yielding a continuous and view-consistent representation.

The asynchronous captured event signals hinder relative pose estimation via feature registration of event frames. Given that the relative pose between the thermal camera and the event camera remains constant, the event camera’s pose change is derived from the thermal camera’s pose change, as shown in Fig. 4. The pseudo ground truth is constructed over $T$ frames based on the sampling rate of the thermal camera. We denote the $T$ thermal frames as $I_{\text{IR}}^{1},I_{\text{IR}}^{2},\dots,I_{\text{IR}}^{T}$. The corresponding partitioned event frames are denoted as $I_{\text{EV}}^{1},I_{\text{EV}}^{2},\dots,I_{\text{EV}}^{T}$. The relative pose of the thermal camera from $t^{\prime}$ to $t$ is denoted as $P_{\text{IR}}^{t^{\prime}\rightarrow{}t}$. Meanwhile, the relative pose from the thermal camera to event camera is denoted as $P^{\text{IR}\rightarrow{}\text{EV}}$. We denote the pose of the event camera from $t^{\prime}$ to $t$ as $P_{\text{EV}}^{t^{\prime}\rightarrow{}t}$, which is derived as

[TABLE]

Given the pose $P_{\text{EV}}^{t^{\prime}\rightarrow{}t}$ and the image $I_{\text{EV}}^{t}$, the warped image is denoted as $W(P_{\text{EV}}^{t^{\prime}\rightarrow{}t},I_{\text{EV}}^{t})$. We consider $t$ as the objective frame and others are reference frame, the pseudo ground truth of frame $t$ is derived as

[TABLE]

where $\mathbf{1}$ is an indicator function. Pixels are binarized by setting them to 1 if they exceed a predefined threshold, and to 0 otherwise. The threshold is set to $\frac{T}{2}$ in the experiment.

To mitigate noise inherent in event data, we perform denoising post-warping by assuming that noise is independently distributed in the spatiotemporal domain. Isolated event points lacking local spatiotemporal support are identified as noise and removed.

3.6 Loss Function

The SIS module and TCC module are trained jointly while being constrained to independent loss functions. We construct an $\mathcal{L}_{1}$ loss to enable the output of SIS to approximate the spatial pseudo ground truth as closely as possible. Given the SIS output $\hat{I}_{f}(t)$ at time $t$, the SIS loss is

[TABLE]

TCC loss enforces temporal consistency by requiring the current frame’s output to resemble both the current input and the corresponding representations of preceding frames. Based on the output $I^{*}(t)$ at $t$, the TCC loss function is defined as

[TABLE]

In addition, we introduce perceptual loss $L_{per}$ for the signage region to further constrain the fusion results. Comparing perceptual similarities in the high-level feature space makes the fused reconstructed semantics visually more natural. Gradient loss $L_{G}$ is introduced to avoid the distortion of high-frequency information by averaging the details of the image. We introduce the Laplacian operator to extract the edges of the semantic part and enhance them at the semantic edges. In summary, the total loss is written as

[TABLE]

Based on the loss function, we construct the end-to-end training integrating the SIS loss and the TCC loss. The SIS loss leverages the network to generate the clear sketched semantics, and the TCC loss constrains the temporal consistency of visual perception.

3.7 Preprocessing and Training Details

The registration of dual-camera systems involves aligning the cameras in both time and space. To achieve time-domain alignment, we generate an external signal that continuously outputs waveforms at predefined intervals. For spatial pose estimation, we utilize an thermal-recognizable checkerboard calibration board for corner detection and camera pose estimation.

It is important to note that the two cameras are fixed side by side. Due to variations in their fields of view and resolutions, the captured images cannot be directly aligned at the pixel level. To resolve this issue, we perform camera registration by aligning the grayscale frames from the DAVIS 346 sensor with the corresponding thermal frames. This is achieved by extracting and matching distinctive features from both image modalities, which allows us to accurately register the camera positions from the two perspectives.

We implemented the network training and inference with PyTorch on NVIDIA RTX 4090 under Ubuntu 20.04. In the training stage, we adopted Adam optimizer with $\beta_{1}=0.9,\beta_{2}=0.9$, and $\beta_{3}=0.95$. The learning rate is initialized to $5e-4$ and gradually decayed to $1e-6$ using the linear reduction. We randomly cropped 1/4 of the original image and employed random flipping and rotation as data augmentation tricks to enhance the image representation. The batch size was set to 16. We conducted 300 epochs until convergence. The total parameter size of the proposed UTA-Sign is 38.27M and the FLOPs is 218.05G for the input size of (448, 448).

4 Experiment Results

In this section, we thoroughly evaluate the performance of our method, referred to as UTA-Sign. We start by explaining the process of dataset construction. Next, we introduce the evaluation metrics and the baseline comparisons. We then present both qualitative and quantitative results obtained from real-world scenarios to demonstrate the effectiveness of UTA-Sign. Additionally, we perform objective detection as a downstream task to showcase the system’s capabilities in low-illumination conditions. Finally, we conduct ablation studies to analyze the various modules.

4.1 Dataset Construction

We curated a comprehensive dataset encompassing a wide range of low-light night-time traffic scenarios. The dataset comprises 34 distinct scenes and a total of 1,530 data groups, with each group consisting of 7 consecutive frames. It is structured into two components: real-world collected data and simulated data.

Real-world Data.  The real-world data is collected under nighttime conditions to showcase the raw data in low light. The acquisition system integrates a thermal camera and an event camera, as illustrated in Fig. 5. The thermal camera used is the HikVision industrial long-wave thermal model MV-CI003-GLN6, which operates within the 8–14 $\mu$m spectral range. It offers a resolution of 640 × 512 pixels and supports a frame rate of 50 frames per second. The event camera, DAVIS 346, can produce up to 12 million events in 1 $\mu$s, with a spatial resolution of 346 × 260 pixels.

Simulated Data.  The simulated data is derived from RGB images captured in real-world low-light scenes. We then apply the method proposed in the sRGB project [33] to convert these RGB images into approximate thermal representations. Additionally, event data are synthesized following the methodology outlined in rpd_vid2e [34]. For each reference RGB frame, we aggregate events within a 30-ms temporal window to construct a single event frame, ensuring temporal alignment between each pair of captured data.

4.2 Metrics and Baselines

Metrics.  We employ NIQE, PI, EN, SD, CNNIQA as the no-reference image quality metrics to evaluate the performance of fusion algorithms, all the indication of the metrics is thoroughly described in research [1]. These metrics investigate whether the fused images satisfy human perceptual requirements and show clear edges, thereby validating which thermal-event fusion scheme satisfies human visual interpretation of the scene.

NIQE evaluates image quality by measuring deviations from natural scene statistics, where lower scores indicate higher perceptual quality. PI is defined as a composite metric with lower values signifying better visual fidelity. EN quantifies the amount of information or detail in an image, with higher values indicating greater richness. SD measures the dispersion of pixel intensities and serves as an indicator of contrast. CNNIQA implicitly establishes the relationship between image content and perceived quality through a neural network, providing a robust and accurate assessment of feature’s clarity.

Baselines.  We utilize six baseline algorithms to assess the semantic quality of images captured in low-light conditions. Given the absence of direct fusion for thermal-event signals, several RGB imaging–related methods are investigated to validate the UTA-Sign’s effectiveness in low-light scenarios. Among these algorithms, Text-IF [16], SHIP [35], and Diff-IF [36] are prominent methods for fusing visible and thermal images. Text-IF employs a text-guided image fusion technique that establishes an interactive framework to improve perceptual quality, especially in degraded scenarios. SHIP enhances the model’s ability to collaborate across multiple modalities, integrating both global and fine-grained features across spatial and channel dimensions. Diff-IF introduces a difference mapping that effectively reduces training instability and prevents pattern collapse in the fusion results.

On the other hand, SCI [37] and Retinexformer [38] focus on enhancing RGB images captured in low-light environments. SCI is an unsupervised method that employs a cascaded approach, wherein each stage independently calibrates and refines image quality. Retinexformer features an illumination-guided Transformer designed to estimate lighting conditions and facilitate interactions across different regions of an image to capture long-range correlations.

Finally, EvLight [39] integrates information from RGB and event-based cameras to enhance visual quality. Specifically, EvLight constructs a comprehensive dataset of paired RGB and event-based images, enabling the reconstruction of low-light visible images that closely approximate the original RGB images captured under natural lighting conditions.

4.3 Qualitative Results

We conducted qualitative experiments using two types of scenarios: one based on simulated data and the other on real-world collected data. The results for the simulated data are illustrated in Fig. 6, while the outcomes based on real-world collected data are presented in Fig. 7. In each scenario, we present 7 consecutive frames to evaluate temporal continuity and visual quality for each method.

Fig. 6 displays the semantic reconstruction of a parking sign. Under extremely low light conditions, the fusion of RGB and thermal images (Text-IF, SHIP, Diff-IF) makes it challenging to identify the semantics clearly. The results of low-light enhancement methods (SCI, Retinexformer) still appear dark and require careful examination to discern any improvement. Although EvLight enhances brightness in ultra-low light conditions, its reconstructed results exhibit higher noise levels compared to our method, UTA-Sign. This increased noise results from EvLight’s failure to align events in the time domain; using raw events for scene reconstruction leads to inconsistent timing of signals.

Fig. 7 showcases the semantic reconstruction of a Chinese traffic sign that reads “Roundabout — No Wrong-Way Driving Allowed,” based on real-world data. In this scenario, UTA-Sign displays the clearest semantic information. In low-light conditions, thermal images lack surface semantic details, and RGB images provide limited usefulness. As a result, the RGB-thermal fusion outcomes closely resemble the thermal imagery alone. Direct enhancement of RGB images results in only marginal improvements in clarity and demonstrates poor adaptability. While the RGB and event fusion approach enhances overall brightness, it ultimately falls short in fidelity and consistency.

In conclusion, Figs. 6 and 7 illustrate that when RGB fails to capture semantic information, all RGB-related solutions also fail. In contrast, event cameras effectively capture lighting changes and compensate for the semantic limitations, making them well-suited for complementary fusion with thermal cameras in low-light scenarios.

4.4 Quantitative Results

This subsection presents quantitative results from perceptual metrics and the detection accuracy of traffic signs to verify the effectiveness of our approach.

**Perceptual Metrics. ** The UTA-Sign represents the initial method aimed at reconstructing traffic signage by integrating thermal and event signals. In low-illumination scenarios, the lack of ground-truth references for texture renders conventional pixel-level evaluation impractical. Therefore, we utilize perceptual evaluation metrics to assess the visual quality of the reconstructed images.

Tab. 1 presents the quantitative results. The method UTA-Sign outperforms other approaches across multiple evaluation metrics, indicating a significant advantage in perceptual quality. Retinexformer achieves the second-best performance in the NIQE and PI metrics, while Evlight ranks second in terms of EN and SD. Diff-IF shows relatively strong performance on the CNNIQA metric. Retinexformer primarily performs global brightness adjustments and denoising, which enhances the image’s natural appearance. Although Evlight improves texture richness and contrast metrics, it suffers from excessive high-frequency noise, which negatively impacts its visual quality.

**Detection Accuracy. ** Since object detection is a crucial task in autonomous driving, we evaluate UTA-Sign and the baseline models with respect to their accuracy in detecting traffic signs, as shown in Fig. 8. The detection module is implemented by training YOLOv9 on the TT100K dataset. To illustrate detection performance, we present two representative scenes, in which red bounding boxes indicate correct detections and yellow bounding boxes indicate incorrect ones. In the first scene, Text-IF, EvLight, and the proposed UTA-Sign can successfully identify the traffic sign; however, EvLight produces an incorrect detection result. Both Text-IF and UTA-Sign effectively localize and accurately classify the semantic content, with UTA-Sign exhibiting superior accuracy.

The second scene in Fig. 8 contains two traffic signs. In this case, UTA-Sign successfully identifies both signs. EvLight localizes both signs but accurately detects only one of them, while Retinexformer correctly detects and classifies a single sign. Although the image enhancement and fusion methods effectively reconstruct structural elements such as buildings, they struggle with the fine-grained signage details. The proposed UTA-Sign achieves reliable semantic sketching to support downstream tasks, enabling the implementation in practical scenes.

4.5 Ablation Studies

In this subsection, we evaluate the impact of the temporal consistency constraint module and the loss function on the UTA-Sign model separately.

**Contribution from the Temporal Consistency Constraint. ** To evaluate the effectiveness of the TCC module, we conduct ablation experiments comparing the model’s performance with and without this module, as shown in Fig. 9.

Without the TCC module, the bicycle and arrow symbols in the first scene of Fig. 9 are only faintly recognizable, and the sparse event points do not provide enough information for information perception. In contrast, with TCC module, the multi-frame sketches enhance the reconstruction of the target frame, leading to denser points and clear instruction.

In the second scenario involving license plate numbers, illustrated in Fig. 9, the absence of the TCC module results in missing license plate information in the third and fifth reconstructed frames, affecting different regions. However, the TCC module compensates for this missing information by utilizing guidance and constraints from multi-frame data. For example, in the fifth frame, the TCC module draws on adjacent frames to restore the missing details. These results demonstrate that the TCC module effectively maintains temporal visual consistency, which in turn enhances the richness of the semantic information.

**Contributions of Different Loss Function Items. ** We further investigate the contribution of different loss components to the overall performance of the UTA-Sign model. Specifically, we conduct ablation experiments by removing the gradient loss and the perceptual loss, respectively, while keeping other settings unchanged.

As shown in Tab. 2, the removal of either the gradient loss or the perceptual loss leads to a decline in performance across most metrics. This demonstrates that both losses are essential for enhancing output quality and stability. Interestingly, when both the gradient and perceptual losses are removed simultaneously (referred to as w/o P&G Loss), certain metrics do not show further deterioration in performance. This can be attributed to the overlapping functions of the two losses, both of which improve edge sharpness and texture details. When only one of the losses is removed, the other can provide some compensation, resulting in a more noticeable drop in performance. In contrast, when both losses are absent, the model training depends entirely on pixel-level loss. This simplifies the objective function and minimizes gradient conflicts that arise from multiple losses, allowing the performance in some metrics to remain close to that observed with the removal of just one loss.

Overall, the full model with all loss components achieves the best quantitative results, confirming the effectiveness of the proposed loss design in improving both low-level image quality and high-level perceptual characteristics.

5 Conclusion

This paper presents a novel solution for augmenting thermal video in low-light autonomous driving perception, focusing on traffic signage sketching by utilizing the strengths of event cameras. The proposed method, referred to as UTA-Sign, constructs a dual-boosting fusion network integrated with an unsupervised training mechanism. This approach targets traffic signage and refines the raw captured data, ensuring robustness and adaptability in various driving scenarios. To address the challenges posed by non-uniform and incomplete signals resulting from the asynchronous acquisition of event cameras, this study utilizes contour information derived from thermal imagery to achieve alignment and enhancement of the event signals. Furthermore, the complete event sequence is used to enrich the semantic information of the raw thermal frames. Experimental results demonstrate the effectiveness of the proposed method in generating high-quality visual reconstructions and supporting downstream detection tasks. This fusion framework offers a promising solution to the longstanding challenge of reliable signage detection and analysis in extremely low-light driving environments, contributing to advancements in autonomous driving technologies.

\printcredits