ISALux: Illumination and Segmentation Aware Transformer Employing Mixture of Experts for Low Light Image Enhancement

TL;DR

ISALux is a transformer-based low-light image enhancement method that integrates illumination and semantic priors using a novel self-attention mechanism and a Mixture of Experts to improve feature extraction and contextual learning.

Contribution

The paper introduces a new transformer architecture with a hybrid self-attention module and MoE-based FFN, incorporating illumination and semantic priors for enhanced low-light image processing.

Findings

Competitive with state-of-the-art methods on multiple datasets

Effective integration of illumination and semantic priors improves results

Ablation study confirms the contribution of each component

Abstract

We introduce ISALux, a novel transformer-based approach for Low-Light Image Enhancement (LLIE) that seamlessly integrates illumination and semantic priors. Our architecture includes an original self-attention block, Hybrid Illumination and Semantics-Aware Multi-Headed Self- Attention (HISA-MSA), which integrates illumination and semantic segmentation maps for en- hanced feature extraction. ISALux employs two self-attention modules to independently process illumination and semantic features, selectively enriching each other to regulate luminance and high- light structural variations in real-world scenarios. A Mixture of Experts (MoE)-based Feed-Forward Network (FFN) enhances contextual learning, with a gating mechanism conditionally activating the top K experts for specialized processing. To address overfitting in LLIE methods caused by distinct light patterns in benchmarking datasets, we enhance the HISA-MSA module with low-rank matrix adaptations (LoRA). Extensive qualitative and quantitative evaluations across multiple specialized datasets demonstrate that ISALux is competitive with state-of-the-art (SOTA) methods. Addition- ally, an ablation study highlights the contribution of each component in the proposed model. Code will be released upon publication.