Entropy-Based Feature Extraction For Real-Time Semantic Segmentation

TL;DR

This paper presents an entropy-based patch encoder (EPE) that adaptively processes image patches with different entropy levels to improve real-time semantic segmentation efficiency and accuracy.

Contribution

The novel EPE module dynamically allocates computational resources based on patch entropy, enhancing segmentation performance with minimal additional cost.

Findings

EPE increases DFANet A's mIOU by 0.9% with 1.2% more parameters.

EPE boosts EDANet's mIOU by 1% with 10% more parameters.

EPE reduces computational cost by processing low-entropy patches with smaller encoders.

Abstract

This paper introduces an efficient patch-based computational module, coined Entropy-based Patch Encoder (EPE) module, for resource-constrained semantic segmentation. The EPE module consists of three lightweight fully-convolutional encoders, each extracting features from image patches with a different amount of entropy. Patches with high entropy are being processed by the encoder with the largest number of parameters, patches with moderate entropy are processed by the encoder with a moderate number of parameters, and patches with low entropy are processed by the smallest encoder. The intuition behind the module is the following: as patches with high entropy contain more information, they need an encoder with more parameters, unlike low entropy patches, which can be processed using a small encoder. Consequently, processing part of the patches via the smaller encoder can significantly reduce the computational cost of the module. Experiments show that EPE can boost the performance of existing real-time semantic segmentation models with a slight increase in the computational cost. Specifically, EPE increases the mIOU performance of DFANet A by 0.9% with only 1.2% increase in the number of parameters and the mIOU performance of EDANet by 1% with 10% increase of the model parameters.