A Multi-objective Optimization Benchmark Test Suite for Real-time Semantic Segmentation

TL;DR

This paper introduces a benchmark test suite, CitySeg/MOP, for multi-objective optimization in hardware-aware neural architecture search for real-time semantic segmentation, facilitating evaluation of algorithms balancing accuracy, speed, and hardware constraints.

Contribution

It develops a novel benchmark suite, CitySeg/MOP, transforming HW-NAS for semantic segmentation into standard multi-objective problems and integrating it into the EvoXBench platform.

Findings

Benchmark suite covers 15 MOPs from Cityscapes dataset.

Demonstrates versatility across various multi-objective evolutionary algorithms.

Provides seamless interface for fitness evaluations in multiple programming languages.

Abstract

As one of the emerging challenges in Automated Machine Learning, the Hardware-aware Neural Architecture Search (HW-NAS) tasks can be treated as black-box multi-objective optimization problems (MOPs). An important application of HW-NAS is real-time semantic segmentation, which plays a pivotal role in autonomous driving scenarios. The HW-NAS for real-time semantic segmentation inherently needs to balance multiple optimization objectives, including model accuracy, inference speed, and hardware-specific considerations. Despite its importance, benchmarks have yet to be developed to frame such a challenging task as multi-objective optimization. To bridge the gap, we introduce a tailored streamline to transform the task of HW-NAS for real-time semantic segmentation into standard MOPs. Building upon the streamline, we present a benchmark test suite, CitySeg/MOP, comprising fifteen MOPs derived from the Cityscapes dataset. The CitySeg/MOP test suite is integrated into the EvoXBench platform to provide seamless interfaces with various programming languages (e.g., Python and MATLAB) for instant fitness evaluations. We comprehensively assessed the CitySeg/MOP test suite on various multi-objective evolutionary algorithms, showcasing its versatility and practicality. Source codes are available at https://github.com/EMI-Group/evoxbench.