Python Tool for Visualizing Variability of Pareto Fronts over Multiple Runs

TL;DR

This paper introduces a Python tool that visualizes the variability of Pareto fronts across multiple runs in multi-objective optimization, aiding in performance stability assessment.

Contribution

The paper presents a novel Python package for visualizing empirical attainment surfaces, addressing the lack of tools for uncertainty visualization in Pareto front analysis.

Findings

Enables visualization of Pareto front variability across runs

Facilitates performance stability evaluation in MOO

Provides an accessible Python package for researchers

Abstract

Hyperparameter optimization is crucial to achieving high performance in deep learning. On top of the performance, other criteria such as inference time or memory requirement often need to be optimized due to some practical reasons. This motivates research on multi-objective optimization (MOO). However, Pareto fronts of MOO methods are often shown without considering the variability caused by random seeds and this makes the performance stability evaluation difficult. Although there is a concept named empirical attainment surface to enable the visualization with uncertainty over multiple runs, there is no major Python package for empirical attainment surface. We, therefore, develop a Python package for this purpose and describe the usage. The package is available at https://github.com/nabenabe0928/empirical-attainment-func.