Transferability Estimation for Semantic Segmentation Task

TL;DR

This paper extends an analytical transferability metric, OTCE, to semantic segmentation tasks, enabling efficient prediction of transfer performance without full transfer, validated on multiple datasets.

Contribution

It adapts the OTCE transferability score for semantic segmentation by sampling pixels to handle high-dimensional outputs, addressing a gap in transferability estimation methods.

Findings

OTCE score correlates highly with actual transfer performance.

Sampling pixels reduces computational cost effectively.

Method validated on Cityscapes, BDD100K, and GTA5 datasets.

Abstract

Transferability estimation is a fundamental problem in transfer learning to predict how good the performance is when transferring a source model (or source task) to a target task. With the guidance of transferability score, we can efficiently select the highly transferable source models without performing the real transfer in practice. Recent analytical transferability metrics are mainly designed for image classification problem, and currently there is no specific investigation for the transferability estimation of semantic segmentation task, which is an essential problem in autonomous driving, medical image analysis, etc. Consequently, we further extend the recent analytical transferability metric OTCE (Optimal Transport based Conditional Entropy) score to the semantic segmentation task. The challenge in applying the OTCE score is the high dimensional segmentation output, which is difficult to find the optimal coupling between so many pixels under an acceptable computation cost. Thus we propose to randomly sample N pixels for computing OTCE score and take the expectation over K repetitions as the final transferability score. Experimental evaluation on Cityscapes, BDD100K and GTA5 datasets demonstrates that the OTCE score highly correlates with the transfer performance.