Deep Model Transferability from Attribution Maps

TL;DR

This paper introduces a fast, annotation-free method to estimate transferability between deep vision networks by comparing attribution maps, preserving task relationships more efficiently than existing approaches.

Contribution

It proposes a novel, simple approach to measure transferability using attribution maps, avoiding annotations and architecture constraints, and significantly reducing computational cost.

Findings

Method is several times faster than taskonomy.

Preserves task-wise topological structure similar to taskonomy.

Requires no human annotations or architecture constraints.

Abstract

Exploring the transferability between heterogeneous tasks sheds light on their intrinsic interconnections, and consequently enables knowledge transfer from one task to another so as to reduce the training effort of the latter. In this paper, we propose an embarrassingly simple yet very efficacious approach to estimating the transferability of deep networks, especially those handling vision tasks. Unlike the seminal work of taskonomy that relies on a large number of annotations as supervision and is thus computationally cumbersome, the proposed approach requires no human annotations and imposes no constraints on the architectures of the networks. This is achieved, specifically, via projecting deep networks into a model space, wherein each network is treated as a point and the distances between two points are measured by deviations of their produced attribution maps. The proposed approach is several-magnitude times faster than taskonomy, and meanwhile preserves a task-wise topological structure highly similar to the one obtained by taskonomy. Code is available at https://github.com/zju-vipa/TransferbilityFromAttributionMaps.