A partition-based similarity for classification distributions

TL;DR

This paper introduces a new measure called task similarity to quantify how well a representation optimized for one classification distribution performs on another, aiding transfer learning and understanding distribution relationships.

Contribution

It proposes a novel, principled similarity measure between classification distributions based on optimal representations and transformations, with practical implications for transfer learning.

Findings

Empirical task similarity correlates with transfer efficiency.

Task similarity depends on the decision rule used for inference.

The measure captures relationships between source and target distributions.

Abstract

Herein we define a measure of similarity between classification distributions that is both principled from the perspective of statistical pattern recognition and useful from the perspective of machine learning practitioners. In particular, we propose a novel similarity on classification distributions, dubbed task similarity, that quantifies how an optimally-transformed optimal representation for a source distribution performs when applied to inference related to a target distribution. The definition of task similarity allows for natural definitions of adversarial and orthogonal distributions. We highlight limiting properties of representations induced by (universally) consistent decision rules and demonstrate in simulation that an empirical estimate of task similarity is a function of the decision rule deployed for inference. We demonstrate that for a given target distribution, both transfer efficiency and semantic similarity of candidate source distributions correlate with empirical task similarity.