# Representation Similarity Analysis for Efficient Task taxonomy & Transfer Learning

## TL;DR

This paper introduces a Representation Similarity Analysis (RSA) method to efficiently assess relationships between visual tasks and models, enabling better task taxonomy and model selection for transfer learning without additional training.

## Contribution

The paper proposes a novel RSA-based approach to evaluate task-model relationships, improving transfer learning efficiency and accuracy in task taxonomy without extra training.

## Key findings

- RSA scores correlate with transfer learning performance
- The method effectively generates task taxonomy on Taskonomy dataset
- It successfully identifies high-performing models for Pascal VOC segmentation

## Abstract

Transfer learning is widely used in deep neural network models when there are few labeled examples available. The common approach is to take a pre-trained network in a similar task and finetune the model parameters. This is usually done blindly without a pre-selection from a set of pre-trained models, or by finetuning a set of models trained on different tasks and selecting the best performing one by cross-validation. We address this problem by proposing an approach to assess the relationship between visual tasks and their task-specific models. Our method uses Representation Similarity Analysis (RSA), which is commonly used to find a correlation between neuronal responses from brain data and models. With RSA we obtain a similarity score among tasks by computing correlations between models trained on different tasks. Our method is efficient as it requires only pre-trained models, and a few images with no further training. We demonstrate the effectiveness and efficiency of our method for generating task taxonomy on Taskonomy dataset. We next evaluate the relationship of RSA with the transfer learning performance on Taskonomy tasks and a new task: Pascal VOC semantic segmentation. Our results reveal that models trained on tasks with higher similarity score show higher transfer learning performance. Surprisingly, the best transfer learning result for Pascal VOC semantic segmentation is not obtained from the pre-trained model on semantic segmentation, probably due to the domain differences, and our method successfully selects the high performing models.