Using a Cross-Task Grid of Linear Probes to Interpret CNN Model Predictions On Retinal Images

TL;DR

This study employs a comprehensive grid of linear probes across multiple retinal image tasks to interpret CNN representations, revealing layer-wise generalizability and task correlations in the UK Biobank dataset.

Contribution

Introduces a cross-task linear probe framework to analyze CNN representations across 93 retinal image tasks, uncovering insights into layer-wise generalization and task relationships.

Findings

Middle layers produce more generalizable features.

Some tasks are predictable regardless of source.

Certain tasks are better predicted from correlated source tasks.

Abstract

We analyze a dataset of retinal images using linear probes: linear regression models trained on some "target" task, using embeddings from a deep convolutional (CNN) model trained on some "source" task as input. We use this method across all possible pairings of 93 tasks in the UK Biobank dataset of retinal images, leading to ~164k different models. We analyze the performance of these linear probes by source and target task and by layer depth. We observe that representations from the middle layers of the network are more generalizable. We find that some target tasks are easily predicted irrespective of the source task, and that some other target tasks are more accurately predicted from correlated source tasks than from embeddings trained on the same task.