Effects of Distance Metrics and Scaling on the Perturbation Discrimination Score

TL;DR

This paper investigates how the Perturbation Discrimination Score (PDS) is affected by different distance metrics and scaling methods in high-dimensional gene-expression data, revealing significant sensitivities and differences among measures.

Contribution

It provides a detailed analysis of PDS behavior in high-dimensional settings, highlighting the impact of metric choice and scaling, and offers geometric insights for future evaluation metric development.

Findings

PDS sensitivity varies significantly with distance measure and scaling.

$\, ext{l}_1$ and $ ext{l}_2$-based PDS differ from cosine-based measures.

Geometric analysis explains differences and guides metric selection.

Abstract

The Perturbation Discrimination Score (PDS) is increasingly used to evaluate whether predicted perturbation effects remain distinguishable, including in Systema and the Virtual Cell Challenge. However, its behavior in high-dimensional gene-expression settings has not been examined in detail. We show that PDS is highly sensitive to the choice of similarity or distance measure and to the scale of predicted effects. Analysis of observed perturbation responses reveals that $\ell_1$ and $\ell_2$-based PDS behave very differently from cosine-based measures, even after norm matching. We provide geometric insight and discuss implications for future discrimination-based evaluation metrics.