Learning Stable Predictors from Weak Supervision under Distribution Shift

TL;DR

This paper investigates the robustness of weak supervision methods under distribution shifts, especially temporal changes, using CRISPR-Cas13d transcriptomic data across different cell lines and timepoints.

Contribution

It formalizes supervision drift as changes in $P(y \,|\, x, c)$ across contexts and provides a benchmark demonstrating the limitations of weak supervision under temporal shifts.

Findings

Weak supervision supports in-domain learning and partial transfer across cell lines.

Temporal transfer under supervision drift collapses, leading to poor performance.

Feature importance stability varies across time, indicating supervision drift causes transfer failures.

Abstract

Learning from weak, proxy, or relative supervision is common when ground-truth labels are unavailable, but robustness under distribution shift remains poorly understood because the supervision mechanism itself may change across environments. We formalize this phenomenon as supervision drift, defined as changes in $P(y \mid x, c)$ across contexts, and study it in CRISPR-Cas13d transcriptomic perturbation experiments where guide efficacy is inferred indirectly from RNA-seq responses. Using publicly available data spanning two human cell lines and multiple post-induction timepoints, we construct a controlled non-IID benchmark with explicit domain (cell line) and temporal shifts, while reusing a fixed weak-label construction across all contexts to avoid changing targets. Across linear and tree-based models, weak supervision supports meaningful learning in-domain (ridge $R^2 = 0.356$, Spearman $\rho = 0.442$) and partial cross-cell-line transfer ($\rho \approx 0.40$). In contrast, temporal transfer collapses across all model classes considered, yielding negative $R^2$ and weak or near-zero $\rho$ (ridge $R^2 = -0.145$, $\rho = 0.008$; XGBoost $R^2 = -0.155$, $\rho = 0.056$; random forest $R^2 = -0.322$, $\rho = 0.139$). Additional robustness analyses using externally recomputed weak labels, shift-score quantification, and simple mitigation baselines preserve the same qualitative pattern. Feature-label association and feature-importance analyses remain relatively stable across cell lines but change sharply over time, indicating that failures arise from supervision drift rather than model capacity or simple covariate shift. These results show that strong in-domain performance under weak supervision can be misleading and motivate feature stability as a lightweight diagnostic for non-transferability before deployment.