scTree: Discovering Cellular Hierarchies in the Presence of Batch Effects in scRNA-seq Data

TL;DR

scTree is a novel VAE-based method that simultaneously corrects batch effects and uncovers hierarchical cellular structures in single-cell RNA sequencing data, improving clustering accuracy and biological interpretability.

Contribution

It introduces a hierarchical clustering approach integrated with batch correction in a VAE framework, advancing analysis of complex scRNA-seq data.

Findings

Outperforms baseline methods across seven datasets

Recovers biologically relevant cellular hierarchies

Effectively corrects batch effects during clustering

Abstract

We propose a novel method, scTree, for single-cell Tree Variational Autoencoders, extending a hierarchical clustering approach to single-cell RNA sequencing data. scTree corrects for batch effects while simultaneously learning a tree-structured data representation. This VAE-based method allows for a more in-depth understanding of complex cellular landscapes independently of the biasing effects of batches. We show empirically on seven datasets that scTree discovers the underlying clusters of the data and the hierarchical relations between them, as well as outperforms established baseline methods across these datasets. Additionally, we analyze the learned hierarchy to understand its biological relevance, thus underpinning the importance of integrating batch correction directly into the clustering procedure.