# Multi-view deep learning of highly multiplexed imaging data improves association of cell states with clinical outcomes

## TL;DR

This paper introduces a deep learning method to better understand cell states using multiple types of data from imaging, improving predictions of clinical outcomes.

## Contribution

The novel contribution is a multi-modal variational autoencoder that integrates multiple imaging data views to better associate cell states with clinical outcomes.

## Key findings

- The integrated multi-modal latent space is more associated with clinical outcomes than existing baselines.
- Ablation analyses reveal which input views most contribute to model performance.
- The method enables cellular representations that align with phenotypes and integrate diverse datasets.

## Abstract

Analysis workflows for highly multiplexed imaging technologies typically summarize each cell in terms of its post-segmentation mean expression, but additional cellular information can be quantified including cell morphology, sub-cellular expression patterns, and spatial cellular context, ultimately giving a multi-modal view of each cell. While deep learning models such as variational autoencoders are well-established for other multi-modal single-cell assays, their ability to integrate these multiple views of a cell from highly multiplexed imaging data remains largely unknown.

Here, we explore the abilities of multi-modal variational autoencoders to learn unified latent cellular representations from multiple views of each single-cell quantified from highly multiplexed imaging, including mean expression, morphology, sub-cellular protein co-localization, and spatial cellular context, while conditioning on technical and batch specific effects. We show that the integrated multi-modal latent space is often more associated with patient-specific clinical outcomes compared to a set of existing baselines. In addition, we perform ablation analyses to understand which input views contribute to model performance, and explore the ability of these models to learn cellular representations that align with cellular phenotypes and enable integration across divergent datasets.

hmiVAE is implemented as a python package and is available at https://github.com/camlab-bioml/hmiVAE