# High performance data integration for large-scale analyses of incomplete Omic profiles using Batch-Effect Reduction Trees (BERT)

## TL;DR

BERT is a high-performance method for integrating incomplete omics data, addressing missing values and batch effects across large-scale datasets.

## Contribution

BERT introduces a novel algorithm for integrating incomplete omic profiles with improved performance and scalability.

## Key findings

- BERT retains up to five orders of magnitude more numeric values compared to existing methods.
- BERT achieves up to 11× runtime improvement using multi-core and distributed-memory systems.
- BERT improves average-silhouette-width by up to 2× when considering covariates and reference measurements.

## Abstract

Data from high-throughput technologies assessing global patterns of biomolecules (omic data), is often afflicted with missing values and with measurement-specific biases (batch-effects), that hinder the quantitative comparison of independently acquired datasets. This work introduces batch-effect reduction trees (BERT), a high-performance method for data integration of incomplete omic profiles. We characterize BERT on large-scale data integration tasks with up to 5000 datasets from simulated and experimental data of different quantification techniques and omic types (proteomics, transcriptomics, metabolomics) as well as other datatypes e.g., clinical data, emphasizing the broad scope of the algorithm. Compared to the only available method for integration of incomplete omic data, HarmonizR, our method (1) retains up to five orders of magnitude more numeric values, (2) leverages multi-core and distributed-memory systems for up to 11 × runtime improvement (3) considers covariates and reference measurements to account for severely imbalanced or sparsely distributed conditions (up to 2 × improvement of average-silhouette-width).

This study presents BERT, an algorithm for high-performance integration of incomplete omics data with robustness to unequal phenotype distribution. It validates the method on simulated and experimental data from proteomics, metabolomics and transcriptomics.