# KADAIF: an anomaly detection method for complex microbiome data

## TL;DR

KADAIF is a new method for detecting anomalies in complex microbiome data, improving accuracy and helping with health studies.

## Contribution

KADAIF generalizes Isolation Forest for microbiome data by incorporating feature subsets and dimensionality reduction.

## Key findings

- KADAIF outperforms existing methods in detecting anomalies in microbiome datasets.
- KADAIF successfully identifies disease onset in longitudinal microbiome data.
- KADAIF improves case-control partitioning based on the Anna Karenina principle.

## Abstract

The gut microbiome plays an important role in human health and disease, prompting large-scale studies that generate extensive datasets. A critical preprocessing step in analysing such datasets is anomaly detection, which aims to identify erroneous samples and prevent misleading statistical outcomes. Microbiome data, however, pose unique challenges such as compositionality, sparsity, interdependencies, and high dimensionality, limiting the effectiveness of conventional methods and highlighting the need for specifically-tailored approaches for anomaly detection in microbiome data.

To address this challenge, we introduce KADAIF, a microbiome-specific anomaly detection method that generalizes the common Isolation Forest (IF) approach. As in IF, KADAIF builds an ensemble of trees, each recursively partitioning the data along randomly selected features, and measures the average depth at which samples are isolated, assuming that anomalous samples will be isolated closer to the root. Unlike IF, however, KADAIF partitions samples based on subsets of features (coupled with dimensionality reduction), addressing microbiome-specific properties such as sparsity and species interactions.

We evaluate KADAIF by simulating common scenarios that introduce anomalous behavior, demonstrating that KADAIF outperforms alternative methods across various settings and datasets. Furthermore, we show that KADAIF outperforms IF in detecting anomalies also in other types of high-dimensional sparse biological data. Finally, we show KADAIF’s application for identifying disease onset in longitudinal microbiome data and for partitioning cases versus controls based on the Anna Karenina principle. Combined, our work highlights KADAIF's potential to enhance microbiome data processing and downstream analyses, with beneficial implications for precision medicine studies.

An implementation of KADAIF, as well as all the code used for the analysis, is available on GitHub (https://github.com/borenstein-lab/KADAIF).