Visual annotations and a supervised learning approach for evaluating and calibrating ChIP-seq peak detectors

TL;DR

This paper introduces a supervised learning method using visual annotations to evaluate and calibrate ChIP-seq peak detection algorithms, enabling more accurate and dataset-specific peak calling.

Contribution

It presents a novel supervised approach leveraging expert annotations to benchmark and calibrate peak detectors for ChIP-seq data analysis.

Findings

Macs performs best for narrow peaks (H3K4me3)

Hmcan.broad excels with broad peaks (H3K36me3)

Annotated datasets are publicly available for benchmarking

Abstract

Many peak detection algorithms have been proposed for ChIP-seq data analysis, but it is not obvious which method and what parameters are optimal for any given data set. In contrast, peaks can easily be located by visual inspection of profile data on a genome browser. We thus propose a supervised machine learning approach to ChIP-seq data analysis, using annotated regions that encode an expert's qualitative judgments about which regions contain or do not contain peaks. The main idea is to manually annotate a small subset of the genome, and then learn a model that makes consistent predictions on the rest of the genome. We show how our method can be used to quantitatively calibrate and benchmark the performance of peak detection algorithms on specific data sets. We compare several peak detectors on 7 annotated region data sets, consisting of 2 histone marks, 4 expert annotators, and several different cell types. In these data the macs algorithm was best for a narrow peak histone profile (H3K4me3) while the hmcan.broad algorithm was best for a broad histone profile (H3K36me3). Our benchmark annotated region data sets can be downloaded from a public website, and there is an R package for computing the annotation error on GitHub.