A critical assessment of conformal prediction methods applied in binary classification settings

TL;DR

This paper critically evaluates conformal prediction methods in binary classification, highlighting potential pitfalls and providing guidance for their proper application in drug discovery research.

Contribution

It offers a comprehensive review of conformal prediction theory, examines current dominant methods in drug discovery, and assesses their limitations through case studies.

Findings

Identifies pitfalls in current conformal prediction applications

Provides critical assessment of methods in binary settings

Guides proper use of conformal predictions in drug discovery

Abstract

In recent years there has been an increase in the number of scientific papers that suggest using conformal predictions in drug discovery. We consider that some versions of conformal predictions applied in binary settings are embroiled in pitfalls, not obvious at first sight, and that it is important to inform the scientific community about them. In the paper we first introduce the general theory of conformal predictions and follow with an explanation of the versions currently dominant in drug discovery research today. Finally, we provide cases for their critical assessment in binary classification settings.