Lo-Hi: Practical ML Drug Discovery Benchmark

TL;DR

The paper introduces Lo-Hi, a practical drug discovery benchmark with two tasks that better reflect real-world applications, along with a novel molecular splitting algorithm and an analysis of existing benchmarks' limitations.

Contribution

It presents the Lo-Hi benchmark for drug discovery, including a new molecular splitting method and an evaluation of model performance in realistic settings.

Findings

Modern benchmarks are overly optimistic and unrealistic.

State-of-the-art models vary in effectiveness under practical conditions.

The Lo-Hi benchmark provides more realistic evaluation scenarios.

Abstract

Finding new drugs is getting harder and harder. One of the hopes of drug discovery is to use machine learning models to predict molecular properties. That is why models for molecular property prediction are being developed and tested on benchmarks such as MoleculeNet. However, existing benchmarks are unrealistic and are too different from applying the models in practice. We have created a new practical \emph{Lo-Hi} benchmark consisting of two tasks: Lead Optimization (Lo) and Hit Identification (Hi), corresponding to the real drug discovery process. For the Hi task, we designed a novel molecular splitting algorithm that solves the Balanced Vertex Minimum $k$-Cut problem. We tested state-of-the-art and classic ML models, revealing which works better under practical settings. We analyzed modern benchmarks and showed that they are unrealistic and overoptimistic. Review: https://openreview.net/forum?id=H2Yb28qGLV Lo-Hi benchmark: https://github.com/SteshinSS/lohi_neurips2023 Lo-Hi splitter library: https://github.com/SteshinSS/lohi_splitter