ControllableGPT: A Ground-Up Designed Controllable GPT for Molecule Optimization

TL;DR

ControllableGPT is a novel unified language model designed for precise, controllable molecule optimization, integrating multiple training approaches to improve drug design tasks with demonstrated superior performance.

Contribution

It introduces a ground-up design combining MLM, CLM, and seq2seq into a controllable GPT framework tailored for drug optimization.

Findings

Outperforms baseline models on viral and cancer drug optimization benchmarks.

Enables precise control over sequence expansion, reduction, and mutation.

Demonstrates effective controllability and optimization in molecular tasks.

Abstract

Large Language Models (LLMs) employ three popular training approaches: Masked Language Models (MLM), Causal Language Models (CLM), and Sequence-to-Sequence Models (seq2seq). However, each approach has its strengths and limitations, and faces challenges in addressing specific tasks that require controllable and bidirectional generation, such as drug optimization. To address this challenge, inspired by the biological processes of growth and evolution, which involve the expansion, shrinking, and mutation of sequences, we introduce ControllableGPT. This initiative represents the first effort to combine the advantages of MLM, CLM, and seq2seq into a single unified, controllable GPT framework. It enables the precise management of specific locations and ranges within a sequence, allowing for expansion, reduction, or mutation over chosen or random lengths, while maintaining the integrity of any specified positions or subsequences. In this work, we designed ControllableGPT for drug optimization from the ground up, which included proposing the Causally Masked Seq2seq (CMS) objective, developing the training corpus, introducing a novel pre-training approach, and devising a unique generation process. We demonstrate the effectiveness and controllability of ControllableGPT by conducting experiments on drug optimization tasks for both viral and cancer benchmarks, surpassing competing baselines.