Exploring Modularity of Agentic Systems for Drug Discovery

TL;DR

This paper investigates the modularity of LLM-based agentic systems in drug discovery, comparing different models and agent types, and highlighting the importance of re-engineering when replacing components for reliable solutions.

Contribution

It provides the first systematic analysis of modularity in agentic systems for drug discovery, comparing various LLMs and agent types, and demonstrating the dependency of performance on specific configurations.

Findings

Certain LLMs outperform others in drug discovery tasks.

Code-generating agents generally outperform tool-calling agents, but results vary.

Component replacement effects depend on the question and model.

Abstract

Large-language models (LLMs) and agentic systems present exciting opportunities to accelerate drug discovery. In this study, we examine the modularity of LLM-based agentic systems for drug discovery, i.e., whether parts of the system such as the LLM and type of agent are interchangeable, a topic that has received limited attention in drug discovery. We compare the performance of different LLMs and the effectiveness of tool-calling agents versus code-generating agents. Our case study, comparing performance in orchestrating tools for chemistry and drug discovery using an LLM-as-a-judge score, shows that Claude-3.5-Sonnet, Claude-3.7-Sonnet and GPT-4o outperform alternative language models such as Llama-3.1-8B, Llama-3.1-70B, GPT-3.5-Turbo, and Nova-Micro. Although we confirm that code-generating agents outperform the tool-calling ones on average, we show that this is highly question- and model-dependent. Furthermore, the impact of replacing system prompts is dependent on the question and model, underscoring that even in this particular domain one cannot just replace components of the system without re-engineering. Our study highlights the necessity of further research into the modularity of agentic systems to enable the development of reliable and modular solutions for real-world problems.