What an Autonomous Agent Discovers About Molecular Transformer Design: Does It Transfer?

TL;DR

This study investigates whether transformer architectures optimized via autonomous search for molecular sequences differ from those for natural language, revealing transferability and domain-specific insights.

Contribution

It systematically compares architecture search versus hyperparameter tuning across molecular and language domains, highlighting transferability of innovations.

Findings

Architecture search is counterproductive for SMILES; tuning hyperparameters suffices.

Natural language benefits significantly from architecture changes, improving performance.

Discovered architectures are domain-specific but transfer well with minimal degradation.

Abstract

Deep learning models for drug-like molecules and proteins overwhelmingly reuse transformer architectures designed for natural language, yet whether molecular sequences benefit from different designs has not been systematically tested. We deploy autonomous architecture search via an agent across three sequence types (SMILES, protein, and English text as control), running 3,106 experiments on a single GPU. For SMILES, architecture search is counterproductive: tuning learning rates and schedules alone outperforms the full search (p = 0.001). For natural language, architecture changes drive 81% of improvement (p = 0.009). Proteins fall between the two. Surprisingly, although the agent discovers distinct architectures per domain (p = 0.004), every innovation transfers across all three domains with <1% degradation, indicating that the differences reflect search-path dependence rather than fundamental biological requirements. We release a decision framework and open-source toolkit for molecular modeling teams to choose between autonomous architecture search and simple hyperparameter tuning.