Reprogramming Language Models for Molecular Representation Learning

TL;DR

This paper introduces R2DL, a novel adversarial reprogramming algorithm that leverages dictionary learning to adapt pretrained language models for molecular tasks, outperforming existing models especially with limited data.

Contribution

The paper proposes R2DL, a new method for reprogramming language models to molecular tasks using dictionary learning, enabling effective transfer learning across domains.

Findings

R2DL matches state-of-the-art toxicity prediction models.

R2DL outperforms baselines with limited training data.

Demonstrates domain-agnostic transfer learning for molecular data.

Abstract

Recent advancements in transfer learning have made it a promising approach for domain adaptation via transfer of learned representations. This is especially when relevant when alternate tasks have limited samples of well-defined and labeled data, which is common in the molecule data domain. This makes transfer learning an ideal approach to solve molecular learning tasks. While Adversarial reprogramming has proven to be a successful method to repurpose neural networks for alternate tasks, most works consider source and alternate tasks within the same domain. In this work, we propose a new algorithm, Representation Reprogramming via Dictionary Learning (R2DL), for adversarially reprogramming pretrained language models for molecular learning tasks, motivated by leveraging learned representations in massive state of the art language models. The adversarial program learns a linear transformation between a dense source model input space (language data) and a sparse target model input space (e.g., chemical and biological molecule data) using a k-SVD solver to approximate a sparse representation of the encoded data, via dictionary learning. R2DL achieves the baseline established by state of the art toxicity prediction models trained on domain-specific data and outperforms the baseline in a limited training-data setting, thereby establishing avenues for domain-agnostic transfer learning for tasks with molecule data.