In-Context Learning for Few-Shot Molecular Property Prediction
Christopher Fifty, Jure Leskovec, Sebastian Thrun
TL;DR
This paper introduces a novel in-context learning algorithm for few-shot molecular property prediction, enabling rapid adaptation to new properties without fine-tuning, outperforming recent meta-learning methods on key benchmarks.
Contribution
It adapts in-context learning concepts from NLP to molecular property prediction, providing a new few-shot learning algorithm that works across different support sizes without fine-tuning.
Findings
Outperforms recent meta-learning algorithms at small support sizes
Competitive with best methods at large support sizes
Effective in rapid adaptation to new molecular properties
Abstract
In-context learning has become an important approach for few-shot learning in Large Language Models because of its ability to rapidly adapt to new tasks without fine-tuning model parameters. However, it is restricted to applications in natural language and inapplicable to other domains. In this paper, we adapt the concepts underpinning in-context learning to develop a new algorithm for few-shot molecular property prediction. Our approach learns to predict molecular properties from a context of (molecule, property measurement) pairs and rapidly adapts to new properties without fine-tuning. On the FS-Mol and BACE molecular property prediction benchmarks, we find this method surpasses the performance of recent meta-learning algorithms at small support sizes and is competitive with the best methods at large support sizes.
Peer Reviews
Decision·ICLR 2024 Conference Withdrawn Submission
1. The formulation of in-context learning for molecular property prediction is interesting as the transformer encoder learns to extracts dynamic representations for the support (molecule, property) pairs alongside the query molecules, which is more powerful than the static representations in metric-based methods and more efficient than gradient-based methods. 2. The analysis and visualization of the embeddings and attention weights of CAMP are helpful for understanding the properties of the lear
1. My main concern is the experiments. In the abstract, the paper claims that > On the FS-Mol and BACE molecular property prediction benchmarks, we find this method surpasses the performance of recent meta-learning algorithms at small support sizes and is competitive with the best methods at large support sizes. However, the above claim is not an accurate description of the results in the paper, because the experimental setup in Section 6.1 is different from the setup of the FS-Mol benchmark.
1. The studied problem is interesting and important. The clarity of writing and structure of the paper effectively convey the methodology to the reader. 2. The proposed method is simple but effective, enabling in-context learning as a viable approach for molecular property prediction.
1. The proposed framework seems to overlap significantly with another anonymous submission[1], especially Figure 1 in both papers. Not sure whether they are from the same authors, but it clearly dilutes the technical contribution of the current work. 2. Missing critical baselines. Given that few-shot molecular property prediction is not a new research topic, there are some previous works proposed specifically for this problem, such as [2]. It is crucial to include the comparisons with these bas
1.This paper is well-written. 2.This paper gives a solution for few-shot molecular property prediction with in-context learning. Empirical results on different datasets and analysis are provided.
The main concern is lack of novelty nor technical highlights. It looks like a simple combination of some existing approaches, directly applying the in-context learning in few-shot molecular property prediction problems. From the perspective of few-shot learning, such a standard “encoder-transformer-mlp” framework can be applied for any few-shot classification problem, and ANALYSIS OF LEARNING MECHANISMS section proves it is very similar with the mechanism of [1]. From the perspective of molecul
(1) The basic idea of the proposed method is clearly illustrated and the fundamental structure of the paper is well-organized; (2) The addressed few-shot molecular property prediction problem is an important challenge for drug discovery applications. This reviewer acknowledges the significance of the problem.
(1) The novelty of the proposed method is kind of limited. It seems the core idea is learning the relationships between support samples and query samples, which is very similar to PAR [1] in nature. PAR learns the relation graph across support samples and query samples. It seems this work only replaces the relation graph with the self-attention. The label encoding could be regarded as an alternative implementation of prototypes; Therefore, according to this reviewer's understanding, CAMP is just
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMachine Learning in Materials Science · Computational Drug Discovery Methods · Fuel Cells and Related Materials