Ontology Pre-training for Poison Prediction

TL;DR

This paper introduces ontology pre-training, a method that embeds structured human knowledge into neural networks, improving toxicity prediction accuracy, interpretability, and training efficiency in life sciences chemistry.

Contribution

The paper presents a novel ontology pre-training approach that integrates ontological knowledge into Transformer networks for toxicity prediction tasks.

Findings

Improved toxicity prediction accuracy over state-of-the-art methods.

Model attention focuses on more meaningful chemical groups.

Reduced training time after ontology pre-training.

Abstract

Integrating human knowledge into neural networks has the potential to improve their robustness and interpretability. We have developed a novel approach to integrate knowledge from ontologies into the structure of a Transformer network which we call ontology pre-training: we train the network to predict membership in ontology classes as a way to embed the structure of the ontology into the network, and subsequently fine-tune the network for the particular prediction task. We apply this approach to a case study in predicting the potential toxicity of a small molecule based on its molecular structure, a challenging task for machine learning in life sciences chemistry. Our approach improves on the state of the art, and moreover has several additional benefits. First, we are able to show that the model learns to focus attention on more meaningful chemical groups when making predictions with ontology pre-training than without, paving a path towards greater robustness and interpretability. Second, the training time is reduced after ontology pre-training, indicating that the model is better placed to learn what matters for toxicity prediction with the ontology pre-training than without. This strategy has general applicability as a neuro-symbolic approach to embed meaningful semantics into neural networks.