The role of graph topology in the performance of biomedical Knowledge Graph Completion models

TL;DR

This paper explores how the structure of biomedical knowledge graphs affects the performance of models used to complete missing information in these graphs.

Contribution

The study introduces a new analysis framework linking graph topology to model accuracy in biomedical knowledge graph completion.

Findings

The topological properties of biomedical knowledge graphs significantly influence model performance.

Publicly available datasets and models show varying effectiveness depending on graph structure.

The release of analysis tools encourages further research into biomedical knowledge graph modeling.

Abstract

Knowledge Graph Completion has been increasingly adopted as a useful method for helping address several tasks in biomedical research, such as drug repurposing or drug–target identification. To that end, a variety of datasets and Knowledge Graph Embedding models have been proposed over the years. However, little is known about the properties that render a dataset, and associated modelling choices, useful for a given task. Moreover, even though theoretical properties of Knowledge Graph Embedding models are well understood, their practical utility in this field remains controversial.

In this work, we conduct a comprehensive investigation into the topological properties of publicly available biomedical Knowledge Graphs and establish links to the accuracy observed in real-world tasks. By releasing all model predictions and a new suite of analysis tools we invite the community to build upon our work and continue improving the understanding of these crucial applications.

The code used to perform experiments and analyze results in this article as well as all experimental data is available at https://github.com/graphcore-research/kg-topology-toolbox/tree/main/the_role_of_graph_topology_paper and archived on Zenodo, at https://doi.org/10.5281/zenodo.12097376.

Figures

The four primary edge topological patterns we consider. We show (h,r,t) as the base triple common to all patterns, while the dashed lines with relations r′, r1, or r2, are edges that realize the defining feature of the pattern. We use r′, r1, or r2 to denote relations distinct from r.

Example of edge cardinalities in a KG with two relation types (blue, red). We use the notation degr(t):degr(h). For example, for the red edge with cardinality 1: M, the head has more than one outgoing edges of the same relation type (red) and the tail has a single red incoming edge.

Counts of composition patterns for key relation types within Hetionet and PrimeKG. Values above the bars indicate the number of distinct pairs of relation types in the composition.

Occurrence of edge cardinalities in the datasets.

Relative frequency of triples when grouped by head out-degree and tail in-degree of the same relation type.

Mean reciprocal rank on the test split achieved by different KGE models, for the six datasets.

Effect of edge cardinality on MRR.

Effect of the head and tail degrees of same relation type on MRR.

The effect of having compositions on MRR, triples grouped by their head and tail degrees of same relation type.

MRR of specific interaction types.

Comparison of models trained on Hetionet and PharMeBINet when testing MRR on a set of common edges.

Tables

degr(h)∖degr(t)	=1	>1
=1	one-to-one (1:1)	many-to-one (M : 1)
>1	one-to-many (1: M)	many-to-many (M: M)

Table 1.. The fraction of edges with a given topological pattern (symmetry, inference, inverse, composition, or none of the four) in the datasets.a

Graph	Sym	Inf	Inv	Comp	None
Hetionet	0.002	0.124	0.001	0.693	0.298
OpenBioLink	0.317	0.372	0.359	0.840	0.12
PharMeBINet	2.42e−4	0.052	0.002	0.598	0.396
PharmKG	0.197	0.124	0.059	0.651	0.279
PrimeKG	0	2.08e−4	0	0.807	0.193
FB15k-237	0.113	0.161	0.217	0.645	0.287

Full text

1 Introduction

Knowledge Graphs (KGs) have become a powerful way of integrating information about a domain into a single structured representation. KGs represent knowledge as a collection of facts in the form of triples $[eqn]$ , which make a statement about the relationship r between two entities h and t. In KGs from the biomedical domain, entities can be genes, diseases, drugs, or pathways (among others), whilst relations represent biological interactions or associations between them (MacLean 2021, Bonner et al. 2022, Zeng et al. 2022). Biomedical KGs have been used to support a variety of tasks within the complex and costly drug discovery process, such as drug repurposing (Bang et al. 2023), identifying synergistic drug combinations (Zitnik et al. 2018), and target discovery (Paliwal et al. 2020). Unlike in other domains, biomedical KGs frequently comprise information from a wide range of conceptual abstraction levels, for both entities and edges. It is common to find edges representing information with a low abstraction level, such as experimentally validated protein–protein interactions (Protein A, interacts, Protein B), in the same KG as mid abstraction level edges, such as associations between genes and bioprocesses (Gene A, associates, BioProcess X), and high abstraction level edges, such as disease ontology hierarchies (Disease X, subclass of, Disease Y). Entities can also have different abstraction levels, with some representing physical entities (e.g. genes) and others representing conceptual entities (e.g. diseases or biological processes). This makes biomedical KGs uniquely interesting to consider when they are used in combination with Machine Learning algorithms, as models must learn to make predictions across these different abstraction levels simultaneously.

Since KGs, including biomedical ones, are typically incomplete, inferring missing triples is a key Machine Learning application—a task known as Knowledge Graph Completion in the literature. Perhaps the most common method of making predictions on KGs involves using Knowledge Graph Embedding (KGE) models, which learn low dimensional representations for all entities and relation types in a KG and combine them through a scoring function to compute the likelihood of a missing edge. In the biomedical domain, Knowledge Graph Completion has been used for a diverse set of tasks including predicting new links between genes and diseases (Paliwal et al. 2020) and possible drug interactions, among others (Gema et al. 2024).

Biomedical KGs are often constructed by practitioners to address a specific purpose (e.g. investigate the pathways of a given disease), requiring decisions about the key entities and relation types to include, the optimal data sources from which to extract information and how best to model the desired knowledge in a KG schema. This manual graph composition process is costly and time-consuming, an issue that can be compounded when KGs are designed as input for ML models, often requiring several iterations to best tune the graph to the predictive task at hand (Bonner et al. 2022, Ratajczak et al. 2022). A solid understanding of desirable topological properties of KGs that can be leveraged by ML models to make better predictions would allow for a more principled approach to this graph construction process, making it cheaper and faster to generate KGs with higher utility to the practitioner.

Thus, there is a growing interest in understanding how any decisions taken which affect topological properties of a KG can ultimately impact predictions made from them. Amongst other topological properties, patterns such as compositions (Fig. 1d), where a relation between two entities is inferred by the presence of a connecting third entity, can be used to link graph structure to model prediction quality. In the biomedical domain, this could be the case when a drug is inferred to treat a disease because they share a common gene connection. Ideally, we would like to discover new drug repurposing opportunities by learning from such patterns in the data. Similarly, the inference pattern (Fig. 1b) uses the presence of a certain relation type between an entity pair to infer the presence of a different relation between the same pair, which could, for example, be used to discover novel therapeutic gene–disease connections by aggregating multiple evidence types. Another example is in protein–protein interactions (PPIs), where an edge could have a clear directionality if representing information flow in a pathway, or be inherently symmetrical (Fig. 1a) if representing simple binary physical interaction.

The four primary edge topological patterns we consider. We show (h,r,t) as the base triple common to all patterns, while the dashed lines with relations r′, r1, or r2, are edges that realize the defining feature of the pattern. We use r′, r1, or r2 to denote relations distinct from r.

While the theoretical implications of these topological patterns for KGE models are well understood—in fact, new KGE models are often designed to capture key topological patterns—their impact on the predictive performance in real-world Knowledge Graph Completion tasks remains an open question. Recent work has started to investigate this (Ali et al. 2022, Jin et al. 2023, Teneva and Hruschka 2023) but has primarily focused on the properties of relation types rather than individual triples. This approach can help to understand which topological patterns can be captured by the relation embedding when comparing or proposing new KGE models. However, when investigating the impact of the graph structure on KGE model performance, this relation-level approach might neglect critical local properties of the graph. Moreover, existing work has focussed on a set of general-domain KGs, which may contain distinct topological patterns compared to biomedical KGs.

Thus, despite the clear need for understanding the relationship between the topology of KGs and model performance specifically in the biomedical domain, there is a gap in the existing literature, which we aim to address in this work by exploring such relationship at an unprecedented level of detail.

Our main contributions are as follows:

We provide an in-depth analysis of the topological properties of six public KGs, focusing on the biomedical domain, and compare the corresponding predictive performance of five well-established KGE models. While previous work has limited the investigation to the macro-level of relation types, we are able to detect stronger patterns linking topological properties to predictive accuracy by zooming in at the level of individual triples.We look in detail at highly relevant relations that biomedical practitioners are most interested in inferring. We give evidence of the topological differences in how they are represented in different KGs and how this reflects on predictive performance.To aid future work in this area, we propose a standardized framework to describe KG topology and release a dedicated Python package kg-topology-toolbox (https://github.com/graphcore-research/kg-topology-toolbox) offering a wide range of tools to compute topological statistics of any KG. To help bridge the gap between industry and academia, we also make the predictions of the trained models available to the community (Cattaneo et al. 2024) to conduct further analysis.

2 Related work

Since the adoption of representation learning for Knowledge Graph Completion, there has been interest in linking predictive performance of models to topological structures in the underlying KGs (Bordes et al. 2013). These structures, taking into account the heterogeneous relation types in KGs, have been split into two categories: edge cardinalities and topological patterns. For the definitions used in this study see Section 3. Indeed, it has become common for new models to be designed with the capacity of capturing these patterns (Cao et al. 2021, Chao et al. 2021, Sun et al. 2019), working under the assumption that better modelling them will result in improved predictive performance (Jin et al. 2023).

Edge cardinality patterns group relations based on the cardinalities of their head and tail entity sets. Four cardinality patterns have been proposed: one-to-one, one-to-many, many-to-one, and many-to-many (Bordes et al. 2013, Wang et al. 2014, Yang et al. 2015). On the other hand, topological patterns categorize relationships based on underlying logical principles that define how entities interact. Several such patterns have been proposed, with common examples being symmetric/anti-symmetric, inverse, and composition (Sun et al. 2019, Ali et al. 2022). In the literature, it is common to consider these patterns at the level of whole relation types, meaning that all triples with a given relation type must conform to a certain pattern for it to be said to be present.

Whilst both pattern types are frequently discussed when comparing architectures, quantifying how model predictive performance correlates with them is less common. Existing work has typically been limited in scale, often only performed on a new approach when it is proposed. For example, the original TransE paper investigated the impact of cardinality patterns on model performance, finding that models could more easily predict heads in one-to-many relations and tails in many-to-one relations (Bordes et al. 2013). The DistMult model observed similar results, whilst also measuring the ability of the model to capture composition relations (Yang et al. 2015). TransH was found to be better at capturing symmetric relations (Wang et al. 2014). RotatE demonstrated better predictive performance than existing models on composition relation types, whilst the authors also found implicit evidence of other topological patterns in the embeddings (Sun et al. 2019).

There has been limited work providing a comprehensive study on the impact of these patterns across different models and datasets, although some studies have started to appear recently. For example, an investigation across three common benchmark KGs (FB15k-237, WN18RR, and Yago310) and 21 KGE models found that symmetric relations were easier to predict that anti-symmetric and compositions ones (Ali et al. 2022). Another study compared the performance of seven KGE models on FB15k-237 and WN18RR, focussing on four different topological patterns (Jin et al. 2023), finding that a model’s theoretical capacity to capture a certain pattern does not guarantee good predictive performance on it. Taking a different approach, a study showed how adding noise to a graph impacted the ability of TransE to learn the composition pattern (Douglas et al. 2022). However, none of these studies explores the impact of edge patterns on biomedical KGs specifically.

Biomedical KGs can suffer from degree imbalance, which can arise naturally from the biology modelled by the KG (e.g. key hub-genes having a high degree) or artificially (e.g. due to inspection bias where more attention is paid to high degree entities). This degree imbalance can impact downstream models: through perturbation of graph topology, degree alone has been shown to be a strong prior predictor for the existence of edges (Zietz et al. 2024). KGE models are known to be strongly impacted by entity degree, with models often being heavily biased towards highly connected entities (Mohamed et al. 2020, Rossi et al. 2021, Rossi et al. 2021, Bonner et al. 2022). However, disentangling the effect of degree from other topological properties when considering model performance is often overlooked by prior studies. One paper showed that as entity degree increases, performance on symmetric relations decreases (Jin et al. 2023). However, it is still an open research question as to how degree obscures other correlations in the performance of models on different topological patterns.

The prevalence of different topological patterns in real-world datasets is also an active area of research. The frequency of topological and cardinality patterns in 29 real-world KGs from a range of domains, including biomedical was investigated in Teneva and Hruschka (2023). The authors observed that, under their chosen pattern definitions, many relations in biomedical KGs are predominantly many-to-many and that there was a lack of composition relations. Other work investigated the prevalence of symmetric and composition relations in three common KG benchmark datasets, finding that the majority of relations were anti-symmetric and that composition relations were rarer (Ali et al. 2022).

It is also crucial to understand how the presence of topological patterns could impact validation metrics if not considered when creating train/test splits. The benchmark dataset FB15k was found to contain inverse triple pairs where one part was in the train set and its counterpart in the test set, meaning that models could simply memorize pairs (Toutanova and Chen 2015). A new dataset, FB15k-237, was created to remove these train/test set leakages. Another common dataset, WN18, was found to have a similar problem, with the WN18RR dataset created to address this (Toutanova and Chen 2015).

3 KG topological properties

3.1 Topological patterns

In terms of describing KG topology, we focus on both the cardinality of the head and tail of triples, as well as their edge topological patterns. Specifically, for a triple $[eqn]$ in a KG $[eqn]$ we define the head out-degree  $[eqn]$ as the cardinality of the set $[eqn]$ and, analogously, the tail in-degree  $[eqn]$ as the cardinality of $[eqn]$ . We further define the head out-degree of same relation type  $[eqn]$ and the tail in-degree of same relation type  $[eqn]$ as the cardinalities of $[eqn]$ and $[eqn]$ , respectively.

The edge cardinality of a triple $[eqn]$ is then defined as:

**: **

Examples of the four possible edge cardinalities are provided using a small sample graph in Fig. 2.

Example of edge cardinalities in a KG with two relation types (blue, red). We use the notation degr(t):degr(h). For example, for the red edge with cardinality 1: M, the head has more than one outgoing edges of the same relation type (red) and the tail has a single red incoming edge.

We also define the following edge topological patterns for triples $[eqn]$ :

$[eqn]$  is symmetric  $[eqn]$ and $[eqn]$ ; $[eqn]$  has inference  $[eqn]$ ; $[eqn]$  has inverse  $[eqn]$ ; $[eqn]$  has composition  $[eqn]$ and $[eqn]$ .

These edge topological patterns are illustrated in Fig. 1. It is important to note that a triple can satisfy multiple topological patterns simultaneously, or none at all.

While the above cardinalities and topological patterns are well defined for individual triples, previous studies (Bordes et al. 2013, Sun et al. 2019, Jin et al. 2023, Teneva and Hruschka 2023) have considered them as properties of relation types as a whole (e.g. relation r is said to be symmetric if $[eqn]$ ), despite the fact that a given property might be satisfied only by a fraction of the triples of a certain relation. We find that such aggregations lack proper formalization in the literature, with inconsistencies in how it is performed, and is prone to introducing noise in results (see Section 4.2). Thus, we consider all the above defined properties at the individual triple level.

3.2 Topological patterns in biomedical KGs

To demonstrate how topological patterns can be observed in biomedical KGs, we consider examples from Hetionet.

3.2.1 Symmetric edges

Hetionet contains only a small number of symmetric edges (see Table 1), with all examples being of the Gene-regulates-Gene relation type. In this instance, the symmetric edges represent that knockdown/overexpression of one gene significantly dysregulated the other (Himmelstein et al. 2017). The creation of symmetric edges when the underlying data is not inherently directional (e.g. physical binding interactions between proteins) is a composition design decision that can, depending upon the scoring function chosen, be used when a KGE model should not learn anything about the directionality of the edge.

3.2.2 Inference edges

Hetionet has several relation types that contain inference edges, for example, for Gene-Disease links, where the Disease-upregulates-Gene or Disease-downregulates-Gene relation types are inference edges with Disease-associates-Gene. Inference edges appear when there are logically or semantically distinct relationships between the same pair of entities. They can also arise when the relationship is captured in more than one underlying data source. Here, there is logically an association between a given disease and gene if an impact on gene expression has been measured. The relation types are also drawn from different sources (associates from sources like DisGeNet and GWAS Catalog, up/downregulates from a differential expression analysis (Himmelstein et al. 2017)).

3.2.3 Inverse edges

Inverse edges are rare in Hetionet, with all examples arising from combinations of different gene–gene relation types (a pair of genes being linked in one direction by regulation edge and in the other by an interaction edge). These examples may not be intentionally inverse and could be a product of the data integration process. Hypothetical inverse examples (Drug-treats-Disease and Disease-treatedby-Drug for example) could be added to a KG to enable simpler querying by an end user however.

3.2.4 Composition edges

Hetionet contains many relation types that are part of composition patterns (e.g. Compound-binds-Gene being part of the composition pattern Compound-treats-Disease and Disease-associates-Gene), as they naturally arise in biomedical data. To illustrate this further, Fig. 3 shows the counts of composition patterns connecting pairs of specific entity types, where we include PrimeKG in addition to Hetionet for comparison. These counts have been stratified for each edge by the type of the intermediate entity n in the composition pattern, and we show the number of unique patterns containing that entity type. This figure shows the abundance and diversity of this pattern across both datasets, where the most abundant composition edge is made up of three genes.

Counts of composition patterns for key relation types within Hetionet and PrimeKG. Values above the bars indicate the number of distinct pairs of relation types in the composition.

While all these four topological patterns are present in Hetionet, it is worth noting that some of the patterns are inherent to the nature of Biomedical KGs, while others may arise due to construction decisions taken during the creation of the KG.

4 Experimental setup

We investigate five public biomedical KGs: Hetionet (Himmelstein et al. 2017), PrimeKG (Chandak et al. 2023), PharmKG (Zheng et al. 2021), OpenBioLink (Breit et al. 2020), PharMeBINet (Königs et al. 2022) (detailed in Table S1, available as supplementary data at Bioinformatics online). Note that in the original PrimeKG graph for every triple $[eqn]$ the reverse triple $[eqn]$ is present as well; we prepocessed PrimeKG to remove these reverse edges. These datasets are among the most common biomedical KGs used for benchmarking in the literature (Teneva and Hruschka 2023) and were chosen with the aim of encompassing a wide variety of sizes and topological patterns. We also include the trivia KG FB15k-237 (Toutanova and Chen 2015) as a baseline, to detect any results unique to the biomedical domain. On all KGs we train five of the most popular KGE models: TransE (Bordes et al. 2013), DistMult (Yang et al. 2015), RotatE (Sun et al. 2019), TripleRE (Yu et al. 2022), and ConvE (Dettmers et al. 2018). Note that ConvE—in addition to entity and relation embeddings—also learns the parameters of a convolutional network, used in the scoring function (details in Table S2, available as supplementary data at Bioinformatics online). Whilst being all well-established among practitioners (Hu et al. 2020), these models differ in complexity and in the relational patterns they can capture (Table S2, available as supplementary data at Bioinformatics online). In particular, TransE and DistMult model the interaction between head entity, relation type, and tail entity in quite simple terms. Nevertheless, despite missing certain topological patterns, they remain strong baselines. On the other hand, RotatE, TripleRE, and ConvE are more recent approaches able to capture all four investigated topological patterns. Notice however that the theoretical capability of a scoring function to model a particular edge topological pattern is not per se a guarantee of stronger predictive performance on such edges (Jin et al. 2023). Our experiments are designed to quantify this impact.

The training scheme and hyperparameter optimization details are presented in Section 2, available as supplementary data at Bioinformatics online. The results reported in the following sections refer to tail predictions generated on the held out test split, by scoring each $[eqn]$ query against all entities in the KG and computing the rank of the ground truth tail t, after masking out scores of other $[eqn]$ triples contained in the graph.

5 Results

5.1 Topological properties of biomedical KGs

We first investigate how prevalent the different topological patterns are in the range of public biomedical KGs we consider. Analysing this, we find that many-to-many is the predominant edge cardinality in all considered KGs, with only scarce occurrence of one-to-one, one-to-many, and many-to-one triples (Fig. 4). This is in line with the majority of triples having a head out-degree and tail in-degree larger than one (Fig. 5 and Fig. S1, available as supplementary data at Bioinformatics online).

Occurrence of edge cardinalities in the datasets.

Relative frequency of triples when grouped by head out-degree and tail in-degree of the same relation type.

Furthermore, we observe that edge topological patters vary in frequency across the different KGs (Table 1). While composition is the most prevalent pattern in all examined KGs, other edge topological patters occur at a high frequency in only a subset of KGs. Of note is the lack of inverse edges in all the biomedical KGs, apart from OpenBioLink, highlighting that biomedical knowledge is typically encoded in a directed manner in these graphs.

5.2 Effect of topological properties on model accuracy

Overall, we observe a significant variance in mean reciprocal rank (MRR) achieved between the different KGs as well as between KGE models (Fig. 6; see also Fig. S2, available as supplementary data at Bioinformatics online, for Hits@1 and Hits@10, where we observe similar variability and consistent trends in the results). In an attempt to understand the root cause of this variance, we focus our analysis on the differences in edge cardinalities and topological patterns. When only considering the average occurrence of these topological properties per dataset, the data supports no conclusion about the effect on model accuracy of either edge cardinalities (Fig. S4a, available as supplementary data at Bioinformatics online) or edge topological patterns (Fig. S4b, available as supplementary data at Bioinformatics online). Previous work (e.g. Teneva and Hruschka 2023) went one step further, classifying relation types based on the predominant edge topological pattern and cardinality and evaluating the model accuracy per relation type. However, this also does not result in a clear connection between topological properties and KGE model accuracy (Fig. S5, available as supplementary data at Bioinformatics online). We hypothesize that this could be due to the confounding effects of covariates, such as node degrees and different topological patterns, that are too difficult to disentangle, as these properties are often not homogenous within a given relation type. Therefore, unlike prior work, we dissect the link between topological properties and model accuracy at the level of individual triples to allow for a finer-grained analysis with an improved statistical power and a reduced impact of covariates.

Mean reciprocal rank on the test split achieved by different KGE models, for the six datasets.

When comparing the edge cardinality of individual triples to the MRR achieved by the models on these triples, the observed effect strongly differs between the investigated KGs (Fig. 7, and similarly for Hits@10 in Fig. S3, available as supplementary data at Bioinformatics online). This result suggests that the exact entity degrees, together with other topological properties, might be better suited to explain the model accuracy than a binary one/many cardinality classification.

Effect of edge cardinality on MRR.

Indeed, for a given triple, we observe a strong positive correlation between model accuracy and the in-degree of the tail entity as well as a strong negative correlation between model accuracy and the out-degree of the head entity (Fig. S6, available as supplementary data at Bioinformatics online). This is observed even more clearly when considering the degrees of same relation type (Fig. 8 and Figs S7 and S8, available as supplementary data at Bioinformatics online; the same patterns are observed for Hits@10, as exemplified in Fig. S9, available as supplementary data at Bioinformatics online). In fact, a high in-degree of the tail node in a tail prediction task has been linked to a higher score (Bonner et al. 2022), therefore increasing the likelihood of the model predicting it, as confirmed in Fig. S10, available as supplementary data at Bioinformatics online. On the other hand, a high out-degree of same relation type of the head node in a query $[eqn]$ , implies multiple correct tail entities. Due to the incompleteness of the KG, some of these correct triples might not be present in the graph and thus cannot be filtered out during inference, making the task of predicting the specific entity that is expected by the model more challenging.

Effect of the head and tail degrees of same relation type on MRR.

The effect of having compositions on MRR, triples grouped by their head and tail degrees of same relation type.

MRR of specific interaction types.

Consequently, in an attempt to adjust for entity degree, we investigate the effect of edge topological patterns within sets of triples binned by head and tail degrees. Overall, we find that the impact of edge topological patterns on model performance is more relevant if the degrees of head and tail entities are small. When this is the case, compositions are beneficial to model accuracy across all datasets and models (see Fig. 9 and Figs S11 and S12, available as supplementary data at Bioinformatics online, for MRR; the same holds for Hits@10, e.g. Fig. S13, available as supplementary data at Bioinformatics online). Interestingly, this is true even for DistMult that cannot explicitly model compositional patterns.

Comparison of models trained on Hetionet and PharMeBINet when testing MRR on a set of common edges.

When analyzing the effect of the patterns is symmetric, has inference, and has inverse, we need to distinguish whether the counterpart of the tested edge was present in the training split. If so, the prediction typically becomes much easier due to the availability of additional information seen during training (Toutanova and Chen 2015), as shown in Fig. S14, available as supplementary data at Bioinformatics online. An exception to this rule is the prediction of symmetric triples with the scoring function TransE: since TransE by design cannot model symmetric relations (Table S2, available as supplementary data at Bioinformatics online), it performs poorly on symmetric triples independently of the counterpart being present in the training data or not. This factor is likely contributing to the subpar accuracy of TransE on OpenBioLink and PharmKG, which have a large fraction of symmetric triples (Fig. 6 and Table 1). On the other hand, our experiments confirm an above average performance of DistMult (which models all relations as symmetric) on symmetric triples, see Fig. S14, available as supplementary data at Bioinformatics online. In the more challenging case, where the counterpart edge is not present in the training data, symmetry shows a detrimental effect on model accuracy, whereas having inverse or inference has only little impact on accuracy (Fig. S14, available as supplementary data at Bioinformatics online).

Note that all the above remarks apply equally to biomedical KGs and the general-domain KG FB15k-237.

5.3 Predicting specific relation types

To apply the analysis conducted in the previous sections to real-world tasks, we focus on the relations between pairs of entity types that are of particular interest to practitioners (gene–gene, drug–gene, and drug–disease). We investigate how they are represented in different biomedical KGs, and how these differences affect the predictive performance of KGE models. Statistics for the interactions of the considered entity types are given in Fig. S15, available as supplementary data at Bioinformatics online, while model accuracy is plotted in Fig. 10 (MRR) and Fig. S17 (Hits@10), available as supplementary data at Bioinformatics online.

One basic way of characterizing the performance of a KGE model is to consider if it can rank entities of the correct type above those of other types when predicting a specific relation (for instance, gene entities when predicting drug-gene interactions)—a task we refer to as demixing. As shown in Fig. S16, available as supplementary data at Bioinformatics online, the trained models tend to consistently predict entities of the correct type, albeit with different levels of accuracy. Interestingly, DistMult often shows worse demixing capabilities than other scoring functions. As a consequence of the generally strong demixing performance, the size of the potential set of tails for a given interaction needs to be taken into consideration when comparing model accuracy across different datasets and relation types (the smaller the set of candidates, the higher the expected MRR achieved by random ranking).

5.3.1 Gene–gene

As shown in Fig. S15a, available as supplementary data at Bioinformatics online, these interactions are characterized by a large number of potential tails and a large head out-degree of same relation type (with the exception of PharmKG for the latter), making predictions hard. However, a high predictive accuracy is observed on PharmKG and OpenBioLink: this is explained by the fact that, for these datasets only (Fig. S15b, available as supplementary data at Bioinformatics online), most triples are symmetric (and a significant portion also have inverse/inference edges), with the counterpart likely to have been seen during training. This also explains the relative ordering of scoring functions for OpenBioLink and PharmKG, with DistMult being the best and TransE the worst. Interestingly, ConvE also struggles on these highly symmetric datasets.

5.3.2 Drug–gene

In Hetionet, PrimeKG, and OpenBioLink, these interactions are easier to predict for the models than gene–gene interactions (Fig. 10). This is likely due to an overall smaller number of potential tails seen during training and, in the case of PrimeKG, a markedly smaller head out-degree. PharmKG and PharMeBINet do not satisfy either of these properties, and indeed we observe no improvements in MRR/Hits@10. The strong predictive power observed for OpenBioLink can be linked to the presence of inverse edges for the vast majority of these triples.

5.3.3 Drug–disease

Despite a small number of training triples (which is reflected in the sub-optimal demixing profile, Fig. S16c, available as supplementary data at Bioinformatics online), drug–disease interactions are easily predicted in Hetionet as only 91 disease entities appear as tails of such triples. Remarkably good predictive performance across all scoring functions is observed for PrimeKG, where the number of candidate tails is larger but the out-degree of head entities remains contained. When both these parameters increase (as in PharMeBINet and PharmKG) performance visibly degrades, despite larger in-degree of tails.

5.4 Case study: effect of additional training data

Even when focussing on specific interactions that are represented in multiple KGs, a direct comparison of KGE predictive power across different datasets is limited by having to compare different test sets. In the case of Hetionet and PharMeBINet, however, the latter is constructed by augmenting the former with data coming from other biological databases (Königs et al. 2022). One can therefore often connect individual triples in Hetionet to their exact match in the larger PharMeBINet, which opens up the possibility of studying at a more fundamental level the impact of additional training data on the predictive performance of KGE models. This is important to practitioners as they construct the training KG for a specific task, which is usually done by sub-sampling relevant triples from larger (often proprietary) databases (Chandak et al. 2023).

We consider eight relation types where we can find a significant number of common triples between the two datasets (statistics are given in Table S4, available as supplementary data at Bioinformatics online). We extract 10% of the shared triples of the considered relation type as test set and use all other edges in each graph for training. This requires retraining each KGE model separately for each investigated relation type. Due to the higher computational complexity of ConvE and its resemblance of the other investigated scoring functions throughout all results, we restrict this investigation to the four shallow embedding models. In the case of Hetionet, in addition to the embedding size maximizing memory utilization, we also train models with the same embedding size chosen for PharMeBINet, which is generally strictly smaller (Hetionet_max and Hetionet_same in Fig. 11; hyperparameters in Table S3, available as supplementary data at Bioinformatics online). To further ensure a fair comparison, at test time we restrict predictions to a custom set of candidate tails that is the same for the two datasets, namely the set of entities appearing as tails in shared triples of the given relation type.

We only report on MRR, as (similarly to previous sections) Hits@10 shows analogous patterns. As shown in Fig. 11 and Fig. S18, available as supplementary data at Bioinformatics online, across all tested relation types there is no indication that the KGE models are able to benefit from the additional data seen when training on the larger PharMeBINet graph. On the contrary, models trained on Hetionet consistently perform better, even when using the same embedding size. Interestingly, the gap in MRR varies strongly across relation types and also across scoring functions, with DistMult showing generally little difference when comparing models trained with the same embedding size, while distance-based scoring functions experience worse degradation. From Table S4, available as supplementary data at Bioinformatics online, we notice that the relation types where the MRR gap is more significant tend to have a larger overall head out-degree (and more relation types coming out from head nodes) for triples in PharMeBINet, compared to Hetionet. We hypothesize that the fact that these nodes are used as head entities for more relation types and triples, many of which are likely not relevant for the specific task at hand, negatively impacts the quality of embeddings. On the other hand, Compound-downregulates-Gene and Compound-causes-Side Effect exemplify relation types where PharMeBINet contains far more triples in addition to the ones in Hetionet. Even in this case, where we could expect the additional training data in the larger dataset to be strictly relevant to the prediction task, all scoring functions show markedly degraded performance when trained on PharMeBINet.

We note that the intention of these results is not to state that one dataset is superior to the other, but rather to highlight any observed differences in how models are able to use training data. However, these results do suggest that, in scenarios where the memory budget is fixed, training on smaller, tailored graphs and increasing the embedding size could be more beneficial than expanding the size of the KG, as the additional data can be a source of confusion for shallow KGE models.

6 Conclusions

This article analyses the topological properties of widely used biomedical KGs and compares the corresponding predictive performance of different KGE models, focussing in particular on link-prediction tasks relevant to practitioners. Deviating from previous studies, we justify the need to look at properties of individual edges when trying to explain results, as pooling at the relation level introduces too much noise to detect any patterns. By going beyond the coarse-level one/many binary classification typically used for edge cardinality, we find that considering the actual degrees of head and tail nodes gives a stronger predictor explaining model accuracy. Interestingly, similar interpretations apply to both biomedical KGs and the general-domain KG. Edge topological patterns also impact predictive accuracy, especially when entity degrees are small. For such patterns, we observe an improved accuracy when the counterpart edge (e.g. the reverse edge for symmetric triples) has been seen during training. This in turn raises the problem of inconsistent model validation schemes and topological property definitions when comparing results of previous studies on similar datasets. We address this problem by releasing all predictions from our experiments, together with a new toolbox for KG topological analysis. Finally, by performing a case study comparing predictions on common sets of edges shared by different KGs, we show that training on larger graphs, encoding more biomedical data, can unintuitively harm predictive performance. This evidence should encourage a wider discussion on the guiding principles to adopt when constructing the training KG for biomedical tasks—a crucial problem for real-world practitioners, but scarcely investigated in the literature.

Author contributions

Alberto Cattaneo (Conceptualization, Formal analysis, Investigation, Software, Writing—original draft), Stephen Bonner (Conceptualization, Formal analysis, Software, Writing—original draft), Thomas Martynec (Conceptualization, Formal analysis, Software, Writing—original draft), Edward Morrissey (Writing—review & editing, Supervision, Resources), Carlo Luschi (Writing—original draft, Supervision, Resources), Ian P Barrett (Writing—original draft, Supervision, Resources), Daniel Justus (Conceptualization, Formal analysis, Investigation, Software, Writing—original draft)

Supplementary Material

btaf547_Supplementary_Data