Error Minimization in Predicting Accurate Adsorption Energies Using Machine Learning

TL;DR

This paper presents a machine learning approach to accurately predict adsorption energies on transition metal surfaces using minimal atomic features, achieving low error rates and developing scaling laws for better estimations.

Contribution

The study introduces a data-centric machine learning method that predicts adsorption energies with high accuracy and establishes scaling laws between different computational approaches.

Findings

Predictions within 0.4 eV RMSE of quantum estimates.

Reduced RMSE to 0.11 eV using precomputed energies.

Developed scaling laws linking many-body and DFT calculations.

Abstract

Finding the "ideal" catalyst is a matter of great interest in the communities of chemists and material scientists, partly because of its wide spectrum of industrial applications. Information regarding a physical parameter termed "adsorption energy", which dictates the degrees of adhesion of an adsorbate on a substrate is a primary requirement in selecting the catalyst for catalytic reactions. Both experiments and \textit{in-silico} modelling are extensively being used in estimating the adsorption energies, both of which are \textit{Edisonian} approach and demands plenty of resources and are time consuming. In this report, by employing a data centric approach almost instantly we predict the adsorption energies of atomic and molecular gases on the surfaces of many transition metals (TMs). With less than 10 sets of simple atomic features, our predictions of the adsorption energies are within a root-mean-squared-error (RMSE) of less than 0.4 eV with the quantum many-body perturbation theory estimates, a computationally expensive with good experimental agreement. Further, we minimized the RMSE up to 0.11 eV by using the precomputed adsorption energies obtained with conventional exchange and correlation (XC) functional as one component of the feature vector. Based on our results, we developed a set of scaling laws between the adsorption energies computed with many-body perturbation theory and conventional DFT XC-functionals.

Tables

Table 1. Table 1: Weight coefficient obtained with ordinary least square (OLS) regression for atomic adsorbate.

	RPA	PBE	RPBE	LDA	BEEF-vdW	vdW-DF2	m-BEEF	m-BEEF-vdW
Eads.	1	1.635	1.672	1.540	1.710	1.826	1.598	1.582
R(ad)	0	-0.113	-0.118	-0.08	-0.070	0.063	-0.151	-0.150
IE(ad)	0	0.180	0.133	0.259	0.102	-0.060	0.221	0.232
${\Delta }_{fus}H$(ad)	0	0.019	-0.008	0.076	0.007	0.101	0.106	0.022
G (surf.)	0	-0.117	-0.118	-0.053	-0.175	-0.246	-0.072	-0.051
${\Delta }_{fus}H$(surf.)	0	-0.087	-0.081	-0.137	-0.097	-0.133	-0.074	-0.087
$r_s$ (surf.)	0	-0.104	-0.071	-0.129	-0.101	-0.087	-0.024	-0.053
$\sigma$ (surf.)	0	0.028	0.023	0.091	0.044	0.016	0.063	0.089

Table 2. Table 2: Weight coefficient obtained with ordinary least square (OLS) regression for O-X adsorbate.

	RPA	PBE	RPBE	LDA	BEEF-vdW	vdW-DF2	m-BEEF	m-BEEF-vdW
Eads.	1	1.369	1.395	1.297	1.377	1.336	1.243	1.251
G(X)	0	0.102	0.105	0.118	0.086	0.097	0.019	0.012
AM(X)	0	0.077	0.070	0.930	0.055	0.039	0.020	0.027
${\Delta }_{fus}H$(X)	0	-0.047	-0.073	-0.044	-0.065	0.136	0.005	0.041
P(X)	0	0.064	0.054	0.079	0.041	0.015	0.019	0.032
G (surf.)	0	-0.145	-0.155	-0.056	-0.209	-0.375	-0.046	-0.059
$r_s$ (surf.)	0	-0.076	0.027	-0.264	-0.013	-0.008	-0.023	-0.108
$\sigma$ (surf.)	0	-0.030	-0.028	-0.039	-0.043	-0.184	0.024	0.031
$E_{LUMO}$ (O-X)	0	-0.015	0.004	-0.027	-0.007	0.062	-0.015	-0.041

Equations

$$E_{ads}^{RPA}=\sum _i{\alpha }_i{D}_i$$

$$E_{ads}^{RPA}(atm.sp.)=1.635\times E_{ads}^{PBE}-0.113\times R(ad)$$

$$+0.180\times IE(ad)+0.019\times {\Delta }_{fus.}H(ad)$$

$$-0.117\times G(surf.)-0.087\times {\Delta }_{fus.}H(surf.)$$

$$-0.104\times r_s(surf.)+0.028\times {\sigma }^{PBE}$$

$$E_{ads}^{RPA}=\sum _i{\beta }_i{D}_i$$

$$E_{ads}^{RPA}(O-X)=1.369\times E_{ads}^{PBE}+0.102\times G(X)$$

$$+0.077\times AM(X)-0.047\times {\Delta }_{fus.}H(ad)$$

$$+0.064\times P(X)-0.145\times G-0.076\times r_s(surf.)$$

$$-0.030\times {\sigma }^{PBE}(surf.)-0.015\times E_{LUMO}(O-X)$$

Full text

Machine Learning, Adsorption Energy, SciKit Learn, Materials

By reason of the “Foundation pillar of green chemistry”, research in the field of catalytic and biocatalytic reactions has been going on from the last two centuryAnastas et al. (2001). The ultimate goal of these research is to find an “ideal catalyst” which is thermally, mechanically and chemically stable, environmental friendly, posses good selectivity, and efficient in reducing the reaction barrier between reactants and productAnanikov and Beletskaya (2012). The mechanism of catalytic reaction is a multi-step process. Especially, in the heterogeneous catalytic process, where catalysts are often surfaces of solids, majorly governed by three steps (i) adsorption of the reactants (i.e. molecules), (ii) hold the reactant in close proximity for the chemical reaction to take place and (iii) lets the product desorb back to the sorrounding. Being the first-stage of the reaction process, adsorption of molecules on catalyst surface have a large influences on the surface reactionNørskov et al. (2008); Hammer and Norskov (1995) which is visible in famous Br$\o$nsted-Evans-Polanyi (BEP) relationshipsBligaard et al. (2004). Classic works of Christensen et alNørskov et al. (2008) and Sabatier et alSabatier (1911) further shows that optimal adsorption energies of reactants will maximize the catalytic activity. Thus, accurate determination of the adsorption energy is important and necessary in selecting suitable catalyst for the reactions process.

Both experiments and theoretical calculations were extensively used in past to determine the adsorption energies and reaction barriers of various adsorbate on many solid sufacesJanssens et al. (2007); Katsanos et al. (1999); Rudzinski and Everett (2012); Liu et al. (2015); Kibsgaard et al. (2015). Ab-initio quantum mechanical simulation has emerged as a powerful tools not only in determing the adsorption energy but also in understanding the microscopic mechanism of catalytic reactions. However, estimates from these quantum mechanical simulations are functional dependent and the obtained results are varies widely with differnt functional. There are numbers of various methods are being developed and are discussed in the literature. Many of these methods are very selective in appropriate estimation to some specific properties only Schmidt and Thygesen (2018) out of which Random Phase Approximation (RPA)Furche (2001); Fuchs and Gonze (2002); Ren et al. (2012); Marini et al. (2006), a method beyond standard DFT is now considered to be a gold standard for solid state systems. Recently, Schmidt et al. Schmidt and Thygesen (2018) benchmarked the RPA estimated adsorption energies to experimentally obtainbed values with an mean signed error (MSE) of $\approx$ -0.10 eV. Authors further shows that using standard DFT functionals the mean absolute error (MAE) in estimation of the adsorption energies are in the ranges of $\approx$ 0.20-0.44 eV Schmidt and Thygesen (2018). However, such remarkable reduction in the error using RPA comes with an overload of computational cost. It is noteworthy to mention here that, calculations using standard DFT XC-functionals itself a time consuming process especially for a large number of systems.

The emergence of contemporary Machine Learning (ML) technique which is a data centric approach to solve problems has shown a great promise in the field of material science too. Recently, many research groups employed this data centric approach to the field of catalysisRas et al. (2013); Takigawa et al. (2016); Chowdhury et al. (2018); Li et al. (2017); Jäger et al. (2018). Rothenberg et al.Ras et al. (2013) shows that with a simple set of feature vector the adsorption energies can be predicted by an RMSE of 0.94-1.16 eV and R2 of 0.95; in the case of gas adsorbate on a range of metal surfaces. The RMSEs in this case are a bit higher. Similarly, Takigawa et al. predicted the energies of d-band centers in metals and bimetals, which is a crucial factor in determination of adsoption energy of gases on metal surfaces, with an RMSE of less than 0.5 eV. Recent work of Chowdhury et al. shows that predicted adsorption energy using ML can be as close to the quantum mechanically calculated values with an MAE of 0.12 eVChowdhury et al. (2018), however the descriptors in this work are not straight forward.

To address the above mentioned issues collectively in this letter, we show that with a suitable model and tuned hyperparameters the adsorption energies can be predictable quite accurately and instantly. More specifically, we predict the accurate adsorption energies of atomic (H, N, and O) and O-X (X=H,C, and N) molecular species on fcc (111) surfaces of 25 different TMs (Sc, Ti, V, Cr, Mn, Fe, Co, Ni, Cu, Y, Zr, Nb, Mo, Ru, Rh, Pd, Ag, Hf, Ta, W, Re, Os, Ir, Pt and Au) within the RMSE of ..computational error by employing an advanced supervised ML technique. We also provides a set of scaling laws to deduce accurate adsorption energies from estimates with relatively low accuracy.

In this work, we used the data, from an openly accesible database Computational Materials Repository (CMR), generated by Schmidt et al. Schmidt and Thygesen (2018). Here, author used first-principles DFT based calculations for the estimation of the adsorption energies and surface free energies with various XC-functional (e.g. PBE, RPBE, BEEF-vdW, etc) along with RPA computed ones. For the estimation of adsorption energies, the adsorbate site was chosen to be at top site of fcc (111) surfaces. To reduce the computational cost, authors calculated the adsorption energies with relatively high surface coverages.

We divided the dataset into two part; (i) 75% for training and (ii) 25% for testing cases. As feature vector we used majorly elemental properties of individual elements which constitute both surfaces and adsorbates. These elemental properties are groups (G), Covalent Radii (R), atomic number (AN), atomic mass (AM), period (P), Electronegativity (EN), first ionization energy (IE), and Enthalpy of fusion ($\Delta_{fus}H$) (see Tab S-1 of suppplementary information (SI) for details). Structural properties such as density ($\rho$) of each species is also used. The molecules are further characterized by their HOMO and LUMO levels which are calculated using an STO-3G basis set and are obtained from NIST database and Ref.Zhang and Musgrave, 2007. The surfaces are further characterized by by their surface free energies ($\sigma$), Work function ($\Phi$)Skriver and Rosengaard (1992) and Weigner seitz radius ($r_{s}$) of the respective elements Deb et al. (1992); Takigawa et al. (2018). Pearson’s correlation between properties of elements are presented in the form of heatmap in Fig.S1 of SI.

To identify the best model for the prediction of adsorption energy of above mentioned gases on TMs surfaces, we employed various linear (e.g. OLS, PLS, Ridge, Lasso and Kernel Ridge) and non-linear regression (e.g. Gaussian Process (GPR), Gradient Boosting (GBR) and Random Forest Regression (RFR)) methods as implemented in Scikit-Learn packagePedregosa et al. (2011). The predictibility of the models are assesed by Monte Carlo cross validation method. For comparative analysis between the RMSE in predictions by different regression model and brief discussions on methods of KRR, GBR and RFR see section II of SI. It is to mention that, prior to all ML regression we standardize the data by transforming them to center it by removing the mean value of each feature, then scale it by dividing non-constant features by their standard deviation, for which we used the preprocessing tools of Scikit-Learn package.

To this end, we arrange rest of the texts in following order;

First we discuss the ML prediction of adsorption energies using a simple set of feature vector 2. 2.

Next we emphasizes on the substantial reduction in errors in prediction of accurate adsorption energies by using less accurately (with low computational cost) precomputed adsorption energies as an addition to the set of previous feature vector

In this work, we treated both atomic and molecular adsorbate separately. We started our analysis in prediction of adsorption energies of atomic gases (H,C and N), on the surfaces of 25 TMs with a total 21-dimensional feature vector. We employed various regression methods, and compare their RMSEs of predictions (see Fig. S2 of SI). It is to note that the mentioned (here and afterwards) RMSEs and SDs are obtained with Monte-carlo cross-validation method. While most linear regressions method can predict the adsorption energies within an RMSE of 0.86 eV and a standard deviations (SD) of $\approx$ 0.11 eV, KRR predicts an RMSE of 0.35 eV with SD of 0.06 eV. Surprisingly linear regression model like KRR predicts with better accuaracies than non-linear regression techniques GPR, GBR and RFR where RMSE (SD) of 0.50(0.27), 0.43(0.13) and 0.50(0.13) eV are noted respectively. The R2 score of the training set in KRR, GBR and RFR is 0.990, 0.999, and 0.987 while for testing set it is 0.95, 0.93 and 0.91, respectively. Such good fitting of data in both training and testing cases indicate that these models neither overfitting or underfitting (see Fig. S2 and Fig. S3(a-c)).

To minimize the number of features, we obtained their importance in prediction of adsorption energy from GBR method (see Fig. 1). Features like covalent radius (R), first ionization energy (IE) of adsorbate and enthalpy of fusion ($\Delta_{fus}H$) at 300 K of adsorbates are important for adsorbate, while for metal surfaces features like Group (G), surface free energies ($\sigma$), Wigner-Seitz radius ($r_{s}$) and enthalpy of fusion ($\Delta_{fus}H$) at 300 K are important. Thus we construct a reduced 7-dimensional feature vector with the above mentioned ones. We find that statistically obtained results with these reduced features are as good as obtained with the earlier 21-dimensional features. The training RMSE (SD) obtained with KRR, GBR and RFR are 0.33 (0.01), 0.01 (0.00), 0.18 (0.12) eV while for testing these are 0.41(0.06),0.43(0.15) and 0.46(0.11) eV respectively (see Fig.2 (a) and Fig. S4 (a-c) for fitting).

Similar to atomic species, we started our prediction of adsorption energies of O-X molecules with a total of 23-dimensional feature vector. Again in this case KRR out-perform other methods considered here (see Fig.S2 of SI). The predicted RMSE (SD) of test set with KRR, GBR and RFR are 0.27(0.05), 0.32(0.09), 0.38(0.09) eV respectively (see Fig.S2 and S3(d-f) of SI). We note that besides these three methods, GPR works relatively well in molecular systems with a prediction RMSE and SD of 0.28 and 0.13 eV respectively. For O-X molecules obtained feature importance from GBR revealed that LUMO levels of O-X molecules, Period (P), Group (G), atomic mass (AM) and enthalpy of fusion ($\Delta_{fus}H$) of X (=H,C, and N) while Group (G), Wigner Seitz radius ($r_{s}$) and surface free energies ($\sigma$) of metal surface is primarily determining the adsorption energies. Based on these observations, we constructed a 8-dimensional reduced descriptors and predict the adsorption energy again using ML. The RMSEs (SD) in predicting the adsorption energies with reduced set of descriptors for training data set with KRR, GBR, and RFR are 0.21(0.01), 0.05(0.00) and 0.12(0.01) eV respectively while for testing cases the obtained RMSEs (SD) are 0.26(0.04), 0.29 (0.08), 0.33 (0.08) eV respectively (see Fig.2(b)).

Thus, our work suggest that using ML technique with a simple set of feature vector, adsorption energies of atomic (O, H, and N) and molecular species (O-X) on TM surfaces can be predictable upto an RMSE of 0.41 eV and 0.26 eV respectively. To further minimize this error, we used a set of precomputed DFT adsorption energies as a component of feature vector. Next part of this paper is dedicated to the same.

We obtained Pearson’s correlation between adsorption energies estimated with various XC-functional which shows that they are highly correlated ($p>0.96$) to each other as well as to RPA estimated ones (see Fig.3). This observation further motivated us to find a set of scaling laws between RPA estimated adsorption energies and less accurately computed with standard DFT XC-functionals, as RPA method required a huge computational resources. Now alongwith the precomputed adsorption energies using standard DFT XC-functionals we also used the important features that we obtained from previous observations of this work.

For the prediction of accurate adsorption energies of atomic adsorbate on various TMs surface an 8-dimensional feature vector is used. Surprisingly, a simple ordinary least squares (OLS) regression method predicts the adsorption energies within an RMSE $<$ 0.15 eV which is more than 130% reduction in the error compared to the previous case where conventional DFT XC-functional’s estimates is not used as a feature. For details of the error in estimating accurate adsorption energies with various DFT XC-functional estimated adsorption energies see Fig.S5 of SI. In Fig.4(a) we presented adsorption energies with RPA and our ML predicted where PBE estimated adsorption energies are used as one of the feature vector. An RMSE in the prediction is noted to 0.11 eV with a SD of 0.017 eV. It is important to note here that on average the adsorption energies computed with RPA deviates by $\approx$ 0.2 eV from experimentally determined values. Thus, our ML predicted adsorption energies are within the small differences of RPA computed and experimentally obtained values. Since an OLS regression predicts the adsorption energies quite accurately, we deduce a set of scaling laws based on the obtained weight coefficient of features which takes a general form of:

[TABLE]

where $\alpha_{i}$s are the weight co-efficients of the individual features $D_{i}$ for a particular XC-functional and are tabulated in Table 1. For example, RPA estimated adsorption energy can be obtained from PBE estimated by using the following relations;

[TABLE]

Similar to PBE, scaling between RPA and other XC-functionals can be written using information given in Table 1.

Further, we scaled down the adsorption energies obtained with different XC-functional to RPA estimates for O-X molecules too. Here we used a 9-dimensional feature vector with which OLS regression predicts adsorption energies within an RMSE of less than 0.11 eV except for LDA and vdW-DF2 functional. Obtained RMSEs of LDA and vdW-DF2 functionals are 0.19 and 0.16 eV respectively. Our prediction of RPA adsorption energies using PBE estimates as a feature has an RMSE of 0.10 eV with a SD of 0.02 eV (see Fig.4(b) and Fig.S5(b) of SI). Here we again find that the reduction in the RMSE value is $\approx$ 130% compare to the previous estimate discussed earlier. Further, relation between RPA and DFT XC-functional based estimate takes a general form of;

[TABLE]

where $\beta_{i}$ has the similar meaning of $\alpha_{i}$ and values are given in Table 2. Scaling relation between RPA and DFT-PBE estimate is given by the following equation;

[TABLE]

Thus, here we show that error (RMSE) in predictions of adsorption energies of atomic and O-X gas adsorbates on many transition metal surfaces can be reduced to the level of the difference between the estimates of computationally powerful many body perturbation theory and the experimentally obtained ones, using a state-of-the-art supervised machine learning tools. We presented a set of scaling laws which can be used to correct the adsorption energies obtained with low accuracy and relatively inexpensive computation (such as DFT-PBE) to the computationally expensive and high accurately computed (RPA) ones. We also suggest here that proposed method to reduce the error in predictions of accurate physical properties need not to be necessarily valid only for adsorption energy but should also be tested for other material properties.

In summary, we predict adsorption energies of H, O, N, O-H, N-O and C-O on fcc (111) surfaces of 25 different elemental transition metals using state-of-the-art supervised machine learning technique. With only atomic informations as features, we predict the accurate adsorption energies within an RMSE of $<$ 0.4 eV. By using a set of pre-computed adsorption energies by conventional exchange and correlation functional, the RMSE in predictions reduced to the level of the error in high level quantum many body perturbation theory estimates as well as experimental methods. In general, we proposed a new method to obtain material properties with high accuracy, obtained with expensive computational cost, from a less accurately precomputed data obtained with relatively inexpensive computational methods.