Towards Quantitative Evaluation of Crystal Structure Prediction Performance

TL;DR

This paper introduces a set of quantitative metrics for evaluating crystal structure prediction algorithms, enabling automatic and consistent assessment of predicted structures against ground truths, thus advancing the field's evaluation standards.

Contribution

It proposes new quantitative structure similarity metrics for CSP, facilitating automated evaluation and comparison of different algorithms.

Findings

Metrics enable automatic quality assessment of crystal structures.

The approach reduces manual inspection effort.

Open-source code is provided for community use.

Abstract

Crystal structure prediction (CSP) is now increasingly used in the discovery of novel materials with applications in diverse industries. However, despite decades of developments, the problem is far from being solved. With the progress of deep learning, search algorithms, and surrogate energy models, there is a great opportunity for breakthroughs in this area. However, the evaluation of CSP algorithms primarily relies on manual structural and formation energy comparisons. The lack of a set of well-defined quantitative performance metrics for CSP algorithms make it difficult to evaluate the status of the field and identify the strengths and weaknesses of different CSP algorithms. Here, we analyze the quality evaluation issue in CSP and propose a set of quantitative structure similarity metrics, which when combined can be used to automatically determine the quality of the predicted crystal structures compared to the ground truths. Our CSP performance metrics can be then utilized to evaluate the large set of existing and emerging CSP algorithms, thereby alleviating the burden of manual inspection on a case-by-case basis. The related open-source code can be accessed freely at https://github.com/usccolumbia/CSPBenchMetrics