Compilation Quotient (CQ): A Metric for the Compilation Hardness of Programming Languages

TL;DR

The paper introduces the compilation quotient (CQ), a novel metric to quantify the compilation difficulty of programming languages by measuring success rates of programs sampled from grammars, revealing significant variation across languages.

Contribution

It proposes the CQ metric, based on large-scale program sampling, to objectively compare compilation hardness among programming languages.

Findings

C has a CQ of 48.11, indicating moderate compilation success.

C++ and Java have very low CQ values, near 0, showing high compilation difficulty.

Rust's CQ is nearly 0, highlighting its challenging compilation process.

Abstract

Today's programmers can choose from an exceptional range of programming languages, each with its own traits, purpose, and complexity. A key aspect of a language's complexity is how hard it is to compile programs in the language. While most programmers have an intuition about compilation hardness for different programming languages, no metric exists to quantify it. We introduce the compilation quotient (CQ), a metric to quantify the compilation hardness of compiled programming languages. The key idea is to measure the compilation success rates of programs sampled from context-free grammars. To this end, we fairly sample over 12 million programs in total. CQ ranges between 0 and 100, where 0 indicates that no programs compile, and 100 means that all programs compile. Our findings on 12 popular compiled programming languages show high variation in CQ. C has a CQ of 48.11, C++ has 0.60, Java has 0.27 and Haskell has 0.13. Strikingly, Rust's CQ is nearly 0, and for C, even a large fraction of very sizable programs compile. We believe CQ can help understand the differences of compiled programming languages better and help language designers.