Machine-learned patterns suggest that diversification drives economic development

TL;DR

This paper introduces PriSDA, a machine-learning method that uncovers key patterns in economic development, revealing that diversification and product sophistication drive growth and convergence among countries.

Contribution

PriSDA is a novel machine-learning approach that identifies important economic quantities and dynamics, providing new insights into the role of diversification in development.

Findings

Diversity in export baskets is the main distinguishing feature of countries.

Countries tend to become richer after increasing diversification.

Middle-income countries with diverse exports are predicted to grow fastest.

Abstract

We develop a machine-learning-based method, Principal Smooth-Dynamics Analysis (PriSDA), to identify patterns in economic development and to automate the development of new theory of economic dynamics. Traditionally, economic growth is modeled with a few aggregate quantities derived from simplified theoretical models. Here, PriSDA identifies important quantities. Applied to 55 years of data on countries' exports, PriSDA finds that what most distinguishes countries' export baskets is their diversity, with extra weight assigned to more sophisticated products. The weights are consistent with previous measures of product complexity in the literature. The second dimension of variation is a proficiency in machinery relative to agriculture. PriSDA then couples these quantities with per-capita income and infers the dynamics of the system over time. According to PriSDA, the pattern of economic development of countries is dominated by a tendency toward increased diversification. Moreover, economies appear to become richer after they diversify (i.e., diversity precedes growth). The model predicts that middle-income countries with diverse export baskets will grow the fastest in the coming decades, and that countries will converge onto intermediate levels of income and specialization. PriSDA is generalizable and may illuminate dynamics of elusive quantities such as diversity and complexity in other natural and social systems.

Tables

Table 1. Table SI-1: 121 removed products (part 1)

0019	Live animals of a kind mainly used for human food, nes
0115	Meat of horses, asses, mules and hinnies, fresh, chilled or frozen
0129	Meat and edible meat offal, nes, in brine, dried, salted or smoked
0451	Rye, unmilled
0452	Oats, unmilled
0742	Mate
1122	Other fermented beverages, nes (cider, perry, mead, etc)
2114	Goat and kid skins, raw, whether or not split
2223	Cotton seeds
2226	Rape and colza seeds
2231	Copra
2232	Palm nuts and kernels
2234	Linseed
2235	Castor oil seeds
2239	Flour or meals of oil seeds or oleaginous fruit, non-defatted
2440	Cork, natural, raw and waste
2512	Mechanical wood pulp
2516	Chemical wood pulp, dissolving grades
2518	Chemical wood pulp, sulphite
2613	Raw silk (not thrown)
2614	Silk worm cocoons and silk waste
2632	Cotton linters
2634	Cotton, carded or combed
2640	Jute, other textile bast fibres, nes, raw, processed but not spun
2652	True hemp, raw or processed but not spun, its tow and waste
2654	Sisal, agave fibres, raw or processed but not spun, and waste
2655	Manila hemp, raw or processed but not spun, its tow and waste
2659	Vegetable textile fibres, nes, and waste
2685	Horsehair and other coarse animal hair, not carded or combed
2686	Waste of sheep’s or lambs’ wool, or of other animal hair, nes
2687	Sheep’s or lambs’ wool, or of other animal hair, carded or combed
2711	Animal or vegetable fertilizer, crude
2712	Natural sodium nitrate
2714	Potassium salts, natural, crude
2741	Sulphur (other than sublimed, precipitated or colloidal)
2742	Iron pyrites, unroasted
2784	Asbestos
2814	Roasted iron pyrites
2816	Iron ore agglomerates
2860	Ores and concentrates of uranium and thorium
2872	Nickel ores and concentrates; nickel mattes, etc

Table 2. Table SI-2: 121 removed products (part 2)

2876	Tin ores and concentrates
2923	Vegetable plaiting materials
3221	Anthracite, not agglomerated
3222	Other coal, not agglomerated
3223	Lignite, not agglomerated
3224	Peat, not agglomerated
3231	Briquettes, ovoids, from coal, lignite or peat
3232	Coke and semi-coke of coal, of lignite or peat; retort carbon
3330	Crude petroleum and oils obtained from bituminous materials
3341	Gasoline and other light oils
3342	Kerosene and other medium oils
3343	Gas oils
3344	Fuel oils, nes
3345	Lubricating petroleum oils, and preparations, nes
3351	Petroleum jelly and mineral waxes
3352	Mineral tars and products
3353	Mineral tar pitch, pitch coke
3354	Petroleum bitumen, petroleum coke and bituminous mixtures, nes
3413	Petroleum gases and other gaseous hydrocarbons, nes, liquefied
3414	Petroleum gases, nes, in gaseous state
3415	Coal gas, water gas and similar gases
3510	Electric current
4233	Cotton seed oil
4236	Sunflower seed oil
4241	Linseed oil
4244	Palm kernel oil
4245	Castor oil
4311	Processed animal and vegetable oils
4314	Waxes of animal or vegetable origin
5163	Inorganic esters, their salts and derivatives
5223	Halogen and sulphur compounds of non-metals
5249	Other radio-active and associated materials
5323	Synthetic tanning substances; tanning preparations
5828	Ion exchangers of the condensation, polycondensation etc
6112	Composition leather, in slabs, sheets or rolls
6113	Calf leather
6121	Articles of leather use in machinery or mechanical appliances, etc
6122	Saddlery and harness, of any material, for any kind of animal
6344	Wood-based panels, nes
6349	Wood, simply shaped, nes

Table 3. Table SI-3: 121 removed products (part 3)

6518	Yarn of regenerated fibres, put up for retail sale
6545	Fabrics, woven of jute or other textile bast fibres of heading 2640
6546	Fabrics of glass fibre (including narrow, pile fabrics, lace, etc)
6576	Hat shapes, hat-forms, hat bodies and hoods
6593	Kelem, Schumacks and Karamanie rugs and the like
6642	Optical glass and elements of optical glass (unworked)
6646	Bricks, tiles, etc of pressed or moulded glass, used in building
6674	Synthetic or reconstructed precious or semi-precious stones
6727	Iron or steel coils for re-rolling
6741	Universal plates of iron or steel
6750	Hoop and strip of iron or steel, hot-rolled or cold-rolled
6784	High-pressure hydro-electric conduit of steel
6793	Steel and iron forging and stampings, in the rough state
6831	Nickel and nickel alloys, unwrought
6880	Uranium depleted in U235, thorium, and alloys, nes; waste and scrap
6912	Structures and parts of, of aluminium; plates, rods, and the like
6932	Barbed iron or steel wire: fencing wire
7163	Rotary converters
7187	Nuclear reactors, and parts thereof, nes
7213	Dairy machinery, nes (including milking machines), and parts nes
7433	Free-piston generators for gas turbines and parts thereof, nes
7511	Typewriters; cheque-writing machines
7521	Analogue and hybrid data processing machines
7524	Digital central storage units, separately consigned
7612	Television receivers, monochrome
7631	Gramophones and record players, electric
7911	Rail locomotives, electric
7912	Other rail locomotives; tenders
7913	Mechanically propelled railway, tramway, trolleys, etc
7914	Railway, tramway passenger coaches, etc, not mechanically propelled
7924	Aircraft of an unladen weight exceeding 15000 kg
7931	Warships
8821	Chemical products and flashlight materials for use in photografy
8941	Baby carriages and parts thereof, nes
9110	Postal packages not classified according to kind
9310	Special transactions, commodity not classified according to class
9410	Animals, live, nes, (including zoo animals, pets, insects, etc)
9510	Armoured fighting vehicles, war firearms, ammunition, parts, nes
9610	Coin (other than gold coin), not being legal tender
9710	Gold, non-monetary (excluding gold ores and concentrates)

Table 4. Table SI-4: Train–test splits used in cross validation, and coefficient of determination ( R 2 superscript 𝑅 2 R^{2} ) averaged across the three prediction problems (predict annual changes in ϕ 0 subscript italic-ϕ 0 \phi_{0} , ϕ 1 subscript italic-ϕ 1 \phi_{1} , and GDPpc ) for the GAM ( 2 ) and a baseline model that predicts the average change observed in the training set. The model’s task is to predict year t 𝑡 t using data about year t − 1 𝑡 1 t-1 .

		Average $R^2$ across $3$ targets
		GAM (2)	Baseline model
Split
0	Train $1963\le t\le 1988$	$0.038$	$-0.002$
	Test $1989\le t\le 1995$	$0.044$	$-0.034$
1	Train $1963\le t\le 1995$	$0.045$	$-0.005$
	Test $1996\le t\le 2001$	$0.019$	$-0.021$
2	Train $1963\le t\le 2001$	$0.046$	$-0.002$
	Test $2002\le t\le 2006$	$-0.083$	$-0.102$
3	Train $1963\le t\le 2006$	$0.044$	$-0.002$
	Test $2007\le t\le 2011$	$0.000$	$-0.010$
4	Train $1963\le t\le 2011$	$0.040$	$-0.002$
	Test $2012\le t\le 2016$	$-0.135$	$-0.167$
Average	Train	$0.042$	$-0.002$
	Test	$-0.031$	$-0.067$

Equations

$$\displaystyle\mathcal{R}_{cpt}=\frac{X_{cpt}/\operatorname*{\mathbb{E}}\left[X_{cpt}|P_{ct}\right]}{\sum_{c}X_{cpt}/\operatorname*{\mathbb{E}}\left[\sum_{c}X_{cpt}\big{|}\sum_{c}P_{ct}\right]}.$$

$$g\left(\Delta {\varphi }_0(t)\right)$$

$$g\left(\Delta {\varphi }_1(t)\right)$$

$$g\left(\Delta \text{GDPpc}(t)\right)$$

$$\displaystyle\mathcal{R}_{cpt}:=\frac{X_{cpt}/\operatorname*{\mathbb{E}}\left[X_{cpt}|P_{ct}\right]}{\sum_{c}X_{cpt}/\operatorname*{\mathbb{E}}\left[\sum_{c}X_{cpt}\big{|}\sum_{c}P_{ct}\right]}$$

$$E\left[X_{cpt}|P_{ct}\right]$$

$$\displaystyle\operatorname*{\mathbb{E}}\left[\sum_{c}X_{cpt}\Bigg{|}\sum_{c}P_{ct}\right]$$

$${\mathcal{R}}_{cpt}$$

$$\tilde{\log }(x)\equiv \{\begin{array}{ll}1+s\log (x)& \text{if }x>0\\ 0& \text{if }x=0.\end{array}$$

$$s\equiv \underset{z\to x_m}{\lim }\frac{z-1}{\log (z)}=\{\begin{array}{ll}1& \text{if }{x}_m=1\\ \left(x_m-1\right)/\log x_m& \text{if }{x}_m\ne 1\end{array},$$

$$x_m\equiv \min \left\{x:x\in X,x>0\right\}.$$

$$\tilde{\log }(x_m)$$

$$\tilde{\log }(1)$$

$$\tilde{\log }(x)$$

$$\tilde{\log }(x)>1\text{ if and only if }x>1.$$

$$R_{cpt}:=\frac{{\mathcal{R}}_{cpt}-\mu \left(\{{\mathcal{R}}_{cpt}:1962\le t\le 1988,c\in \mathcal{C}\}\right)}{\sigma \left(\{{\mathcal{R}}_{cpt}:1962\le t\le 1988,c\in \mathcal{C}\}\right)}$$

$${\text{diversity}}_{ct}:=\left\{p:{\text{RCA}}_{cpt}>1\right\}.$$

$${\text{RCA}}_{cpt}\equiv \frac{X_{cpt}/{\sum }_p{X}_{cpt}}{{\sum }_c{X}_{cpt}/{\sum }_{cp}{X}_{cpt}}.$$

$${\Vert R_c\Vert }^2$$

$$=[\mathrm{H}{\mathrm{H}}^T{]}_{cc},$$

$$={\Vert H_c\Vert }^2,$$

$$=\sum _k{\varphi }_k(c{)}^2.$$

$$M_{cpt}=\{\begin{array}{l}1,\text{if Rcpt>0}\\ 0,\text{if Rcpt≤0}\end{array}.$$

$$d_c\approx {\Vert R_c\Vert }^2\approx {\varphi }_0(c{)}^2+{\varphi }_1(c{)}^2.$$

Full text

Computers can learn to predict the future using vast datasets too large for humans to grasp. The logic behind a machine’s prediction, however, is often inscrutable [60], and accuracy, not insight, is often the goal. Scientists, meanwhile, generate new theories using intuition or derive them incrementally from existing models. Recent advances in interpretable machine learning are enabling the generation of theories to be automated: machines have identified mathematical laws in physical and biological systems that took many years for scientists to solve manually [12, 54, 21, 66, 14]. But while pendulums and fluid dynamics may follow elegant governing equations, systems comprising capricious humans may not. Thus, in social sciences it is challenging for machines to teach humans better theories.

We introduce a method for identifying interpretable patterns in high-dimensional times-series called Principal Smooth-Dynamics Analysis (PriSDA), and apply it to a unifying question within the social sciences: why some countries are rich and others are poor. PriSDA ingests time-series data from a system characterized by potentially many dimensions, and it identifies a small number of dynamical equations involving smoothing splines that model how the system changes over time. The resulting model is optimized for accuracy yet is readily interpretable.

Traditionally, one builds an economic theory by manually choosing a dimension-reduced representation of a complex process, such as “utility” or an aggregated value such as gross domestic product. Then one constructs models of how those few quantities change over time, and finally one estimates the derived model with data. These models often neglect important patterns, such as the emergence of the tremendous diversity of goods or why rich countries trade with other rich countries [39]. Agnostic of explicit traditions of economics, what theory of economic growth would an automated economist find?

Here, we analyze machine-reduced dimensions of countries’ export baskets (corrected for population size) using principal components analysis [40], and then we train generalized additive models (GAMs) [61] to predict yearly changes in export baskets and in per-capita income. The results indicate the fundamental importance of an economy’s (complexity-weighted) diversity, both as a summary statistic—diversity captures more variance of export baskets across time than any other direction—and as a predictor of growth. The method, PriSDA, is generally applicable and may illuminate the emergence of diversity and complexity in other biological, physical, and social systems.

1 Methods

The PriSDA method identifies equations that predict changes over time (using nonparametric regression, here GAMs) in a complex system described using dimension reduction and potentially other aggregate quantities. Crucial to its success is providing it data that allows large and small units (here, national economies) to be comparable yet allow for absolute growth.

1.1 Multidimensional characterization of the productive capabilities of economies

There is growing interest in multi-dimensional metrics of economic development and poverty [3, 36]. We track macro-level multidimensional economic development based on annual exports, for which there is high quality data for all countries over the past half century. A country’s exports indicate its international competitive advantages, and unlike domestic production, exports share a common classification system.

Instructions for accessing the data and details on its preprocessing are in Sections SI-2 and SI-3. Because data on exports are noisy, products are aggregated into $59$ categories (Section SI-3A). The value of a country $c$’s exports of a product $p$ in year $t$, denoted $X_{cpt}$, tends to correlate positively with the size of the country’s population, $P_{ct}$. To account for that relationship, $X_{cpt}$ is divided by an expectation according to a null model of a country’s expected value of an export given that country’s population. To remove the effects of global price shocks, this quantity is divided by the total value of the world’s exports of that product, which is also normalized by a null model that predicts global export value using global population. We call the resulting quantity the absolute advantage of a country $c$ in a certain product $p$ in year $t$, denoted $\mathcal{R}_{cpt}$:

[TABLE]

Details are in Sec. SI-3B. We consider countries as length-$59$ vectors of $\mathcal{R}_{cpt}$ across all products; a two-dimensional projection of two trajectories of $\mathcal{R}_{cpt}$ is shown in Fig. 1(A). Unlike relative quantities such as revealed comparative advantage [7], a country can grow its absolute advantage arbitrarily. For example, in 2016 Belgium and the Netherlands were the only countries that “punched above their weight” (i.e., had $\mathcal{R}_{cpt}>1$) for all $59$ products $p$.

To put products on equal footing with each other, we center and scale $\mathcal{R}_{cpt}$ by the mean and standard deviation of $\mathcal{R}_{cpt}$ across all countries and across all years $t\leq{1988}$ (Fig. 1(B)). We call the resulting quantity scaled absolute advantage and denote it by ${R}_{cpt}$. Scaled absolute advantage is the number of standard deviations above the pre-${1988}$ mean of all countries’ absolute advantage in that product. ${R}_{cpt}>0$ means that country $c$ excels at producing and exporting product $p$ in year $t$. Making products comparable—by dividing by the product’s global export market in $\mathcal{R}_{cpt}$ and by centering and scaling each product in ${R}_{cpt}$—enables detecting how expertise in one product enables developing expertise in another, regardless of the sizes of the markets of those products.

1.2 Reducing dimensions of export baskets

We reduce dimensions using principal components analysis (PCA) [40] because the resulting dimensions are interpretable and because summing exports reduces the noise in export data. Other methods are discussed in Sec. SI-1.

Figure 2 shows the loadings (weights) of the first three principal components on the $59$ products. The score of a country’s export basket on the $k$th principal component—denoted $\phi_{k}$—is the dot product [illustrated in Fig. 1(C)] of the country’s export basket, $({R}_{cpt})_{p\in\mathcal{P}}$, with that principal component’s loading vector, drawn as a row of rectangles in Fig. 2(A). We interpret this PCA in Sec. 2.

1.3 Inferring dynamics of export baskets

To understand patterns in economic development, next we examine how two summary measures of an export basket, $\phi_{0}$ and $\phi_{1}$, interact with per-capita income [63] (transformed logarithmically): $\texttt{GDPpc}\equiv\log_{10}(\text{GDP per capita})$. Because excelling at exports reflects the capabilities and know-how within a country [33, 34, 31], by inferring a model of how the triple $(\phi_{0},\phi_{1},\texttt{GDPpc})$ changes over time, we aim to shed light on fundamental economic patterns.

The three variables $(\phi_{0},\phi_{1},\texttt{GDPpc})$ are aggregate descriptions of an economy, so we expect them to change smoothly over time. A natural choice for a smooth model are cubic smoothing splines [61, Chapters 3 and 4]. This method provides us with the following system of dynamical equations:

[TABLE]

​​​​​where $\Delta$ takes the expected difference in time, $\Delta f(t)\equiv\operatorname*{\mathbb{E}}\left[f(t+1)-f(t)\right]$; the $c_{i}$ are intercept terms; the $s_{i,j}$ are cubic smoothing splines with smoothing strength chosen using nested-in-time cross validation (Sec. SI-5A); and the link function $g\left(x\right)\equiv\text{sign}(x)\lvert x\rvert^{1/2}$ is applied to make the residuals’ distributions closer to a normal (Sec. SI-5B). The goodness of fit ($R^{2}\approx 0.04$) and the GAM’s competitiveness with other statistical learning methods are discussed in Sec. SI-5A.2.

The terms of (2) are plotted in Figure 3, where one can compare not only the shapes but also the magnitudes. The GAM (2) can be understood as a dynamical model inferred from the data, with which we can attempt to predict the future; it can also be understood as a histogram smoother that helps us see signal amid the noise.

1.4 Data and code availability

The data, available at Dataverse, were analyzed using open-source software, including NumPy, pandas, SciPy, and pyGAM [55]. Code to reproduce this work is available at GitHub.

2 Results

2.1

How a machine summarizes an economy

The first principal component is the direction in product space along which countries are most spread out in terms of variance [40]. We find this direction to be associated with a measure of diversification, as we show below. This first component explains more than half of the variation in export baskets ($57.2\%$) across countries and years.

Mathematically, a country’s score $\phi_{0}$ on the first principal component is a weighted sum of ${R}_{cpt}$ (scaled absolute advantage) across all products. The weights are fixed once PCA has been fitted, but since export baskets change in time, scores $\phi_{0}$ change from year to year (and from country to country). The weights of this principal component are all positive and are depicted in the top row of Fig. 2(A). The score $\phi_{0}$ is highly correlated with per-capita exports summed across products, $\sum_{p}X_{cpt}/P_{ct}$ (Pearson $\rho=0.82$; Fig. SI-6). This correlation, however, is trivial and unsurprising given that $\phi_{0}$ is a positively-weighted sum across products. However, the finding that the weights are all positive, together with the fact that this principal component explains almost $60\%$ of the variation, is not trivial and is of economic significance. It implies that in the dimension defined by the first principal component, countries are separated by their export diversification. Indeed, values of $\phi_{0}$ are most correlated with existing notions of diversification, once we control for other covariates which include Worldwide Governance Indicators and measures of educational attainment (see Fig. SI-6–SI-10).

To see why we refer to the score $\phi_{0}$ on the first principal component as “complexity-weighted diversification” (as opposed to simply “diversification”), notice in the top row of Fig. 2(A) that the loadings are not uniform: they are about twice as large on the more complex products. These variations in the loadings, in fact, are highly correlated with the Product Complexity Index [34] (Pearson $\rho=0.81$; Fig. SI-5). The score $\phi_{0}$, in other words, captures the diversification [34] of an economy, with an emphasis on more complex goods.

The next direction that most spreads out export baskets across countries and across time—conditional on being orthogonal to the first component—loads highly on machinery and negatively on agricultural products [Fig. 2(A), middle row]. Thus, after knowing a country’s complexity-weighted diversity $\phi_{0}$, the next characteristic that most spreads out countries is how much more they export in machinery relative to agricultural goods. This second principal component explains $5\%$ of the variance, $11$ times less than that explained by $\phi_{0}$.

Garments have long been considered to be the first sector to industrialize in a country, including in England during the industrial revolution and in many East Asian countries since the 1960s [11]. However, these products are not an important direction of variation of export baskets between $1962$–${1988}$ in the first and second principal components. Only in the third principal component are textile products substantially loaded [Fig. 2(A), bottom row]. Because this component only explains $3.3\%$ of the variance of export baskets, hereafter we focus on the first two principal components.

2.2 Complexity-weighted diversity predicts growth

The partial dependence plots in Fig. 3 show how yearly changes in $\phi_{0}$, $\phi_{1}$, and GDPpc are predicted by sums of one-dimensional functions of those same variables. The rows of Fig. 3 depict the three equations (2a)–(2c). Notice that per-capita income is not a strong predictor of changes in export baskets as measured by $\phi_{0}$ and $\phi_{1}$ (see the top and middle plots in the third column of Fig. 3). In contrast, the score $\phi_{0}$ on the first principal component is associated with significant growth in income, even though $\phi_{0}$ and $\phi_{1}$ were defined independently of income. The fact that $\phi_{0}$, the complexity-weighted diversity of an economy, seems to drive income, and not the reverse, is consistent with the hypothesis that income is the outcome: income emerges from the productive capabilities of an economy, captured here by $\phi_{0}$ and $\phi_{1}$.

If the “off-diagonal” terms $s_{01}\left(\phi_{1}\right)+s_{02}\left(\texttt{GDPpc}\right)$ in (2a) were absent, then $\phi_{0}$ would settle onto a value near $-10$. This amount is approximately the value of $\phi_{0}$ of the poorest countries in $2016$, such as Liberia, Angola, and the Democratic Republic of the Congo. However, the large intercept in (2a) (the top-right plot in Fig. 3) suggests a general positive tendency to diversify, regardless of the country’s absolute advantage in machinery relative to agriculture ($\phi_{1}$). Once a country has complexity-weighted diversification $\phi_{0}>0$, it can expect significant growth in income. Interestingly, simply exporting more per capita, regardless of the allocation across products, is not associated with growth: when $\phi_{0}$ is substituted with total per-capita exports, the relationship with income growth flattens (Fig. SI-14). Exporting more kinds of goods matters: Replacing $\phi_{0}$ with another notion of diversity [34] preserves the positive relationship with income growth.

We note, in addition, that the lack of clear association between $s_{21}\left(\phi_{1}\right)$ and growth of GDPpc implies that there are weak returns to specialization. In fact, $\phi_{1}$ tends toward zero, regardless of the other two variables, meaning that countries tend toward a diversified export basket that balances agriculture with machinery. These results suggest that export baskets tend to increasingly resemble one another. Next we examine this convergence in more detail.

2.3 2D projections of the data and of the learned dynamics

In Fig. 4 we compare the model (2) with the empirical data. This figure projects the data onto $(\phi_{0},\phi_{1})$ (top row) and onto $(\phi_{0},\texttt{GDPpc})$ (bottom row). The left-hand column shows empirical data, with some countries’ trajectories highlighted. The right-hand column visualizes the vector field of (2) as a “stream plot”, with the third variable not plotted taken to be the pre-${1988}$ mean. That is, the arrows are the expected movement for countries whose third variable (the one not plotted) equals the pre-$1988$ mean; for other countries, the arrows approximate their expected movement.

The data in Fig. 4(A) show that countries tend to move from left to right (they export more and diversify) and toward the middle of vertical axis (they move toward $\phi_{1}\approx 0$, a balance between agriculture and machinery). The gray streamlines of the inferred model in Fig. 4(B) confirm this pattern, suggesting that countries converge toward the trajectory like that of Thailand’s (purple, labeled THA). In the bottom row, the data in Fig. 4(C) show that development success stories like South Korea (KOR), Thailand, and China (CHN) share a common trajectory of increasing $\phi_{0}$ and income. The inferred model’s streamlines in Fig. 4(D) suggest that poor countries will follow in their footsteps, but also that income in the richest countries may fall. The “J” shape in Figs. 4(C) and 4(D) suggests that growth only takes over after diversification reaches a critical value.

2.4 Stream plots of export baskets at different levels of income

In Fig. 5 we vary GDPpc across three values, the $10$th, $50$th, and $90$th percentiles of per-capita income in year ${1988}$. As a country’s per-capita income rises, the map of how its export basket moves through the space of products (as described by $\phi_{0},\phi_{1}$) morphs from the plot on the left to the plot on the right. The colors denote the model’s predicted change in per-capita income [(2c)]. In the plot on the left, we see that the poorest countries tend toward a fixed point: what little they export ($\phi_{0}\approx-8$) tends toward a balance between agriculture and machinery ($\phi_{1}$ tends to zero). Countries with per-capita income near the median (\2764$per year) tend to grow their complexity-weighted diversification$\phi_{0}$ (notice the trend to the right in the middle plot of Fig. 5), a pattern that continues for the richest countries (right-hand plot of Fig. 5). It appears that one need not be very rich to begin to diversify.

This movement in product space $(\phi_{0},\phi_{1})$ appears to maximize expected short-run increases in income, according to (2) (Fig. SI-18). High-income countries tend to be best at moving toward higher income (except for brief periods), and China has been exceptional at it since $1990$.

2.5 Long-run predictions of per-capita income

Research on economic complexity has focused on growth predictions as validation [34, 20] and, recently, research [58] has benchmarked these predictions against those of the International Monetary Fund. We found that predicting the change of export baskets simultaneously with the change of per-capita income was inherently a hard problem across different statistical learning methods [Sec. SI-5A.2]. Instead, we found low-dimensional models to be better suited for generating interpretable, qualitative insights rather than making competitive predictions. With this caveat in mind, we investigate the model’s long-run predictions by iterating 1-year predictions starting from $2016$ data.

Figure 6 shows the model’s long-run predictions of growth in per-capita income as a function of (A) per-capita income and (B) $\phi_{0}$ in $2016$. The model predicts that the diverse, middle-income countries today will significantly catch up to the richest ones, growing at an annual rate of $2\%$. Meanwhile, it predicts that poor countries (such as Liberia) and middle-income countries with low diversity $\phi_{0}$ (such as Angola) are predicted to grow between [math] and $1\%$ annually. Rich countries like Norway are predicted to barely grow at all. The next economic success stories, according to this model, are those with intermediate income and diversification today. These results are consistent with one of the “New Kaldor facts” [38] that rich countries grow more slowly than middle income countries.

3 Discussion

This investigation sits at the intersection of three recent developments in the quantitative social, natural, and physical sciences: (1) the roles of complexity and diversity as drivers of economic growth [35, 34]; (2) identifying universal, low-dimensional patterns of complex human systems over time [59]; (3) using machine learning to uncover governing laws of biological and physical systems [21, 66, 14].

We accordingly developed a new method, Principal Smooth-Dynamics Analysis (PriSDA), by applying tools from statistical learning—namely, dimension reduction and generalized additive models—to identify stylized patterns in economic development. Our measure of countries’ proficiencies in exporting $59$ product categories allows for small and large countries to be comparable, adjusts for global shocks, and can account for absolute economic growth. Given this data, PriSDA found a complexity-weighted measure of diversity, and it approximately recovered the Product Complexity Index [34].

Our analysis generated two core insights. First, diversity appears to drive per capita income rather than the other way around. Second, countries are not predicted to split into rich and poor clubs, nor into manufacturing hubs and agricultural hubs, but instead to converge on the same increasingly diverse basket of goods (and capabilities). We hope that future research reconciles these patterns of diversification with the specialization predicted by Ricardian theories of comparative advantage [4, 22]. The most dynamic economies of the $21^{\text{st}}$ century are predicted to be middle-income economies that are somewhat diversified across products. The least diversified countries have dismal prospects for economic growth, consistent with previous findings [35].

The importance of this approach rests on its applicability beyond the specific case studied here. In general, systems whose evolution is described by a multiplicity of properties are amenable to analysis such as the one we propose here. For example, it is known that wildfires typically reduce the number of species that inhabit an ecosystem, but then species recolonize over time as diversity rises in a process called ecological succession [51]. The composition of species in a system can also converge due to migration [48]. PriSDA could reveal other patterns, still unknown, in such ecological systems. These commonalities suggest the possibility of general theories of complex systems, uncovered by machines less tied to disciplinary paradigms.

Rapidly advancing ways for machines to learn interpretable models bode well for followup studies. Pairwise interactions could be modeled using GA2M [42], or high-dimensional data could be fitted with GAMs that both smooth the data and select terms, such as GAMSEL [18] or SPLAM [41]. Principal Smooth-Dynamics Analysis (PriSDA) takes a step toward this broader goal of using machines to generate fundamental theories of complex natural and social systems.

Acknowledgments

C.D.B. and M.H.B. acknowledge funding from the James S. McDonnell Foundation for the Postdoctoral Award and the Scholar Award (respectively) in Complex Systems.

Competing Interests

The authors declare that they have no competing financial interests.

Correspondence

Correspondence and requests for materials should be addressed to C.D.B. (email: brummitt@gmail.com).

Supplementary Information

SI-1 Related work

Recent work in economics has embraced the multidimensional nature of an economy by compressing information about the products that the economy exports. The “complexity index” [34, 32, 2] and “fitness” [56, 19, 57, 19, 20] of an economy summarize the sophistication of its capabilities. These measures can be defined in many ways [1]; what they share in common is essentially a sum over products weighted by some notion of difficulty of producing that product. They all tackle an ambitious challenge: to describe an economy’s complexity with just one number.

Other work is investigating how compressible economies and societies are. Machado and Mata [43] reduce the dimensions of four time-series (per-capita income, exports relative to income, school enrollment, and lifetime expectancy) to two dimensions using multidimensional scaling. Hruschka et al. [36] create multidimensional models of wealth by reducing the dimensions of responses to household surveys about ownership of assets such as TVs, land, and electricity. Turchin et al. [59] find that societies across millennia tended to move along a common, low-dimensional trajectory in which the complexity of social organization steadily increased over time.

The existence of common patterns in the trajectories of economies can enable forecasts using simple models. For example, economists have fit Markov chains [49] (and continuous versions of them [50] and variants of Markov chains with constraints from growth theory [5]) to time-series data on per-capita incomes. They used the stationary distribution to predict whether countries will converge to similar incomes, or whether they diverge to different “convergence clubs”. References [49, 50, 5] predict a bimodal distribution of incomes in the future. What is needed are models inferred from high-dimensional data [6, p. 42 in Sec. 4.1]. Stochastic methods have also been used to model changes in global exports of many products [17].

Our approach was inspired by recent advances in statistical machine learning aimed at identifying governing laws of motion in data. One method, called SINDy [14], expands features using a hand-picked library of functions and then selects among them using sparse regression. It has since been extended to partial differential equations [53], differential equations with rational terms [45], information criteria [46], and control problems [15]. SINDy has proved successful in discovering laws of physics and microbiology, where we can expect polynomials and other simple functions. We found SINDy challenging to work well with noisy economic data with significant outliers, and great care must be taken in choosing the library of functions so that iterated predictions of the future do not diverge. Other approaches to system identification have used symbolic regression and genetic algorithms [12, 54]; least angle regression [66]; and nested hierarchies of models of smooth, nonlinear dynamics [21].

Forecasting economic time-series has a long history, with the method of choice often being autoregressive–moving-average (ARMA) models [13]. Like the “diffusion index” (or “factor augmented forecasts”) [37], we use principal components to reduce dimensions.

SI-2 Data on exports

Our data has three main stages, which we will refer to as the raw data, the cleaned and standardized data, and the final, aggregated data.

The raw data is the data one can download freely from the United Nations’ Commodities Trade Statistics website; 2. 2.

the cleaned and standardized data is after the raw data has been expressed using standard classifications across years, and problems of the reliability of the raw records addressed and corrected; 3. 3.

the final aggregated data is after we have removed countries and products, and then aggregated into higher level product codes, all with the goal of having reliable statistics.

Under a “Premium Site License” that Harvard has with the United Nations’ Commodities Trade Statistics (COMTRADE), we provide our clean and standardized data, free to download, through the following link:

http://atlas.cid.harvard.edu/, specifically on this page (https://intl-atlas-downloads.s3.amazonaws.com/index.html) by clicking on S2_final_cpy_all.dta (a 451 MB file).

COMTRADE, the original source of our raw data, is the repository of the official trade transactions between importers and exporters. Products traded are codified in three different commodity classifications, but we express all transactions using the Standard International Trade Classification (SITC) system, Revision 2, because it covers the longest span of time. The cleaned and standardized data that we provide through the link above consists of approximately 8 million rows, each representing what a country exported of a 4-digit coded product in a year. Countries are coded following the International Organization for Standardization (ISO). We have a total of 231 unique country codes, 781 unique product codes, and 53 years (1962–2016).

One of the main issues with the raw data is that different countries use different classifications, and even when an importer and an exporter use the same classification, they may use different revisions.111SITC codes have had four revisions: SITC Rev. 1 in 1961, Rev. 2 in 1975, Rev. 3 in 1988, and Rev. 4 in 2006. Each transaction in the raw data reports the code as was originally submitted by each party. Hence, to analyze the data one has to standardize the records into a single classification. COMTRADE provides concordance tables that can be used to express a product from one classification to another (http://unstats.un.org/unsd/cr/registry/regdnld.asp). But since concordance tables are typically “many-to-many” mappings rather than “one-to-one”, the act of re-expressing data from one classification to another introduces additional noise because one must make some arbitrary decisions for how to split the data.

As a consequence, to get the cleaned and standardized data, the raw data has been transformed through a long process of correction of reported transactions, cleaning of misreported records, and standardization of country and product codes. The process is described in detail in [16]. The general approach to do this is referred to in the literature as “mirroring” [10, 47, 64, 65, 52, 27, 44, 9, 25, 8, 26, 28, 24]. Mirroring consists of reconciling between what exporters and importers report, since each transaction should in principle be reported twice. But the difference between previous efforts for creating a trade dataset for research (e.g., the National Bureau of Economic Research [NBER] dataset, and the Centre d’Études Prospectives et d’Informations Internationales [CEPII] BACI dataset) and that of Bustos and Yildirim [16] is that the latter accounts for transaction costs and restrictions implicit in trade reports, and they develop indices of reliability for importers, exporters, and products, which enable them to correctly impute exports of small and developing countries. Thus, the dataset of Bustos and Yildirim [16] is more complete because it increases the number of countries with available data and additional country-product combinations (see [16]), even at very disaggregated levels of the product classification.

SI-3 Preprocessing the exports data

Preprocessing the exports data occurs in five steps described below:

Remove some products and countries (Sec. SI-3.1) 2. 2.

Normalize by population and by global exports (itself normalized by global population) (Sec. SI-3.2) 3. 3.

Apply a logarithmic transformation that preserves zero values and that preserves the number of values above 1 (Sec. SI-3.3) 4. 4.

Center and scale for each product (Sec. SI-3.4) 5. 5.

Reduce dimensions (Sec. SI-3.5)

SI-3.1 Filtering countries and products

First, we filter the data by removing small countries and products that are not exported widely enough. The filters are similar to those in [2] with some differences. One difference is that we avoid path dependence of the filters: we take the union of the countries and products selected by each filter, and then we remove those countries and products all at once. Another difference is that we chose not to set to zero all export values below a certain small threshold (such as \text{US}\5000)$ so that we do not discard information; we let the models handle noisy, small values rather than choose an arbitrary threshold. The last difference is that we remove products that have first digit in their SITC classification equal to either 3 (Fuels, lubricants & related materials) or 9 (Other), which includes products such as zoo animals, coins, and gold).

The steps below completely specify our filtering of products and countries. Countries are specified by their ISO-3166-1 alpha-3 country codes, while products are specified using the SITC classification, both found in [23].

Initialize $\texttt{CountriesToRemove}=\varnothing$ and $\texttt{ProductsToRemove}=\varnothing$ (the empty set). 2. 2.

Remove countries with a small population: Select the countries with population less than 1.25 million in 2008. This selection results in the following list of $81$ countries:

$\texttt{CountriesToRemove}:=\texttt{CountriesToRemove}\,\cup\,\{$ABW, AIA, AND, ANS, ANT, ASM, ATA, ATF, ATG, BHR, BHS, BLZ, BMU, BRB, BRN, BTN, BVT, CCK, COK, COM, CPV, CXR, CYM, CYP, DJI, DMA, ESH, FJI, FLK, FRO, FSM, GIB, GNQ, GRD, GRL, GUM, GUY, IOT, ISL, KIR, KNA, LCA, LUX, MAC, MDV, MHL, MLT, MNE, MNP, MSR, MUS, MYT, NCL, NFK, NIU, NRU, PCN, PLW, PYF, SGS, SHN, SLB, SMR, SPM, STP, SUR, SWZ, SYC, TCA, TKL, TLS, TON, TUV, TWN, UMI, VAT, VCT, VGB, VUT, WLF, WSM$\}$ 3. 3.

Remove countries with little total export value: Select the countries with total export value smaller than $1$ billion USD in 2008. This selection results in the following $81$ countries:

$\texttt{CountriesToRemove}:=\texttt{CountriesToRemove}\,\cup\,\{$AFG, AIA, AND, ARM, ASM, ATA, ATF, ATG, BDI, BEN, BFA, BLZ, BRB, BTN, BVT, CAF, CCK, COK, COM, CPV, CXR, CYM, DJI, DMA, ERI, ESH, FJI, FLK, FRO, FSM, GIB, GMB, GNB, GRD, GRL, GUM, GUY, HTI, IOT, KIR, KNA, LCA, LSO, MDV, MNE, MNP, MSR, MWI, MYT, NER, NFK, NIU, NPL, NRU, PCN, PLW, PSE, PYF, RWA, SGS, SHN, SLB, SLE, SMR, SOM, SPM, STP, SYC, TCA, TGO, TKL, TLS, TON, TUV, UMI, VAT, VCT, VGB, VUT, WLF, WSM$\}$ 4. 4.

Remove countries that export very few products: Select countries with zero export value for at least $95\%$ of products in some year. This selection results in the following list of $52$ countries:

$\texttt{CountriesToRemove}:=\texttt{CountriesToRemove}\,\cup\,\{$AIA, ATA, ATF, BDI, BTN, BVT, CCK, COK, COM, CPV, CXR, ERI, ESH, FLK, FSM, GNB, GNQ, GUF, HMD, IOT, KIR, LAO, LCA, MDV, MHL, MNG, MNP, MRT, MTQ, NFK, NIU, NPL, NRU, PCI, PCN, PYF, RWA, SGS, SSD, STP, SYC, TCA, TLS, TON, TUV, UMI, VGB, VIR, VUT, WLF, WSM, YEM$\}$ 5. 5.

Remove war-torn countries: Add Afghanistan (AFG), Iraq (IRQ), and Chad (TCD) to the set of countries to remove:

$\texttt{CountriesToRemove}:=\texttt{CountriesToRemove}\,\cup\,\{$AFG, IRQ, TCD$\}$ 6. 6.

Remove all products in the categories of fossil fuels and miscellaneous: Add to the set of products to remove all the products with first digit (in the SITC classification scheme) equal to 3 (fossil fuels) or 9 (miscellaneous products such as art and coins):

$\texttt{ProductsToRemove}:=\texttt{ProductsToRemove}\,\cup\,\{{\texttt{3}^{*},\texttt{9}^{*}}\}$

Here, $\texttt{3}^{*}$ means any product code that begins with 3. 7. 7.

Remove products exported by few countries: Select products not exported by at least $80\%$ of countries in at least one year. This selection results in the following $78$ product codes:

$\texttt{ProductsToRemove}:=\texttt{ProductsToRemove}\,\cup\,\{$ 0019, 0115, 0451, 0452, 0742, 2114, 2223, 2226, 2231, 2232, 2234, 2235, 2512, 2516, 2518, 2613, 2634, 2652, 2654, 2655, 2659, 2685, 2712, 2714, 2741, 2742, 2784, 2814, 2816, 2860, 2872, 2876, 3223, 3224, 3231, 3341, 3342, 3343, 3344, 3415, 3510, 4233, 4236, 4241, 4244, 4245, 5163, 5223, 5249, 5323, 5828, 6112, 6113, 6121, 6344, 6546, 6642, 6674, 6727, 6741, 6750, 6784, 6793, 6831, 6880, 7187, 7433, 7521, 7524, 7911, 7912, 7913, 7914, 7924, 7931, 8821, 8941, 9110 $\}$ 8. 8.

Remove products with little global exports: Select products with global exports $<10$ million in some year. This selection results in the following $37$ products:

$\texttt{ProductsToRemove}:=\texttt{ProductsToRemove}\,\cup\,\{$ 0019, 0742, 1122, 2114, 2232, 2235, 2239, 2634, 2652, 2711, 2714, 3224, 3415, 4311, 5223, 5323, 5828, 6112, 6113, 6121, 6122, 6349, 6546, 6642, 6646, 6674, 6741, 6750, 6880, 6912, 7187, 7213, 7433, 7521, 7524, 8941, 9110 $\}$ 9. 9.

Remove products with little market share: Select products whose market share is below the fifth percentile in year 2008. This selection results in the following $39$ products:

$\texttt{ProductsToRemove}:=\texttt{ProductsToRemove}\,\cup\,\{$ 0129, 0742, 2114, 2231, 2232, 2235, 2440, 2614, 2632, 2640, 2652, 2654, 2655, 2659, 2685, 2686, 2687, 2712, 2714, 2742, 2923, 3231, 3415, 4233, 4314, 6112, 6121, 6518, 6545, 6576, 6593, 6642, 6880, 6932, 7163, 7511, 7521, 7612, 7631 $\}$

In the end, these filters remove 121 products (listed in Tables SI-1, SI-2, and SI-3) and the following 112 countries:

Afghanistan (AFG); American Samoa (ASM); Andorra (AND); Anguilla (AIA); Antarctica (ATA); Antigua and Barbuda (ATG); Armenia (ARM); Aruba (ABW); Bahamas (BHS); Bahrain (BHR); Barbados (BRB); Belize (BLZ); Benin (BEN); Bermuda (BMU); Bhutan (BTN); Bouvet Island (BVT); British Indian Ocean Territory (IOT); British Virgin Islands (VGB); Brunei (BRN); Burkina Faso (BFA); Burundi (BDI); Cape Verde (CPV); Cayman Islands (CYM); Central African Republic (CAF); Chad (TCD); Christmas Island (CXR); Cocos (Keeling) Islands (CCK); Comoros (COM); Cook Islands (COK); Cyprus (CYP); Djibouti (DJI); Dominica (DMA); Equatorial Guinea (GNQ); Eritrea (ERI); Falkland Islands (FLK); Faroe Islands (FRO); Fiji (FJI); French Guiana (GUF); French Polynesia (PYF); French South Antarctic Territory (ATF); Gambia (GMB); Gibraltar (GIB); Greenland (GRL); Grenada (GRD); Guam (GUM); Guinea-Bissau (GNB); Guyana (GUY); Haiti (HTI); Heard Island and McDonald Islands (HMD); Holy See (Vatican City) (VAT); Iceland (ISL); Iraq (IRQ); Kiribati (KIR); Laos (LAO); Lesotho (LSO); Luxembourg (LUX); Macau (MAC); Malawi (MWI); Maldives (MDV); Malta (MLT); Marshall Islands (MHL); Martinique (MTQ); Mauritania (MRT); Mauritius (MUS); Mayotte (MYT); Micronesia (FSM); Mongolia (MNG); Montenegro (MNE); Montserrat (MSR); Nauru (NRU); Nepal (NPL); Netherlands Antilles (ANT); New Caledonia (NCL); Niger (NER); Niue (NIU); Norfolk Island (NFK); Northern Mariana Islands (MNP); Pacific Island (US) (PCI); Palau (PLW); Palestine (PSE); Pitcairn Islands (PCN); Rwanda (RWA); Saint Helena (SHN); Saint Kitts and Nevis (KNA); Saint Lucia (LCA); Saint Pierre and Miquelon (SPM); Saint Vincent and the Grenadines (VCT); Samoa (WSM); San Marino (SMR); Sao Tome and Principe (STP); Seychelles (SYC); Sierra Leone (SLE); Solomon Islands (SLB); Somalia (SOM); South Georgia South Sandwich Islands (SGS); South Sudan (SSD); Suriname (SUR); Swaziland (SWZ); Taiwan (TWN); Timor-Leste (TLS); Togo (TGO); Tokelau (TKL); Tonga (TON); Turks and Caicos Islands (TCA); Tuvalu (TUV); United States Minor Outlying Islands (UMI); Vanuatu (VUT); Virgin Islands (VIR); Wallis and Futuna (WLF); Western Sahara (ESH); Yemen (YEM).

Final dataset

After removing countries and products, we have a dataset of $138$ countries, $665$ products at the 4-digit level, and $6377$ distinct (country, year) pairs. Summing export values at the $2$-digit level results in $59$ products. Merging this exports data with population data from the World Bank [63] and from [23] drops 240 (country, year) samples, resulting in $6137$ distinct (country, year) pairs.

This dataset has on average $92\%$ of the global population (minimum $86\%$, maximum $96\%$) and $77\%$ of global trade (minimum $67\%$, maximum $85\%$). Time-series of those values are plotted in Figure SI-1.

SI-3.2 Normalize export values by population and by global exports

To make small and large countries comparable, we divide the value of a country $c$’s exports of a product $p$ in year $t$, denoted $X_{cpt}$, by a null model of a country’s expected value of its exports of that product given that country’s population, $\operatorname*{\mathbb{E}}\left[X_{cpt}\mid P_{ct}\right]$. To remove the effects of global price shocks, we divide this quantity by the total value of the world’s exports of that product, which we also normalize by a null model that predicts global export value using global population. Formally, for each country $c$ in a set of $123$ countries $\mathcal{C}$ and for each product $p$ in the set of $59$ products $\mathcal{P}$, we define the absolute advantage of country $c$ in product $p$ as

[TABLE]

SI-3.2.1 Null models of export values based on population size

Countries with more people tend to export more, but typically not in proportion to their population size. To allow for product-specific variation in the relationship between exports and population, we assume that the expectations in (SI-3) follow power laws of population size.

A country with population double that of another country typically exports more, but rarely does it export twice as much. For intuition, consider a disk-shaped country with its population distributed evenly across space and with exports occurring at the border in proportion to the size of the perimeter. That country’s exports increase with the square root of the population size. (This example is more extreme than reality: the exponent is $\approx 0.88$ rather than $0.5$.) Motivated by this intuition, we create a null model of exports by assuming that export value of a certain product, either by a certain country or by the whole world, grows with population size raised to some power, and that this exponent varies from one product to another. Specifically, we assume that

[TABLE]

With (SI-4) and (SI-5), our measure of a country $c$’s absolute advantage in producing the product $p$ in year $t$ is

[TABLE]

This quantity $\mathcal{R}_{cpt}$ captures how proficient a country $c$ is in exporting product $p$ in year $t$, relative to an average country of its population size.

The distributions of the exponents $\beta_{p}$ and $\delta_{p}$ are plotted in Figure SI-2. The exponents $\beta_{p}$ have average value of $0.88$; the minimum is $0.49$ for the product Dairy products and birds’ eggs (product code 02), and the maximum is $1.21$ for the product Crude rubber (including synthetic and reclaimed) (product code 23). Thus, the export value of certain product tends to grow sublinearly with the population size, in accordance with the hypothetical disk-shaped country described above. Meanwhile, the exponents $\delta_{p}$ are much larger: the average (across all $59$ products) is $5.37$. The minimum is $2.50$ for the product Textile fibers (not wool tops) and their wastes (not in yarn) (product code 26), and the maximum is $8.18$ for the product Office machines and automatic data processing equipment) (product code 75). Thus, global exports of a product tend to grow superlinearly with global population.

SI-3.3 Logarithmically transforming data with lots of zeros in it

In this paper, we consider yearly export values $X_{cpt}$ of $59$ two-digit products. These export values range from zero to nearly a trillion US dollars per year. China, for example, has recently exported over \300billion in *Electric machinery, apparatus and appliances, nes, and parts, nes* (product code 77) in one year. After normalizing by population and by global exports with ([SI-3](#S3.E3)), the values are still rather heavy-tailed and range from [math] to7.3\times 10^{4}$ US dollars per year; see the left and middle panels of Figure SI-3.

One way to logarithmically transform heavy-tailed data with zeros in it is to add one before applying the natural logarithm, so that zero maps to zero. However, we found that this transformation resulted in data that was approximately exponentially distributed rather than normally distributed, and we found that adding one introduces a scale in the data. To avoid these outcomes, we applied a different logarithmic transformation that is plotted in Fig. SI-4:

[TABLE]

where the scaling factor

[TABLE]

and $x_{m}$ is the smallest positive value of all elements of the matrix $X$:

[TABLE]

The limit in (SI-8) ensures that $s$ exists for all $x_{m}>0$; in particular, $s=1$ when $x_{m}=1$. Note that

[TABLE]

Equations (SI-10) and (SI-11) are direct computations. Equation (SI-12) holds because $\left(z-1\right)/\log(z)$ is positive for $z>0$. A consequence of (SI-11) and of (SI-12) is that

[TABLE]

Statement (SI-13) is an important property for a logarithmic transformation of data like that studied here: because the data is normalized by dividing by the prediction of a null model, being above one (or not) is meaningful, so we wish our logarithmic transformation to preserve which values are above one and which values are below one.

SI-3.4 Centering and scaling

Next we pivot the data so that the rows are observations of a certain country in a certain year, and the columns are the values of $\mathcal{R}_{cpt}$ for each of the $59$ many products $p$. We center and scale the columns using the pre-$1989$ column means and standard deviations:

[TABLE]

where $\mu$ denotes mean and $\sigma$ denotes standard deviation. The column means and standard deviations, like all other preprocessing steps such as dimension reduction described next, are fit to data from year ${1988}$ or earlier. That way, we can split the data into cross validation sets that are nested in time, and all preprocessing is done with the earliest set of data (years ${1962}$ to ${1988}$, inclusive).

SI-3.5 Reduce dimensions

Next we reduce dimensions using principal components analysis (PCA) [40]. Because the data was centered (see Section SI-3.4), PCA is equivalent to doing a truncated singular value decomposition. More insights from PCA applied to this exports data are given next in Sec. SI-4.

SI-4 Further analysis of the principal components

SI-4.1 Correlation between the loading on the first principal component and the Product Complexity Index

Recall from Fig. 2 that the first principal component loads positively on all products. But the loadings are not equal: the first principal component loads more on complex products like power generating machinery (product code 71) that are produced by few countries, compared to simpler products like vegetables and fruit (product code 06) that are produced by many countries. In fact, as shown in Fig. SI-5, these loadings are highly correlated with the Product Complexity Index [34], a notion of complexity (or knowledge intensity) of products based on the complexity (or knowledge intensity) of the countries that produce them. The second principal component is also correlated with the Product Complexity Index, but less so (Pearson correlation $\rho=0.70$ versus $\rho=0.81$).

SI-4.2 Interpreting a country’s score on the first principal component

SI-4.2.1 Pair-wise correlations

Figure SI-6 shows that a country’s score on the first principal component, $\phi_{0}$, is highly correlated with its total export value per capita [Pearson correlation $\rho=0.82$, Fig. SI-6(D)], which is not surprising given that the loadings of products on the first principal component are all positive. However, $\phi_{0}$ is more correlated with the Economic Complexity Index [34] [$\rho=0.82$, Fig. SI-6(B)] than is total export value per capita [$\rho=0.63$, Fig. SI-6(E)]. The score on the first principal component, $\phi_{0}$, is also somewhat correlated with a certain notion of diversification of the export basket [$\rho=0.67$, Fig. SI-6(G)]. These observations (together with more reasons given below in Sec. SI-4.2.3) are why we refer to $\phi_{0}$ as “complexity-weighted diversity”. Here, we consider the notion of diversification of exports used in [34, Equation 3], namely the number of products $p$ such that the revealed comparative advantage $\textrm{RCA}_{cpt}$ exceeds one:

[TABLE]

where

[TABLE]

Figure SI-6(G) indicates that export baskets with the highest score $\phi_{0}$ on the first principal component tend to have $\textrm{RCA}_{cpt}$ larger than one for approximately half of the $59$ $2$-digit products, while the export baskets with the lowest $\phi_{0}$ tend to have $\textrm{RCA}_{cpt}$ larger than one for fewer than $10$ out of the $59$ $2$-digit products. Thus, the direction in the space of products in which export baskets over the past $50$ years are most spread out is, loosely speaking, one that distinguishes undiversified, small export baskets from diversified, large ones.

SI-4.2.2 Intuition behind the correlations

First, we give intuition the correlation between exports per capita and $\phi_{0}$ in Fig. SI-6(D). As shown in Fig. 2 and SI-5, the loadings of the first principal component are positive and range from $0.05$ to $0.15$. This homogeneity of the loadings means that the scores $\phi_{0}$ capture an average scaled absolute advantage. Given the formula of scaled absolute advantage [(SI-3), centered and scaled via (SI-14)], we therefore expect $\phi_{0}$ to be highly correlated with the logarithm of exports per capita [$\rho=0.82$, Fig. SI-6(D)].

For similar reasons, we also expect $\phi_{0}$ to be correlated with the diversification of the export basket [Fig. SI-6(G)]. To see why, let us drop the index $t$ for clarity of exposition. From the definition of principal component analysis as a singular value decomposition, we get that the matrix $\mathrm{R}$ of absolute advantages can be factored as $\mathrm{R}=\mathrm{H}\mathrm{V}^{T}$. Here, $H_{ck}$ is the score in the $k$-th principal component for country $c$ (i.e., the $c$-th element in the vector $\phi_{k}$ in our notation), and $V_{pk}$ is how much product $p$ weights (or loads) on component $k$. The matrix $\mathrm{V}$ is orthogonal, and thus $\mathrm{V}^{T}\mathrm{V}=\mathrm{I}$ is the identity matrix. Given this decomposition, the 2-norm length of the export vector of country $c$ is

[TABLE]

In other words, the norm of the export basket vector of country $c$ is equal to the norm of $c$’s vector in the space of principal components. Now define a country’s diversification in terms of absolute advantage (rather than in terms of RCA as in (SI-15) and [34]) by discretizing the elements of the vector $R_{c}$:

[TABLE]

Having $R_{cpt}>0$ would mean that the country $c$ has an absolute advantage larger than the mean absolute advantage that countries have in that product $p$ in that year $t$. (We could discretize the matrix in other ways, but the result is qualitatively the same.) Then diversity (in terms of absolute advantage) is $d_{c}=\sum_{p}M_{cp}$, but it is also $d_{c}=[\mathrm{M}\mathrm{M}^{T}]_{cc}$. All together, we conclude that when there is a first principal component that explains most of the variation, then

[TABLE]

Thus, we would expect diversity to be correlated with the square of the first principal component score.

SI-4.2.3 Regressions of $\phi_{0}$

To investigate whether the score on the first principal component captures information beyond these three quantities Economic Complexity Index, log-exports, and diversification, we use the following datasets:

Worldwide Governance Indicators (WGI)

from http://info.worldbank.org/governance/wgi/index.aspx#home. According to the source, this dataset comprises “aggregate and individual governance indicators for over 200 countries and territories over the period 1996–2016, for six dimensions of governance: Voice and Accountability, Political Stability and Absence of Violence, Government Effectiveness, Regulatory Quality, Rule of Law, and Control of Corruption.”

Barro-Lee Educational Attainment Data

from http://barrolee.com/data/Lee_Lee_v1.0/LeeLee_v1.dta or http://www.barrolee.com/data/BL_v2.2/BL2013_MF1599_v2.2.csv, which reports “educational attainment data for 146 countries in 5-year intervals from 1950 to 2010”. It also provides information about the distribution of educational attainment of the adult population over age 15 and over age 25 by sex at seven levels of schooling: no formal education, incomplete primary, complete primary, lower secondary, upper secondary, incomplete tertiary, and complete tertiary. Average years of schooling at all levels—primary, secondary, and tertiary—are also measured for each country and for regions in the world.

International Data on Cognitive Skills

from http://hanushek.stanford.edu/sites/default/files/publications/hanushek%2Bwoessmann.cognitive.xls which was studied in [29].

The question is: how much do the quantities and indicators in these datasets explain $\phi_{0}$?

Figures SI-7, SI-8, SI-9, and SI-10 show the results of the standardized coefficients for different univariate and multivariate regressions. In light of the quadratic relationship in (SI-16) between diversity (in terms of absolute advantage, $d_{c}=\sum_{p}M_{cp}$) and the score on the first principal component, we use the square root of diversity as a predictor of the score on the first principal component. While all regressors predict $\phi_{0}$ to some extent when we carry out univariate regressions, when all are put together only exports per capita, diversity, government effectiveness, and rule of law survive. In the multivariate regressions done per year, the coefficients for exports per capita and diversity are consistently significant and positive, and have similar magnitudes. In light of these regressions and of the relationship between the loadings on the first principal component with product complexity (Fig. SI-5), in the main text we refer to the score $\phi_{0}$ on the first principal component as “complexity–weighted diversity”.

SI-4.3 More ways of interpreting the first three principal components

One-digit product codes

To help interpret the principal components, in Fig. SI-11 we group the $59$ products at the $1$-digit level and plot the mean and standard deviation. Recall from Sec. SI-3.1 that we removed product category 3 (i.e., all products with SITC product code that begins with 3) to avoid focusing on endowments of fossil fuels.

Most and least loaded 2-digit products in the second and third principal components

Figure SI-12 shows the top 10 most and least loaded products in the second and third principal components. These “top 10 lists” aid the interpretation of the first three principal components.

SI-4.4 Substituting per-capita exports or diversification for the score $\phi_{0}$ on the first principal component suggests that diversification, not simply a rise in total exports per capita, precedes economic growth

Rich countries are usually big exporters, have diversified economies, and produce complex products. Hence, these quantities correlate positively with each other, which makes it particularly difficult to interpret the meaning of scores resulting from PCA. We find in the main text that high levels in $\phi_{0}$ precede growth in income. But the scores of $\phi_{0}$ are correlated with both exports per-capita and product diversification, so this relationship could mean either that high levels of export per-capita precede growth, or that high levels of diversification precede growth, or both. To better understand what $\phi_{0}$ represents in our analysis and what it reveals about economic development, here we substitute another variable for it in the GAM: either per-capita export value, or another definition of “product diversification”.

SI-4.4.1 Exports per capita are less strongly associated with growth in incomes compared to $\phi_{0}$

Figure SI-13 shows a partial dependence plot (like Fig. 3) from a model fitted to the same dataset except that $\phi_{0}$ is replaced by total export value per capita. Note in particular the bottom-left plot: The 95% confidence interval of the relationship between economic growth and export value per capita contains zero or is slightly below zero, suggesting a weak relationship between rises exports (no matter the product) and economic growth. Contrast this flat relationship with the positive trend in the bottom-left plot of Fig. 3.

SI-4.4.2 Replacing $\phi_{0}$ with another notion of diversification results in qualitatively similar results

Next we tried replacing $\phi_{0}$ by diversification as defined in Eq. [3] in [34]: the number of products with revealed comparative advantage (RCA) larger than one. The resulting GAM is approximately linear and behaves qualitatively similarly to the model in the main text; in fact, it appears to be a linear approximation of that GAM. In light of this resemblance to the model in the main text, it seems reasonable to call $\phi_{0}$ something akin to diversification; here, we call $\phi_{0}$ “complexity-weighted diversification”. The comparison between Figs. 3, SI-13, and SI-14 suggests that exporting a large diversity of complex products precedes economic growth.

SI-5 Details about the generalized additive model: Training and performance

SI-5.1 Cubic smoothing splines using the B-spline basis

Generalized additive models were estimated using the package pyGAM 0.2.17 [55], which uses a B-spline basis, computed using De Boor recursion. The basis functions extrapolate linearly past the end-knots. Details on cubic smoothing splines are in [61, Chapters 3 and 4] and [30, Sec. 5.4].

SI-5.1.1 Nested-in-time cross validation

The generalized additive model (GAM) (2) has two hyperparameters: the smoothing strength $\lambda$ that penalizes wiggliness, and the number of splines. Following the advice of [61], we tried relatively large values for the number of splines (uniformly distributed over $\{15,16,\dots,60\}$) and let the smoothing penalty do the regularization. We sampled $\log_{10}\lambda$ uniformly over $[-3.0,10.0]$.

To choose the best hyperparameters, we split the data into five training sets that are nested in time as follows. The task is to predict the change in the time-series between year $t-1$ and $t$ given the value of the time-series at year $t-1$ (i.e., autoregression with lag $1$). We put the earliest $39\%$ of samples in the first training set, and then we partition the remaining samples into roughly equal-size sets. (Because countries appear and disappear, some care needs to be taken with time-series of different lengths; we use quantiles of the times of all the samples to find where to split the data.) The result is that the first training set is data with $t$ between ${1962}$ and ${1988}$, and the corresponding test set is data with $t$ between $1989$ and $1995$. The train–test splits are shown in Table SI-4.

The model is always tested on data from the future relative to the test set. With this cross validation scheme, the hyperparameters with best performance on the test sets were smoothing strength $\lambda=2748.5$ and $37$ splines. These values were used for each of the three equations in equation (2).

SI-5.1.2 The GAM outperforms a baseline model that predicts the average change in the test set

Table SI-4 and Fig. SI-15 show the performance of the GAM (2) in terms of the coefficient of determination, $R^{2}$. The GAM does better than a simple baseline model that simply predicts the average change observed in the training set. (This baseline model predicted the test set more accurately than a baseline model that predicted the median of the training set.) However, the performance of the GAM has considerable room for improvement: its $R^{2}$, averaged across the three prediction problems (of predicting $\phi_{0}$, $\phi_{1}$, and GDPpc) is $0.042$ on the training sets and $-0.031$ on the test sets. For some reason, the score $\phi_{0}$ is particularly difficult to predict in the most recent test set ($2011$–$2016$, in “Split 4”). Also, per-capita incomes were difficult to predict in the test set of Split 2 ($2001$–$2006$).

We tried several alternative modeling strategies other than GAMs, including neural networks, random forests, and kernel ridge regression. None of these competing methods significantly outperformed GAMs in their accuracy on test sets, and they were less readily interpretable than GAMs, so we chose to focus on GAMs. That more flexible modeling strategies could not significantly outperform GAMs, despite our best efforts at searching over a large set of hyperparameters, indicates just how difficult it is to predict the dynamics of national economies.

SI-5.2 Quantile–quantile plots and raising the response variable sto the ${1/2}$ power

Figure SI-16 shows a quantile-quantile (QQ) plot of the residuals of generalized additive models of the form (2) with the response transformed by $g\left(x\right)\equiv\text{sign}(x)\lvert x\rvert^{1/2}$ (bottom row) or not (top row). The residuals are conditioned on the fitted model coefficients and scale parameter. The closer the QQ-plot is to a straight line, the better the distributional assumptions are satisfied. The QQ-plots were made using the function qq.gam in the package mgcv by Simon Wood [62].

Because the QQ-plots are closer to a straight line when we transform the response with $g\left(x\right)\equiv\text{sign}(x)\lvert x\rvert^{1/2}$ (compare bottom row and top row in Fig. SI-16), we transform the response variables with the invertible function $g\left(x\right)\equiv\text{sign}(x)\lvert x\rvert^{1/2}$. (An example of such a transformation based on the results of qq.gam is given on page 230 in Section 5.2.1 in [61].) When making iterated predictions (as in Figure 6), we invert $g\left(x\right)$ in order to feed the response back into the model as a predictor variable.

SI-5.3 Errors averaged by country

This model tends to be more accurate for more developed countries, as shown in Figure SI-17. Figure SI-17 shows the squared errors averaged by country, for predicting export baskets that have been dimension-reduced with PCA (left column) and for predicting the export baskets themselves (right-column).

The trajectories of poorer countries in Figures 4 and 5 in the main text appear to be laminar. By contrast, Cristelli et al. [20] found that the trajectories of the poorest countries are turbulent when they analyzed yearly changes in “fitness” and per-capita incomes. The dynamics in our model (2) are laminar because a large smoothing strength is chosen in cross validation (Sec. SI-5.1.1). However, the greater predictability of richer countries (Fig. SI-17) is consistent with the finding of Cristelli et al. [20] that richer countries move in a more laminar, predictable path through the space defined by per-capita income and by a summary measure of export baskets.

SI-5.4 Alignment of changes in export baskets with the gradient of per-capita incomes

SI-5.4.1 Countries tend to “hill climb” to higher incomes

Do economies’ export baskets change in ways that lead to rising incomes? To explore that question, we plot in the left column of Figure SI-18 the direction in $(\phi_{0},\phi_{1})$ that would most increase per-capita incomes [i.e., the gradient $\left(s_{20}^{\prime}(\phi_{0}),s_{20}^{\prime}(\phi_{1})\right)$. For comparison, in the right-hand column of Figure SI-18 we plot the typical movement in $(\phi_{0},\phi_{1})$ according to the fitted model (2). In these plots on the right-hand column, at each point in a fine grid of points, we find the GDPpc of the closest sample to that grid point. This procedure results in more wiggles in the streamlines compared to when GDPpc is fixed at a certain value, as in Figure 5 in the main text.

By comparing the left- and right-hand plots in Figure SI-18, we see how well economies tend to “hill climb” toward higher per-capita incomes according to the model (2). Except for two extreme points where few observations are found (very high and very low $\phi_{1}$), countries do tend to move along the gradient of per-capita income.

Figure SI-19 shows the cosine similarity of countries’ movement in $(\phi_{0},\phi_{1})$ and the gradient of how $\Delta\texttt{GDPpc}$ depends on $(\phi_{0},\phi_{1})$. By this measure, countries with higher incomes tend to be better hill climbers (i.e., high cosine similarity), and China has become an unusually good hill climber since the late $1990\text{s}$, while Madagascar has only recently moved slightly aligned with the gradient of per-capita incomes.