Dropping diversity of products of large US firms: Models and measures

TL;DR

This study uses text mining of product descriptions from large US firms from 1997 to 2017 to reveal a consistent decline in product diversity, challenging the assumption of increasing diversity in the economy.

Contribution

It introduces a novel text-based methodology to measure product diversity over time using publicly available data and compares multiple similarity metrics.

Findings

Product diversity of large US firms has steadily declined from 1997 to 2017.

Multiple similarity metrics confirm the downward trend in product diversity.

The results enable testing hypotheses about the causes of declining product diversity.

Abstract

It is widely assumed that in our lifetimes the products available in the global economy have become more diverse. This assumption is difficult to investigate directly, however, because it is difficult to collect the necessary data about every product in an economy each year. We solve this problem by mining publicly available textual descriptions of the products of every large US firms each year from 1997 to 2017. Although many aspects of economic productivity have been steadily rising during this period, our text-based measurements show that the diversity of the products of at least large US firms has steadily declined. This downward trend is visible using a variety of product diversity metrics, including some that depend on a measurement of the similarity of the products of every single pair of firms. The current state of the art in comprehensive and detailed firm-similarity measurements is a Boolean word vector model due to Hoberg and Phillips. We measure diversity using firm-similarities from this Boolean model and two more sophisticated variants, and we consistently observe a significant dropping trend in product diversity. These results make it possible to frame and start to test specific hypotheses for explaining the dropping product diversity trend.