NLP-enhanced inflation measurement using BERT and web scraping

TL;DR

This paper uses NLP and web scraping to create a custom inflation index that aligns closely with official measures while capturing detailed market fluctuations.

Contribution

A novel approach to inflation measurement using BERT and web scraping for detailed price tracking.

Findings

BERT achieved 94.56% accuracy in classifying consumer electronics into COICOP categories.

The custom index using weighted and median methods aligned closely with the HICP while capturing more detailed price changes.

Monthly trends showed variability in COICOP 091 contrasting with the stable HICP.

Abstract

In this research note, we explore the integration of natural language processing (NLP) and web scraping techniques to develop a custom price index for measuring inflation. Using the Harmonized Index of Consumer Prices (HICP) as a benchmark, we created a database of consumer electronics product data through web scraping. Using the BERT model for classification, we achieved a high-performance classification of approximately 10,000 items into COICOP categories, with an accuracy of 94.56 %, macro precision of 79.41 %, and weighted precision of 94.07 % on validation data. Our custom index, particularly with weighted and median methodologies, demonstrated closer alignment with the official HICP while capturing more detailed price fluctuations within the market. Monthly inflation trends revealed variability that reflects price changes in the COICOP 091 category, contrasting with the relative stability of the official HICP. This work provides an alternative perspective on inflation measurement, highlighting the potential of computational approaches to enhance economic analysis.

Figures