JobPulse: A Big Data Approach to Real-Time Engineering Workforce Analysis and National Industrial Policy

TL;DR

This paper presents a big data approach to analyze real-time demand for engineering jobs in the semiconductor industry, revealing workforce needs and skill mismatches in Southern California.

Contribution

It introduces a novel data processing scheme using web scraping to estimate job market demand and mismatches from job postings, focusing on a critical U.S. industry sector.

Findings

Provides near real-time insights into workforce demand.

Addresses semantic challenges in large-scale employer data analysis.

Details the semiconductor engineering ecosystem in Southern California.

Abstract

Employment on a societal scale contributes heavily to national and global affairs; consequently, job openings and unemployment estimates provide important information to financial markets and governments alike. However, such reports often describe only the supply (employee job seeker) side of the job market, and skill mismatches are poorly understood. Job postings aggregated on recruiting platforms illuminate marketplace demand, but to date have primarily focused on candidate skills described in their personal profiles. In this paper, we report on a big data approach to estimating job market mismatches by focusing on demand, as represented in publicly available job postings. We use commercially available web scraping tools and a new data processing scheme to build a job posting data set for the semiconductor industry, a strategically critical sector of the United States economy; we focus on Southern California as a central hub of advanced technologies. We report on the employer base and relative needs of various job functions. Our work contributes on three fronts: First, we provide nearly real-time insight into workforce demand; second, we discuss disambiguation and semantic challenges in analysis of employer data bases at scale; and third, we report on the Southern California semiconductor engineering ecosystem.