Adaptively selecting occupations to detect skill shortages from online job ads

TL;DR

This paper presents a data-driven method to detect skill shortages in real-time from online job ads, focusing on Data Science and Analytics skills in Australia, using adaptive similarity techniques and five key indicators.

Contribution

It introduces a novel adaptive skills similarity method and five variables for real-time skill shortage detection from large-scale online job ad data.

Findings

Identified 306,577 DSA job ads across 23 occupations from 2012-2019.

Detected significant skills shortages in Australia for technical DSA skills.

Provided insights for policymakers and educators on future skill demands.

Abstract

Labour demand and skill shortages have historically been difficult to assess given the high costs of conducting representative surveys and the inherent delays of these indicators. This is particularly consequential for fast developing skills and occupations, such as those relating to Data Science and Analytics (DSA). This paper develops a data-driven solution to detecting skill shortages from online job advertisements (ads) data. We first propose a method to generate sets of highly similar skills based on a set of seed skills from job ads. This provides researchers with a novel method to adaptively select occupations based on granular skills data. Next, we apply this adaptive skills similarity technique to a dataset of over 6.7 million Australian job ads in order to identify occupations with the highest proportions of DSA skills. This uncovers 306,577 DSA job ads across 23 occupational classes from 2012-2019. Finally, we propose five variables for detecting skill shortages from online job ads: (1) posting frequency; (2) salary levels; (3) education requirements; (4) experience demands; and (5) job ad posting predictability. This contributes further evidence to the goal of detecting skills shortages in real-time. In conducting this analysis, we also find strong evidence of skills shortages in Australia for highly technical DSA skills and occupations. These results provide insights to Data Science researchers, educators, and policy-makers from other advanced economies about the types of skills that should be cultivated to meet growing DSA labour demands in the future.