Multilingual hierarchical classification of job advertisements for job vacancy statistics

TL;DR

This paper presents a multilingual hierarchical transformer-based classifier for occupation codes in online job ads, improving accuracy and supporting international occupational classification standards.

Contribution

It introduces a novel hierarchical classification model utilizing transformer architecture and multilingual data, including a new Central Job Offers Database, for occupation coding.

Findings

Hierarchical structure improves prediction accuracy by 1-2 percentage points.

The model supports 24 languages, enhancing international comparability.

Open-source software is provided for the statistics community.

Abstract

The goal of this paper is to develop a multilingual classifier and conditional probability estimator of occupation codes for online job advertisements in accordance with the International Standard Classification of Occupations (ISCO) extended with the Polish Classification of Occupations and Specializations (KZiS), which is analogous to the European Classification of Occupations. In this paper, we utilise a range of data sources, including a novel one, namely the Central Job Offers Database, which is a register of all vacancies submitted to Public Employment Offices. Their staff members code the vacancies according to the ISCO and KZiS. A hierarchical multi-class classifier has been developed based on the transformer architecture. The classifier begins by encoding the jobs found in advertisements to the widest 1-digit occupational group, and then narrows the assignment to a 6-digit occupation code. We show that incorporation of the hierarchical structure of occupations improves prediction accuracy by 1-2 percentage points, particularly for the hand-coded online job advertisements. Finally, a bilingual (Polish and English) and multilingual (24 languages) model is developed based on data translated using closed and open-source software. The open-source software is provided for the benefit of the official statistics community, with a particular focus on international comparability.