Exploring a New Competency Modeling Process with Large Language Models

TL;DR

This paper introduces a novel, data-driven competency modeling process using large language models that automates and enhances traditional expert-driven methods, improving validity, consistency, and reproducibility.

Contribution

It reconstructs the competency modeling workflow with LLMs, enabling automated extraction, mapping, and adaptive integration of behavioral data, along with a new validation procedure.

Findings

Strong predictive validity demonstrated in real-world application

High cross-library consistency achieved

Robustness of the structured framework confirmed

Abstract

Competency modeling is widely used in human resource management to select, develop, and evaluate talent. However, traditional expert-driven approaches rely heavily on manual analysis of large volumes of interview transcripts, making them costly and prone to randomness, ambiguity, and limited reproducibility. This study proposes a new competency modeling process built on large language models (LLMs). Instead of merely automating isolated steps, we reconstruct the workflow by decomposing expert practices into structured computational components. Specifically, we leverage LLMs to extract behavioral and psychological descriptions from raw textual data and map them to predefined competency libraries through embedding-based similarity. We further introduce a learnable parameter that adaptively integrates different information sources, enabling the model to determine the relative importance of behavioral and psychological signals. To address the long-standing challenge of validation, we develop an offline evaluation procedure that allows systematic model selection without requiring additional large-scale data collection. Empirical results from a real-world implementation in a software outsourcing company demonstrate strong predictive validity, cross-library consistency, and structural robustness. Overall, our framework transforms competency modeling from a largely qualitative and expert-dependent practice into a transparent, data-driven, and evaluable analytical process.