HR-MultiWOZ: A Task Oriented Dialogue (TOD) Dataset for HR LLM Agent

TL;DR

This paper introduces HR-MultiWOZ, a novel, fully-labeled dataset of 550 HR-related conversations designed to facilitate NLP research and development of LLM agents in the HR domain, addressing privacy and data scarcity issues.

Contribution

It presents the first open-source HR conversation dataset, a detailed, adaptable data generation pipeline using LLMs with minimal human effort, and comprehensive data analysis and evaluation.

Findings

First HR domain-specific conversation dataset

Data generation pipeline is efficient and adaptable

Human evaluations confirm data quality

Abstract

Recent advancements in Large Language Models (LLMs) have been reshaping Natural Language Processing (NLP) task in several domains. Their use in the field of Human Resources (HR) has still room for expansions and could be beneficial for several time consuming tasks. Examples such as time-off submissions, medical claims filing, and access requests are noteworthy, but they are by no means the sole instances. However, the aforementioned developments must grapple with the pivotal challenge of constructing a high-quality training dataset. On one hand, most conversation datasets are solving problems for customers not employees. On the other hand, gathering conversations with HR could raise privacy concerns. To solve it, we introduce HR-Multiwoz, a fully-labeled dataset of 550 conversations spanning 10 HR domains to evaluate LLM Agent. Our work has the following contributions: (1) It is the first labeled open-sourced conversation dataset in the HR domain for NLP research. (2) It provides a detailed recipe for the data generation procedure along with data analysis and human evaluations. The data generation pipeline is transferable and can be easily adapted for labeled conversation data generation in other domains. (3) The proposed data-collection pipeline is mostly based on LLMs with minimal human involvement for annotation, which is time and cost-efficient.