HelpSteer: Multi-attribute Helpfulness Dataset for SteerLM

TL;DR

HelpSteer is a multi-attribute dataset that annotates responses for correctness, coherence, complexity, and verbosity, enabling more nuanced training of helpfulness models like SteerLM, which achieves state-of-the-art scores without relying on proprietary data.

Contribution

This paper introduces HelpSteer, a comprehensive multi-attribute helpfulness dataset, and demonstrates its effectiveness in training models that outperform existing open models on helpfulness benchmarks.

Findings

SteerLM trained on HelpSteer achieves 7.54 on MT Bench.

HelpSteer dataset covers multiple helpfulness attributes.

Models trained on HelpSteer outperform previous open models.

Abstract

Existing open-source helpfulness preference datasets do not specify what makes some responses more helpful and others less so. Models trained on these datasets can incidentally learn to model dataset artifacts (e.g. preferring longer but unhelpful responses only due to their length). To alleviate this problem, we collect HelpSteer, a multi-attribute helpfulness dataset annotated for the various aspects that make responses helpful. Specifically, our 37k-sample dataset has annotations for correctness, coherence, complexity, and verbosity in addition to overall helpfulness of responses. Training Llama 2 70B using the HelpSteer dataset with SteerLM technique produces a model that scores 7.54 on MT Bench, which is currently the highest score for open models that do not require training data from more powerful models (e.g. GPT4). We release this dataset with CC-BY-4.0 license at https://huggingface.co/datasets/nvidia/HelpSteer