Crowd-SFT: Crowdsourcing for LLM Alignment

TL;DR

Crowd-SFT introduces a scalable, fair crowdsourcing framework for fine-tuning large language models, reducing costs and bias while maintaining alignment quality through incentive mechanisms and iterative updates.

Contribution

It presents an open crowdsourcing approach for LLM fine-tuning that enhances scalability and fairness without requiring extensive annotator training.

Findings

Achieved up to 55% reduction in target distance with multi-model selection.

Validated the point-based reward system's alignment with Shapley values.

Demonstrated improved scalability and fairness in LLM alignment processes.

Abstract

Large Language Models (LLMs) increasingly rely on Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF) to align model responses with human preferences. While RLHF employs a reinforcement learning approach with a separate reward model, SFT uses human-curated datasets for supervised learning. Both approaches traditionally depend on small, vetted groups of annotators, making them costly, prone to bias, and limited in scalability. We propose an open, crowd-sourced fine-tuning framework that addresses these limitations by enabling broader feedback collection for SFT without extensive annotator training. Our framework promotes incentive fairness via a point-based reward system correlated with Shapley values and guides model convergence through iterative model updates. Our multi-model selection framework demonstrates up to a 55% reduction in target distance over single-model selection, enabling subsequent experiments that validate our point-based reward mechanism's close alignment with Shapley values (a well-established method for attributing individual contributions) thereby supporting fair and scalable participation.