Learning from Crowds with Crowd-Kit

TL;DR

Crowd-Kit is a versatile Python toolkit that streamlines the implementation and benchmarking of computational quality control algorithms for crowdsourcing, supporting diverse answer modalities and facilitating rapid prototyping.

Contribution

It introduces a comprehensive, open-source toolkit with dataset loaders, example notebooks, and evaluation tools for systematic benchmarking of quality control methods in crowdsourcing.

Findings

Effective benchmarking across multiple datasets

Support for various answer modalities

Open-source code and data release

Abstract

This paper presents Crowd-Kit, a general-purpose computational quality control toolkit for crowdsourcing. Crowd-Kit provides efficient and convenient implementations of popular quality control algorithms in Python, including methods for truth inference, deep learning from crowds, and data quality estimation. Our toolkit supports multiple modalities of answers and provides dataset loaders and example notebooks for faster prototyping. We extensively evaluated our toolkit on several datasets of different natures, enabling benchmarking computational quality control methods in a uniform, systematic, and reproducible way using the same codebase. We release our code and data under the Apache License 2.0 at https://github.com/Toloka/crowd-kit.