Data-Juicer 2.0: Cloud-Scale Adaptive Data Processing for and with Foundation Models

TL;DR

Data-Juicer 2.0 is a scalable, versatile data processing system designed for multimodal datasets, enhancing foundation model workflows with improved efficiency, usability, and compatibility across large-scale environments.

Contribution

It introduces a comprehensive, adaptive data processing framework supporting multimodal data and integrates seamlessly with popular data hubs and computing engines, advancing prior systems in scalability and usability.

Findings

Processes TB-level data with 10k+ CPU cores efficiently.

Supports diverse data modalities including text, image, video, and audio.

Widely adopted in research and industry, including Alibaba Cloud PAI.

Abstract

Foundation models demand advanced data processing for their vast, multimodal datasets. However, traditional frameworks struggle with the unique complexities of multimodal data. In response, we present Data-Juicer 2.0, a data processing system backed by 100+ data processing operators spanning text, image, video, and audio modalities, supporting more critical tasks including data analysis, synthesis, annotation, and foundation model post-training. With seamless compatibility and dedicated optimization for popular dataset hubs like Hugging Face and computing engines like Ray, it improves upon its predecessor in terms of usability, efficiency, and programmability. It features an easily accessible user interface layer that supports decoupled Python interactions, RESTful APIs, and conversational commands. Its new runtime layer offers adaptive execution across diverse scales and environments, abstracting away system complexities. Extensive empirical evaluations demonstrate Data-Juicer 2.0's remarkable performance and scalability, highlighting its capability to efficiently process TB-level data with 10k+ CPU cores. The system is publicly available and has been widely adopted in diverse research fields and real-world products such as Alibaba Cloud PAI. We actively maintain the system and share practical insights to foster research and applications of next-generation foundation models.