Managing, Analyzing and Sharing Research Data with Gen3 Data Commons

TL;DR

Gen3 is an open-source platform that simplifies building, managing, and sharing large-scale research data commons with automated data portals and FAIR APIs, supporting interoperability and community access.

Contribution

This paper introduces Gen3, a scalable, standards-based platform that automates data portal generation and API access for research data commons.

Findings

Over 28 PB of data managed across multiple commons

Automated generation of data portals and APIs

Supports interoperability with other data platforms

Abstract

Gen3 is an open-source data platform for building data commons. A data commons is a cloud-based data platform for managing, analyzing, and sharing data with a research community. Gen3 has been used to build over a dozen data commons that in aggregate contain over 28 PB of data and 64 million FAIR data objects. To set up a Gen3 data commons, you first define a data model. Gen3 then autogenerates 1) a data portal for searching and exploring data in the commons; 2) a data portal for submitting data to the commons; and 3) FAIR APIs for accessing the data programmatically. Gen3 is built over a small number of standards-based software services, which are designed to support current and future Gen3 components so that Gen3 can interoperate with other data platforms and data ecosystems.