Large Language Models to the Rescue: Reducing the Complexity in Scientific Workflow Development Using ChatGPT

TL;DR

This paper explores how ChatGPT, a large language model, can assist users in understanding and modifying complex scientific workflows, highlighting its strengths and limitations through user studies.

Contribution

It demonstrates the potential of LLMs like ChatGPT to support scientific workflow development and identifies areas needing improvement.

Findings

LLMs effectively interpret workflows in scientific domains.

Performance drops when exchanging components or extending workflows.

Limitations of LLMs in complex workflow modifications are characterized.

Abstract

Scientific workflow systems are increasingly popular for expressing and executing complex data analysis pipelines over large datasets, as they offer reproducibility, dependability, and scalability of analyses by automatic parallelization on large compute clusters. However, implementing workflows is difficult due to the involvement of many black-box tools and the deep infrastructure stack necessary for their execution. Simultaneously, user-supporting tools are rare, and the number of available examples is much lower than in classical programming languages. To address these challenges, we investigate the efficiency of Large Language Models (LLMs), specifically ChatGPT, to support users when dealing with scientific workflows. We performed three user studies in two scientific domains to evaluate ChatGPT for comprehending, adapting, and extending workflows. Our results indicate that LLMs efficiently interpret workflows but achieve lower performance for exchanging components or purposeful workflow extensions. We characterize their limitations in these challenging scenarios and suggest future research directions.