Enabling Seamless Transitions from Experimental to Production HPC for Interactive Workflows

TL;DR

This paper introduces a structured transition pathway at OLCF that enables scientific workflows to move seamlessly from experimental to production HPC environments, supporting interactive and real-time data analysis.

Contribution

It presents a comprehensive framework combining technological and policy solutions for transitioning HPC workflows from experimental to production settings.

Findings

Developed data streaming architectures for HPC workflows

Implemented secure service interfaces for transition

Enhanced resource scheduling for interactive workloads

Abstract

The evolving landscape of scientific computing requires seamless transitions from experimental to production HPC environments for interactive workflows. This paper presents a structured transition pathway developed at OLCF that bridges the gap between development testbeds and production systems. We address both technological and policy challenges, introducing frameworks for data streaming architectures, secure service interfaces, and adaptive resource scheduling for time-sensitive workloads and improved HPC interactivity. Our approach transforms traditional batch-oriented HPC into a more dynamic ecosystem capable of supporting modern scientific workflows that require near real-time data analysis, experimental steering, and cross-facility integration.