High-Performance Workflows: In Situ Data Transport and Workflow Management

Abstract: In situ workflows have gained traction in the high-performance computing (HPC) community because of the need to analyze increasing data volumes together with the ever-growing gap between the computation and I/O capabilities of HPC systems. In situ workflows run within a single HPC system as a collection of multiple tasks, which are often large and parallel programs. These tasks communicate over memory or the interconnect of the HPC system, bypassing the parallel file system. We will present our latest progress in data transport and workflow management in HPC in situ workflows.