Skip to main content
Publication

Experimental Evaluation of a Flexible I/O Architecture for Accelerating Workflow Engines in Ultrascale Environments

Authors

Duro, Francisco; Garcia Blas, Javier; Isaila, Florin; Carretero, Jesus; Wozniak, Justin; Ross, Rob

Abstract

The increasing volume of scientific data and the limited scalability and performance of storage systems are currently presenting a significant limitation for the productivity of the scientific workflows running on both high-performance computing (HPC) and cloud platforms. Clearly needed is better integration of storage systems and workflow engines to address this problem. This paper presents and evaluates a novel solution that leverages codesign principles for integrating Hercules—an in-memory data store—with a workflow management system. We consider four main aspects: workflow representation, task scheduling, task placement, and task termination. The experimental evaluation on both cloud and HPC systems demonstrates significant performance and scalability improvements over existing state-of-the-art approaches.