Making complex workflow systems easy to use

“Simple things should be simple; complex things should be possible.” That quote from computer pioneer Alan Kay reflects his life-long approach to programming languages, interfaces and code writing. A similar thought motivated a team of researchers from the U.S. Department of Energy’s Argonne National Laboratory and Lawrence Berkeley National Laboratory as they worked to improve how scientists run and manage complex computing tasks.

Their focus was on ​“in situ workflows.” Such workflows let researchers analyze data as it is being created—without having to move or save large amounts of it first. While powerful, however, these workflows can be difficult to use. Task codes are coupled, use different formats or work at different speeds. Adding artificial intelligence (AI) tasks can make things even more complicated.

“We wanted in situ workflow systems to be easier to use, while still being scalable and flexible,” said Orcun Yildiz, an assistant computer scientist in Argonne’s Mathematics and Computer Science division.  ​“And we wanted to support topologies ranging from simple linear workflows to complex ensembles involving many connected tasks.”

Their solution is a system called Wilkins.

How Wilkins Works

Wilkins is made up of four main parts that transfer data, define how data is organized, execute tasks and coordinate the whole workflow. It uses the LowFive library, built on top of the widely used HDF5 data model, to handle redistribution and communication between different tasks or processes. 

The big advantage of Wilkins is that it seamlessly couples scientific tasks without requiring code modifications; users just provide a configuration file and the task codes. They don’t have to spell out how tasks depend on each other – Wilkins figures that out from the input and output names. Users can also customize Wilkins to match their specific scientific objectives. 

From Complex Workflows to Simple Setup

How does Wilkins perform in real-world scientific experiments? 

In one test the research team used Wilkins to run a molecular dynamics simulation (LAMMPS) in situ with an instrument that detects crystals in a diamond-shaped lattice. This kind of experiment usually involves many small simulations with slightly different configurations. Using Wilkins, the team ran LAMMPS with a water model comprising 4,360 atoms over a million timesteps. The results, shown in Figure 1, indicate that increasing the number of ensembles from 1 to 64 didn’t slow things down much—less than 2% overhead. 

Best of all, they didn’t need to change the simulation or detector source code with Wilkins. To define the ensemble tasks, they had to add only one line, the number of task instances, to the configuration file. Wilkins took care of the rest — creating the communication channels between the tasks and generating a graph representing the workflow configuration automatically.

In another experiment the researchers tested a cosmology workflow. Again, only two small changes to the configuration file were needed to customize it, and no code changes were required. 

“These examples show that Wilkins makes it much easier to handle complex scientific workflows,” said Yildiz. The team is now exploring ways to combine Wilkins with AI tools and high-performance computing for advanced heterogeneous workflows. 

For details about the study, see the paper by O. Yildiz, D. Morozov, A. Nigmetov, B. Nicolae, T. Peterka, ​“Wilkins: HPC In Situ Workflows Made Easy,” Frontiers in High Performance Computing Journal, 2, 2024, https://​www​.fron​tiersin​.org/​j​o​u​r​n​a​l​s​/​h​i​g​h​-​p​e​r​f​o​r​m​a​n​c​e​-​c​o​m​p​u​t​i​n​g​/​a​r​t​i​c​l​e​s​/​1​0​.​3​3​8​9​/​f​h​p​c​p​.​2​0​2​4​.​1​4​7​2​7​1​9​/full.