Smart anomaly detection for scientific workflows

Sudden slowdowns in execution time or unexpected errors in code can seriously impact an application’s performance. Detecting these issues — known as anomalies — is considered key to ensuring the reliability and reproducibility of large-scale scientific workflows. But training models to detect anomalies requires massive amounts of data, and collecting such data is both time consuming and resource intensive. 

A method called active learning — a special case of machine learning — is a promising approach to this problem. The active learning method selects the most informative data points to label in order to improve model performance efficiently. However, the present approach is useful only in scenarios where unlabeled data is abundant, but labeling is expensive and time consuming or requires expert input. Expensive experiments must still be executed. 

To address these concerns, researchers from the U.S. Department of Energy’s Argonne National Laboratory, the University of North Carolina, Renaissance Computing Institute (RENCI) and DOE’s Oak Ridge National Laboratory have developed a new framework that detects anomalies in workflows while generating data as needed — making the process faster and more efficient. 

As illustrated in Figure 1, the new framework has three main components: a neural network that can use either labeled or unlabeled data to learn patterns; an experimental infrastructure, called Poseidon-X, that executes experiments and generates the data on the fly; and an active learning module that gathers new training data to keep improving the model. 

Poseidon-X runs on two infrastructures — FABRIC and Chameleon Cloud. FABRIC is a testbed supporting advanced networking, application and security research across wide geographical areas (for example, from local to global); and Chameleon offers powerful computing resources. Researchers use a tool called Pegasus to run scientific workflows on both infrastructures, collect statistics and train models. Poseidon-X can inject different types and levels of anomalies, which are tracked automatically and correlated with the workflows to help train the system. 

To demonstrate their approach, the researchers used three workflows, including one live experiment. Active learning was employed to sample the data and guided by measurements of the confidence of the model, determine the next task. 

“We focused on low-confidence regions to trigger new experiments in areas of uncertainty,” said Krishnan Raghavan, an assistant computer scientist in Argonne’s Mathematics and Computer Science division and leader of the study. ​“This approach might seem the opposite of what one normally would select, but we’ve found that the biggest improvements come from these regions.”

The results are impressive. For example, for the 1000Genome workflow shown in Figure 2, the active learning approach achieves far higher accuracy than does the standard learning model without active learning. 

Moreover, the results indicate that by selectively choosing from the full data and explicitly leveraging uncertainty, the new approach uses far fewer computational resources.  

“This is a significant step forward in building machine learning systems that don’t need massive amounts of training data to detect anomalies,” Raghavan said. 

For further information, see the paper by Krishnan Raghavan, George Papadimitriou, Hongwei Jin, Anirban Mandal, Mariam Kiran, Ewa Deelman, and Prasanna Balaprakash, ​“Advancing anomaly detection in computational workflows with active learning,” Future Generation Computer Systems 166 (2025): 107608, https://​arx​iv​.org/​a​b​s​/​2​4​0​5​.​06133