Data analysis at the speed of light source experiments

Why it matters

• Integrated infrastructure: By connecting APS beamlines with ALCF supercomputers, Argonne is enabling rapid analysis of the massive datasets produced by the upgraded light source.
• Experiment-time analysis: Researchers can analyze X-ray data as it is generated rather than waiting until an experiment ends.
• Steering experiments: Real-time feedback allows scientists to adjust samples, refine hypotheses and change course while experiments are underway.
• A new research model: The work advances DOE’s efforts to accelerate discovery science by bringing data-intensive experiments, high performance computing and AI resources together into a unified platform.

At the upgraded Advanced Photon Source (APS), a powerful X-ray light source at the U.S. Department of Energy’s (DOE) Argonne National Laboratory, new analysis capabilities are changing how experiments unfold. Instead of waiting until an experiment ends, researchers can now use near real-time feedback from the X-ray beamlines to guide their next steps.

“Coming back to the APS after the upgrade was a completely different experience,” said Ryan Poling-Skutvik, assistant professor of chemical engineering at the University of Rhode Island. ​“Based on the early signals we were getting from our experiments, we were able to bring materials to the wet lab and make new samples to better target the dynamics we were studying. That’s something that’s impossible to do if you don’t have the real-time data.”

After its recent upgrade, the APS produces X-ray beams that are up to 500 times brighter than before, enabling researchers to study materials with higher resolution and at time scales that were previously out of reach. But those enhanced capabilities are also producing more data than its local computing systems can handle.

“The fact that it was so frictionless allowed me to focus on the science I wanted to go after rather than the details of how to manage the data and run the analyses.” — Ryan Poling-Skutvik, University of Rhode Island

“When I started using the APS around 2015, the fastest time scales we could probe were on the order of milliseconds,” Poling-Skutvik said. ​“Now we’re collecting data on microsecond time scales, which opens up a much broader view of material behavior. At that rate, you can imagine collecting 10,000 frames in less than a second, which would completely swamp everything. Having analysis proceed at a comparable speed allows us to fully realize the potential of the upgraded beamline technology.”

To make this possible, APS experiments are now tightly integrated with supercomputers at the Argonne Leadership Computing Facility (ALCF). Building on years of collaboration between APS and ALCF, Argonne researchers created an automated pipeline that streams experimental data from the beamlines to ALCF systems for analysis as it is being collected. The APS and ALCF are DOE Office of Science user facilities.

The Argonne team’s work is helping to advance DOE’s Genesis Mission, a national AI initiative to build a powerful scientific platform for accelerating discovery science, strengthening national security and driving energy innovation. In particular, the rapid data analysis capabilities have prepared the way for new efforts under the American Science Cloud (AmSC), a cornerstone of the Genesis Mission. AmSC is an integrated, federated platform that connects AI models, curated scientific data, workflows and computing resources across DOE laboratories.

Linking light and compute

The connection between APS and ALCF is powered by the APS Data Management System and Globus. The APS Data Management System provides a uniform way to connect to data from the approximately 100 unique instruments at the APS. It also keeps track of information about data and experiments at the facility. Globus, a research automation and data management platform developed at Argonne and the University of Chicago, handles the movement of data between the APS and the ALCF’s Polaris supercomputer, automatically running analyses and returning results to the beamline while experiments are still underway.

“The actual data collection is triggering all of the data movement — the storage, the access permissions, the processing on Polaris and the transfer back to the APS,” said Thomas Uram, ALCF data services and workflows team lead. ​“All of this is happening without any intervention by the scientists.”

Making this seamless access possible involved more than simply connecting APS instruments to remote supercomputers. It also meant changing long-standing policies and operational practices around how DOE computing resources are accessed and used during live experiments.

Traditionally, supercomputing facilities rely on individual user accounts and standard job queues, which can slow down time-sensitive experiments. To overcome these hurdles, the ALCF implemented service accounts — secure credentials tied to a specific experiment rather than an individual researcher — and on-demand queues, which dedicate supercomputer nodes for immediate processing of experimental workloads.

Bringing all the pieces together required extensive collaboration. Teams spanning X-ray science, beamline operations, data management and scientific software worked alongside ALCF staff and Globus developers to map out how each beamline collects data, when to launch processing workflows and how to best integrate APS control systems with Globus and remote supercomputers.

“By combining the expertise of multiple teams with powerful computing resources, we were able to build reliable data processing pipelines that can return analysis results quickly enough to guide experiments as they happen,” said Hannah Parraga, a software engineer at the APS developing scientific data workflows that run on supercomputers for many of the facility’s beamlines.

With this infrastructure in place, researchers visiting the APS are now using the analysis tools across many of the facility’s beamlines, taking full advantage of the brighter X-ray beams and faster detectors provided by the upgrade. While Polaris is the primary system currently supporting APS experiments, researchers will also be able to tap ALCF’s Aurora exascale system and next-generation supercomputers for increasingly data-intensive work.

One of the first beamlines to employ the enhanced computing capabilities is the X-ray Photon Correlation Spectroscopy (XPCS) beamline. XPCS enables scientists to observe how materials behave over time and under different conditions at the nanoscale.

“We’re already seeing how the faster analysis is helping researchers steer XPCS experiments in ways that weren’t possible before,” said Suresh Narayanan, APS physicist and group leader. ​“Our users can adapt their experimental setup on the fly, adjust their hypotheses as new data comes in and make more efficient use of their beam time.”

Probing soft materials with experiment-time feedback

Poling-Skutvik’s team is using XPCS to study soft materials, a broad class of materials found in everyday products like shampoos and paints as well as biological systems such as cells and tissues. Understanding their dynamics is essential for designing materials that move and respond predictably under stress.

“What we’re really asking is how can we design these materials to be more functional,” Poling-Skutvik said. ​“One of the biggest challenges is that we don’t have a good understanding of the dynamics that are present inside of these materials.”

XPCS is uniquely suited to tackle such problems, allowing researchers to probe motion deep within materials at the length and time scales relevant to soft matter. With faster analysis, Poling-Skutvik’s team can now get early insight into those dynamics while an experiment is happening, rather than reconstructing them after the fact.

This rapid feedback is particularly useful because many soft materials can be synthesized quickly by adjusting physical parameters like salt concentration. During recent experiments, Poling-Skutvik’s team brought a range of candidate materials and used early measurements to guide what they tested next.

“We could make new samples with different molecular weights and concentrations, put them on the beamline to see the dynamics, and then go back to the lab to modify the next samples,” he said. ​“With the ability to process the dynamics really fast, we were able to iterate through multiple designs within a day.”

Making beam time count

Researchers from the University of Texas and the University of Michigan are taking advantage of the rapid analysis capabilities in XPCS experiments involving metal oxide nanocrystals. Their work aims to shed light on how nanocrystals assemble into gel-like networks with tunable optical and electronic properties.

“We have a very good plan going into our experiments, but we like to treat the beam time as this living and breathing thing because XPCS allows you to see things you cannot see anywhere else at conditions you cannot measure anywhere else,” said William Brackett, a graduate student at the University of Texas. ​“You want to be flexible because the data often reveals unexpected or interesting results worth deeper investigation.”

Before the APS-ALCF integration, the team typically collected data that would be analyzed long after beam time ended. Now, each dataset triggers an automated analysis on Polaris, with results returning to the beamline in minutes. That speed proved especially valuable when studying gel systems that evolved at very different rates.

The team was able to quickly identify whether a gelation process would be fast or slow, enabling them to adjust their experimental plan accordingly. ​“The name of the game when you have beam time is to maximize your efficiency,” Brackett said. ​“We were able to group gels by their gelation time and achieve much higher experimental throughput during our allotted time.”

The quick analysis also helped the team zero in on a narrow temperature window where a material rapidly switches between liquid and gel states. With immediate feedback, they were able to refine their measurements on the spot.

“We needed to figure out how the dynamics evolve in a quickly arresting gel, and that wouldn’t have been possible unless we could have seen the data right then and there,” Brackett said. ​“That allowed us to tweak some of the experiments to isolate the temperatures and get a more resolved idea of what’s going on around that gel point.”

Adapting experiments as data comes in

In another XPCS experiment, researchers from the Olsen Lab at the Massachusetts Institute of Technology (MIT) are using the beamline to study complex, disordered materials.

“Our research bridges the realm between protein and polymer dynamics,” said Brian Carrick, a postdoctoral researcher at MIT. Gaining insights into the behavior of these materials could help inform the design of recyclable plastics and self-healing materials.

“We deal with a lot of systems with reversible bonds,” Carrick said. ​“You can think of it like Silly Putty. You can rip it into two pieces and put them back together, and it kind of heals. We’re trying to understand the molecular characteristics of these healing processes so we can make better recyclable materials.”

Before the APS upgrade, experiments often came with limited feedback during beam time. Without the ability to analyze data quickly, Carrick’s team had less visibility into how samples were responding during experiments, including whether the X-rays were altering the materials.

“We’ve tried running these exact same materials on XPCS in the past, but we could never analyze the data in real time,” he said. ​“So, everything we collected was either damaged by radiation, or we just couldn’t get a good enough signal. And since we couldn’t reduce the data on the beamline, we couldn’t really quantify any of it.”

With the enhanced data analysis capabilities in place, that constraint has been removed. ​“We were able to perform a measurement and less than three minutes later get our data back,” Carrick said. This allowed the team to screen samples for stability, tune exposure conditions and decide what to measure next while their experiments were still underway.

“With this kind of real-time analysis, you can start with a hypothesis at point A and then evolve the questions you’re trying to probe as your understanding grows,” Carrick said. ​“Because you have that flexibility and can see your data in real time, you can push the frontier a little bit faster.”

A new model for experiment-driven discovery

Argonne continues its work to extend these data processing capabilities to more APS beamlines and other experimental facilities, enabling scientists to integrate high performance computing seamlessly into their workflows to speed up the pace of discovery.

At the APS, that shift from delayed analysis to near-instant feedback is already changing how experiments are designed, executed and refined. With computing infrastructure operating smoothly in the background, scientists like Poling-Skutvik can concentrate on their experiments rather than data management and processing.

“The fact that it was so frictionless allowed me to focus on the science I wanted to go after rather than the details of how to manage the data and run the analyses,” Poling-Skutvik said. ​“That’s the best-case scenario — when the infrastructure exists to let you do what you need to do without limiting your ability to do it.”

This work was supported in part by the DOE Office of Science’s Advanced Scientific Computing Research (ASCR) and Basic Energy Sciences programs. Access to ALCF computing resources was provided through the DOE ASCR Leadership Computing Challenge award, ​“Enhancing APS-Enabled Research through Integrated Research Infrastructure,” led by Argonne’s Nicholas Schwarz. Additional funding was provided by DOE’s AmSC project.