Scientific instruments such as light sources and telescopes can generate data at rates of multiple gigabits per second (Gbps). This data, if analyzed in real time, can increase the science productivity using these expensive instruments. Such real-time analysis, however, requires extensive compute resources, which are often not available in-house at the experimental facilities. Thus, the data needs to be moved, usually over a wide area network that traverses multiple domains.

To tackle this challenge, researchers from four U.S. Department of Energy (DOE) national laboratories – Argonne, Brookhaven, Los Alamos and Oak Ridge – conducted a demonstration involving automatic provisioning of multidomain software-defined network resources on demand. This work is part of the Software Defined Network Science Flows (SDN-SF) project funded by DOE’s SDN-Enabled Terabits Optical Networks for Extreme-Scale Science program. 

The team streamed data generated by light source experiments from Brookhaven to Argonne at ~10 Gbps. The task required setting up network resources end to end automatically by using SDN application program interfaces, then dynamically transferring the data and reconstructing a portion of it in real time.

The demo was led by Rajkumar Kettimuthu, an Argonne computer scientist and co-principal investigator (PI) of the SDN-SF project. Kettimuthu attributed much of the success of the demo to the extensive testing of network provisioning tools developed by Argonne postdoctoral appointee Joaquin Chung; to the effective support of the SDN-SF team from other labs; and to a high-performance tool called Trace, developed by Argonne assistant computational scientist Tekin Bicer.

“Joaquin’s testing helped us find the incompatibilities in network provisioning APIs early enough to address them before the demo,” Kettimuthu said. ​“With Trace we were able to do streaming analysis of a shale sample obtained from a light source experiment and obtain the 3-D reconstruction from 2-D projections of the data.”

The timeliness of demonstration was emphasized by Ian Foster, director of the Data Science and Learning division at Argonne. ​“Real-time analysis of large-scale data is becoming a critical need for a number of experimental, observational and simulation science workflows,” Foster said. ​“With this new approach, scientists at multiple institutions can share data and process results rapidly and can use the knowledge, for example, to help guide (the next steps of) ongoing experiments.”

Software-defined networks have been successfully deployed in many commercial settings, but their adoption in science environments has been limited. ​“One reason is the challenge of developing solutions tailored to the strict security policies of DOE site organizations,” said Nageswara S.V. Rao, UT-Battelle corporate fellow at Oak Ridge and lead-PI of the SDN-SF project. ​“These security policies are created with significant attention paid to protecting these valuable resources, and we need to respect and comply with them,” Rao said. ​“We would like to develop best practices to realize the required programmability in the network under the strict security settings of these sites.”

Live demonstration of experimental features involving resources at multiple institutions is a complex task under unpredictable conditions. So, the team had a primary plan and a backup plan. In fact, the primary plan failed because of an unexpected hardware failure, and the team resorted to the backup plan.

“Having the tools deployed and tested at multiple institutions came in handy,” said Dimitrios Katramatos, a technology architect at Brookhaven and co-PI of the SDN-SF project. ​“Our extensive experience with OSCARS – a tool to provision resources on the Energy Sciences Network – and our partnership with the OSCARS team helped to overcome version incompatibilities in OSCARS in short order and complete the demo successfully,” Katramatos said.

Despite some previous efforts, automated provisioning of last-mile network resources – hence affording a simple and seamless way to provision network resources end to end – has remained elusive. ​“Using software-defined networking is a great way to provision network resources for applications that generate data in memory at one location and process them in memory at a remote location by moving them over the network,” said Bradley Settlemyer, a research scientist at Los Alamos and co-principal investigator of the SDN-SF project.

 ​“A number of applications move data into the storage system before processing them. Provisioning the associated resources including the storage area networks involves several challenges, and our team is addressing those challenges as well,” Settlemyer said.  

The demonstration took place at SC18, the premier international conference on high-performance computing, networking, storage and analysis.