Integrating Oak Ridge Leadership Computing Facility’s High Performance Computing and Neutron Facilities with a Performance-Portable CPU/GPU Ecosystem
Abstract: We explore the development of a performance-portable CPU/GPU ecosystem to integrate the Oak Ridge Leadership Computing Facility (OLCF) and the Spallation Neutron Source (SNS), both of which are housed at Oak Ridge National Laboratory.
We select a data reduction workflow use-case to obtain the differential scattering cross-section from data collected by SNS’s CORELLI and TOPAZ instruments. Inconvenient execution times from complex data processing techniques and large volumes of data discourages users from utilizing all experimental data.
The current CPU-only production implementation using the Garnet Python multiprocess package based on the C++ Mantid framework is compared against our proposed CPU/GPU implementation that uses the LLVM-based Julia scientific language and the JACC.jl performance portability package. To understand and address performance challenges, two proxy apps were developed: (i) an app for extracting relevant Mantid kernels (MDNorm) in C++ and (ii) the Julia MiniVATES.jl miniapp.
We introduce algorithmic improvements: (i) parallelization strategies across multiple GPU nodes, (ii) reformulating key calculations into a tall-skinny matrix multiplication kernel and (iii) exploring intermediate data layout representations for optimizing reading and processing times. Performance results from NVIDIA A100, H200 and AMD MI100 GPUs and AMD EPYC and NVIDIA Grace CPUs will be presented.
These computational experiments show 10x+ speed ups and provide insights for future generations of data reduction software that can take advantage of developments in productivity and performance portability, making high performance computing more accessible for an integrated research infrastructure across DOE’s experimental and computational facilities.
Bio: Steven Hahn is a software scientist in the Computer Science and Mathematics Division at ORNL. He received his Ph.D. in Condensed Matter Physics from Iowa State University in 2012.