Keahey leads workshop on practical reproducibility in computer science

Reproducibility has become an important concern in high-performance computer (HPC) science. Numerous questions arise, both in packaging an experiment for reproducibility and in reproducing the experiment itself. What resources does the experiment need, and how should they be configured? In what kind of environment should the experiment run? What obstacles have researchers encountered? How does prioritizing reproducibility affect an application? How can the HPC community better integrate reproducibility into mainstream research and education? 

These questions and more were the focus of the Community Workshop on Practical Reproducibility in HPC, held November 18, 2024, in Atlanta, Georgia. The full-day workshop brought together educators, researchers, and infrastructure providers to bridge the gap between theoretical reproducibility and its practical application in HPC research. 

The fifth in a series of annual Chameleon User Meeting, this year’s event focused on sharing experiences particularly in working with the Chameleon testbed.

“As a configurable experimental environment for computer science research, Chameleon provides an excellent platform for creating and sharing reproducible research,” said Kate Keahey, organizer of the workshop, Chameleon PI, and a senior computational scientist in the Mathematics and Computer Science Division at Argonne National Laboratory.

Keahey emphasized the use of the term ​“practical reproducibility” in the workshop title. It refers to ​“reproducibility that is easy enough to be a mainstream method of research sharing.” Instead of just presenting results, scientists could also share aspects of how the results were obtained, enabling other researchers to explore new ideas, for example, by changing the experiment parameters.

To illustrate Chameleon’s benefits in making reproducibility for HPC experiments practical, Keahey delivered a keynote address titled ​“Adaptable Infrastructure for Reproducible Science – The Chameleon 4 Approach.” She discussed how Chameleon 4 extends its deeply reconfigurable edge-to-cloud architecture to support emerging research needs – through enhanced virtualization capabilities, expanded edge computing functionalities, and advanced mechanisms for sharing digital artifacts.

Several of the other workshop speakers discussed their experiences with Chameleon’s cloud-based infrastructure. Topics ranged from approaches for assessing reproducibility to performance optimization strategies to reproducible package management. 

“The workshop was a big success in encouraging open dialogue about tools, services and approaches that best support reproducibility,” Keahey said. ​“And, not all the talks focused solely on Chameleon successes. The discussions made clear important reproducibility challenges that remain: improvements are needed in testing methodologies, hardware-agnostic energy profiling approaches for HPC workload reproducibility, and ways to reproduce experiments not only on Chameleon but also on other public cloud platforms.”

The next step is to produce a report capturing the community’s collective knowledge and recommendations for advancing practical reproducibility in HPC.

For further information:
The workshop was co-sponsored by another project called REPETO, led by Keahey, which is building an international network of researchers working on reproducibility. REPETO also funds undergraduate and graduate student summer projects facilitating reproducibility; see the current call for project proposals for the 2024 Summer of Reproducibility (due Feb. 22, 2024) at https://​repeto​.cs​.uchica​go​.edu/sor/.

For more information about practical reproducibility, see the paper ​“Three Pillars of Practical Reproducibility” by Keahey et al., in Proceedings of the 2023 IEEE 19^th International Conference on e-Science, 2023, pp. 1–6.