Abstract: As numerical simulations advance toward exascale capabilities, the volume of generated data is growing at an unprecedented rate. This increasing gap between computational power and I/O capacity poses significant challenges [4]. Traditional post-hoc data analysis methods are becoming impractical. For example, in simulations like Gysela5D [1], each time step can produce over 100 TB of data. Analyzing such massive datasets necessitates loading them entirely into memory, demanding impractically large amounts of RAM on a compute node. Additionally, as simulations become larger and more detailed, identifying relevant areas within the data becomes a major challenge. Often, only a subspace of the complete dataset is of interest, but it can be obscured within the full data domain. This issue is exacerbated by the increasing number of dimensions in simulations.
We present our work in progress tool called SPOT (Simulation Pattern Observation Tool), a tool developed on top of PDI [3] and Deisa [2] that detects and localizes physical events in simulations. When SPOT identifies a physical event, it triggers adjustments to simulation parameters, such as I/O frequency. We illustrate the effectiveness of SPOT with a multisource 2D heat simulation, demonstrating its capability to identify areas of interest and selectively write subsets of data, effectively reducing the load on the file system.
References:
[1] Virginie Grandgirard et al. “A 5D gyrokinetic full-f global semi-lagrangian code for flux-driven ion turbulence simulations”. In: Computer Physics Communications 207 (2016), p. 35. doi: 10.1016/j.cpc.2016.05.007.
[2] Amal Gueroudji, Julien Bigot, and Bruno Raffin. “DEISA: Dask-Enabled In Situ Analytics”. In: 2021 IEEE 28th International Conference on High Performance Computing, Data, and Analytics (HiPC). 2021 IEEE 28th International Conference on High Performance Computing, Data, and Analytics (HiPC). ISSN: 2640-0316. Dec. 2021, pp. 11–20. doi: 10.1109/HiPC53243.2021.00015.
[3] PDI. url: https://pdi.dev.
[4] François Tessier, Venkatram Vishwanath, and Emmanuel Jeannot. “Adding topology and memory awareness in data aggregation algorithms”. In: Future Generation Computer Systems 159 (Oct. 1, 2024), pp. 188–203. issn: 0167-739X. doi: 10.1016/j.future.2024.05.016.
Bios:
Benoit Martin is a research scientist in computer science at Maison de la Simulation, a collaborative laboratory involving CEA, CNRS, Université Paris-Saclay, and Université Versailles Saint-Quentin. Benoit earned his PhD in distributed systems from Sorbonne Université in Paris, where he specialized in the actor programming model, data consistency models, and the safe management of shared memory between actors.
Julien Bigot is a permanent CEA computer scientist at Maison de la Simulation where he leads the Science of Computing team. Julien leads the CExA project that contributes to the Kokkos ecosystem. He also takes part in the NumPEx programme, that intends to contribute to the French Exascale software stack. In this programme, Julien co-leads the Exa-DoST project about IO and data analysis libraries and tools, in the Exa-DI project, he co-leads the work on software packaging, deployment, and CI/CD, and in the Exa-Soft project, Julien takes part in the work GPU programming models.