Research by computational scientists in Argonne’s Mathematics and Computer Science (MCS) division, presented virtually at the 29th International Meshing Roundtable in June 2021, includes the design of a new strategy to generate a spectral element mesh around 352,000 densely packed spherical pebbles. The mesh, which comprises 99 million elements, is used with the Argonne-based Nek5000/NekRS fluid-thermal simulation codes to compute the flow field. The results of the study indicate that reactor designers will be able to conduct parameter studies on exascale platforms.
The flow of coolant through densely packed beds is of particular interest in the design of new pebble-bed reactors. For simulations that resolve turbulent eddies in the flow, high-order methods such as the spectral element method can reduce by two to three orders of magnitude the number of elements required by standard finite element methods.
“The reduction in count implies that the meshes are relatively coarse-grained, placing stringent requirements on mesh quality and conformity,” said Misun Min, an MCS computational scientist.
To address these issues, the MCS team devised an approach for constructing high-quality all-hex meshes for the interstitial space in dense-packed spheres. The innovative approach is based on a decomposition of the domain into Voronoi cells surrounding each sphere. Facets of each cell are tessellated into quadrilaterals that are projected onto the sphere surface to generate a base mesh.
A key part of the research has focused on NekRS, a spectral element code being developed by Min’s team that is supported by the U.S. Department of Energy’s (DOE) Center for Efficient Exascale Discretizations (CEED) project. NekRS, a variant of the thermal-hydraulics code Nek5000, uses the power of graphics processing units (GPUs) to help DOE applications prepare for exascale architectures such as Aurora, anticipated to be deployed at Argonne in 2022.
“The tremendous size of these computations puts pressure on all aspects of the fluid-thermal simulation workflow,” Min said. Over the past year, the NekRS team has improved the pressure preconditioners and communication in NekRS, devising several strategies that have reduced thermal-fluid simulation times by a factor of four.
The researchers have successfully run Nek5000/NekRS on a broad range of mesh sizes, from 146 pebbles on workstations to 352,000-pebble simulations on the Summit supercomputer at Oak Ridge National Laboratory. The largest case corresponds to 99 million spectral elements and 50.5 billion gridpoints, as shown in the figure.
“Several steps are critical to automatic generation of production-quality meshes,” said Paul Fischer, a senior computational scientist in Argonne’s MCS division and a professor at the University of Illinois in Urbana-Champaign. For example, the software performs edge collapse to remove very thin facets, or slivers, that can lead to poorly conditioned elements. It also inserts vertices to balance the resolution of facets with edges that were longer than desired, and then smooths the all-hex mesh through a combination of optimization and boundary smoothing techniques. Argonne predoctoral researcher Yu-Hsiang Lan has been instrumental in implementing and scaling the mesh generation capabilities to the full-core configuration of 352,000 pebbles.
The various mesh configurations have provided the researchers with new insights into where regions of order and disorder occur in the spheres along the domain wall. “The packing order directly influences the flow conditions near the domain boundary and is an important consideration for thermal-hydraulics analysis,” Fischer said.
Particularly exciting is the fact that the largest case — which corresponds to a full reactor core — takes only 0.233 seconds per step when running on all 4,608 nodes (27,648 V100 GPUs) of Summit.
“This configuration would require only 6 hours to compute a single flow-through time on all of Summit,” Min said. “The results are especially significant because they imply that reactor designers will be able to conduct parameter studies on exascale platforms.”
For further information, see the paper by Y. Lan, P. Fischer, E. Merzari, and M. Min, “All-hex meshing strategies for densely packed spheres,” presented at the 29th International Meshing Roundtable in June 2021. A preprint is available at https://arxiv.org/abs/2106.00196.
NekRS development is described in this preprint: https://arxiv.org/abs/2104.05829.