Advanced parallel implementation of the coupled ocean–ice model FEMAO (version 2.0) with load balancing

• Published in: Geoscientific Model Development
• Main Authors: Perezhogin, Pavel, Chernov, Ilya, Iakovlev, Nikolay
• Format: Text
• Language: English
• Published: 2021
• Subjects: Arctic; Arctic Ocean; White Sea; Sea ice
• Online Access: https://doi.org/10.5194/gmd-14-843-2021; https://gmd.copernicus.org/articles/14/843/2021/

In this paper, we present a parallel version of the finite-element model of the Arctic Ocean (FEMAO) configured for the White Sea and based on MPI technology. This model consists of two main parts: an ocean dynamics model and a surface ice dynamics model. These parts are very different in terms of the number of computations because the complexity of the ocean part depends on the bottom depth, while that of the sea-ice component does not. In the first step, we decided to locate both submodels on the same CPU cores with a common horizontal partition of the computational domain. The model domain is divided into small blocks, which are distributed over the CPU cores using Hilbert-curve balancing. Partitioning of the model domain is static (i.e., computed during the initialization stage). There are three baseline options: a single block per core, balancing of 2D computations, and balancing of 3D computations. After showing parallel acceleration for particular ocean and ice procedures, we construct the common partition, which minimizes joint imbalance in both submodels. Our novelty is using arrays shared by all blocks that belong to a CPU core instead of allocating separate arrays for each block, as is usually done. Computations on a CPU core are restricted by the masks of non-land grid nodes and block–core correspondence. This approach allows us to implement parallel computations into the model that are as simple as when the usual decomposition to squares is used, though with advances in load balancing. We provide parallel acceleration of up to 996 cores for the model with a resolution of <math xmlns="http://www.w3.org/1998/Math/MathML" id="M1" display="inline" overflow="scroll" dspmath="mathml"><mrow><mn mathvariant="normal">500</mn><mo>×</mo><mn mathvariant="normal">500</mn><mo>×</mo><mn mathvariant="normal">39</mn></mrow></math> <svg:svg xmlns:svg="http://www.w3.org/2000/svg" width="73pt" height="10pt" class="svg-formula" dspmath="mathimg" md5hash="8b73d4157dc8cae869a10d58c2b05ea5"><svg:image xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="gmd-14-843-2021-ie00001.svg" width="73pt" height="10pt" src="gmd-14-843-2021-ie00001.png"/></svg:svg> in the ocean component and 43 sea-ice scalars, and we carry out a detailed analysis of different partitions on the model runtime.