\documentclass[a4paper]{article}

                                % Packages

\usepackage[utf8]{inputenc}
\usepackage{ifpdf}

\ifpdf
\else
\usepackage[active]{srcltx}
\fi

\usepackage{amsmath}

\usepackage[T1]{fontenc}
\usepackage{lmodern}

\usepackage{geometry}

\usepackage[np]{numprint}

\usepackage{hyperref}
\ifpdf
\hypersetup{pdftitle={Tutorial Parallelism}, pdfauthor={LMDZ team},
  hypertexnames=false, pdfstartview=FitBH}
\fi

\author{LMDZ team}
\title{LMDZ tutorial: Parallelism}
\date{November 22nd, 2021}


\geometry{top=2cm,bottom=2cm,left=2.5cm,right=2.5cm}

\begin{document}

\maketitle

This tutorial focuses on setting up, compiling 
and running LMDZ in a parallel environment.\\

This document can be downloaded as a pdf file:
\begin{verbatim}
wget http://lmdz.lmd.jussieu.fr/pub/Training/Tutorials/Tutorial_Parallel.pdf
\end{verbatim}
which should ease any copy/paste of command lines to issue.\\

%\def\infol#1{#1}
%\def\infol#1{\verb+ #1 +}
\def\infol#1{{\bf #1}}
%\newcommand{\infol}[1]{\verb\+ #1 \+}

\def\resol#1#2#3{#1$\times$#2-L#3} %


This tutorial is for users who want to learn the basic
steps needed to be able to run parallel versions of LMDZ
on their computer. Note that this implies the
prerequisite that there is
a working MPI library
already installed on the machine, which is the case for the
laptops provided for this training session (on which the MPICH library
is installed; the OpenMPI library is also know to work).\\


Throughout this tutorial we will assume that you are working on the
provided laptops, for which an MPI library and utilities
(using the gfortran compiler)
are installed in the following directories\footnote{
If running on your own machines, you will of course have to adapt MPI
paths to point to the appropriate location. There are various available
MPI libraries downloadable from the Internet, such as OpenMPI or MPICH.
What is important to know is that the MPI library must have been compiled
with the same compiler than the one used to compile LMDZ.
}:
\begin{verbatim}
/usr/bin /usr/lib /usr/lib/openmpi 
\end{verbatim}
Depending on your environment variables and
settings\footnote{Ideally setting stacksize and \infol{LD\_LIBRARY\_PATH}
should be set in your \~/.bashrc to avoid having issue these commands
in each terminal and in every session.}, it is recommended
(often mandatory!) to set available stacksize (which is roughly the
amount of memory your program is allowed to request from the system)
to maximum and add the MPI library path to \infol{LD\_LIBRARY\_PATH}
 (this variable
tells programs where to look for needed dynamical libraries at run time):
\begin{verbatim}
ulimit -s unlimited
export LD_LIBRARY_PATH=/where/your/mpi/lib/is/lib:$LD_LIBRARY_PATH
\end{verbatim}

Note that for the provided laptops there is no need to modify
\infol{LD\_LIBRARY\_PATH} since the library is in a common standard location.

\section{Running the install\_lmdz.sh script}

The \infol{install\_lmdz.sh} script can do the necessary to install the model; all that is required is to use the \infol{-parallel} option, which may be set to \infol{none} (default), \infol{mpi} to compile and run in MPI only, \infol{omp} to compile and run in OpenMP only, or \infol{mpi\_omp} to compile and run the hybrid (mixed MPI and Openmp) parallel version of LMDZ.\newline

Create a separate directory to install the model (assuming you want to keep previous work done with the serial version) and run\footnote{An alternative to working in a different directory than previously would be to use the
\infol{-name} option of \infol{install\_lmdz.sh} to download, install and run in a different target directory than the default \infol{LMDZtrunk}}:
\begin{verbatim}
wget http://www.lmd.jussieu.fr/~lmdz/pub/install_lmdz.sh
chmod +x install_lmdz.sh
./install_lmdz.sh -parallel mpi_omp -d 32x32x39 -v 20211102.trunk
\end{verbatim}

Just as in tutorial \#1, the script will end after a short run of the \resol{32}{32}{39} test case has run in parallel. %You should check that you obtained the same output (\infol{restart.nc}, \infol{histday.nc}, etc.) as with the original serial install.

\section{About the arch files}

In order to enable MPI and OpenMP, one has to set the corresponding options in
arch files prior to compilation. In practice, this means one must create (the \infol{install\_lmdz.sh} script takes care of it) an
\infol{arch-local.fcm} files (in the \infol{arch}
subdirectory of the \infol{LMDZ} directory).\\
This \infol{arch-local.fcm}
should be something like\footnote{
Check out the files in the \infol{arch} directory for examples relative to
other compilers; e.g. ifort in the Ada arch files.
}:
\begin{verbatim}
%COMPILER            /usr/bin/mpif90
%LINK                /usr/bin/mpif90
%AR                  ar
%ARFLAGS             rU
l%MAKE                make
%FPP_FLAGS           -P -traditional
%FPP_DEF             NC_DOUBLE 
%BASE_FFLAGS         -cpp -ffree-line-length-0 -fdefault-real-8 -DNC_DOUBLE
%PROD_FFLAGS         -O3
%DEV_FFLAGS          -Wall -fbounds-check
%DEBUG_FFLAGS        -g3 -Wall -fbounds-check -ffpe-trap=invalid,zero,overflow -O0
%MPI_FFLAGS          -fcray-pointer
%OMP_FFLAGS          -fopenmp  -fcray-pointer
%BASE_LD             -Wl,-rpath:/usr/lib
%MPI_LD              
%OMP_LD              -fopenmp
\end{verbatim}
Where additions mainly concern the \infol{COMPILER}, \infol{LINK}, 
\infol{MPI\_FFLAGS} and \infol{MPI\_LD} lines for MPI, and the
\infol{OMP\_FFLAGS} and \infol{OMP\_LD} lines for OpenMP.\newline


Having correctly set an arch file, one can then compile and run in the various parallel modes, or even just in serial. In practice, it is convenient to put these instructions in scripts (see for instance the \infol{compile.sh} in the \infol{LMDZ} directory and the \infol{bench.sh} and \infol{bench\_parallel.sh} scripts in the \infol{BENCH32x32x39}), but in what follows the minimal necessary sequence of commands are given and explained one at a time.

\section{Compiling and running LMDZ in MPI}

In the \infol{LMDZ} directory,
compile the model in MPI mode:
\begin{verbatim}
./makelmdz_fcm -arch local -parallel mpi -d 32x32x39 -j 8 -rad rrtm gcm
\end{verbatim}
The executable, \infol{gcm\_32x32x39\_para\_mem.e}, is generated in the
\infol{bin} subdirectory.\\


Once the model has been successfully recompiled, run a simulation.
To do so, create a new subdirectory, e.g. \infol{BENCH32x32x39\_mpi},
in \infol{LMDZ} and copy boundary
conditions, initial conditions and parameter files
(\infol{limit.nc, star*nc, *.def}) over from
directory \infol{BENCH32x32x39}, along with the newly created excutable.\\


To run, you will need to use the \infol{mpirun} utility (from the MPI library)
and specify the number of processes to run on (using the -np option), e.g. 4:
\begin{verbatim}
mpirun -np 4 ./gcm_32x32x39_phylmd_para_mem.e > listing 2>&1
\end{verbatim}
The run should be much shorter than if using only one process (verify this!).\\


When running in parallel, each MPI process creates its own hist* files.
You will thus obtain for instance a files histday.0001.nc, histday.0001.nc,
histday.0002.nc and histday.0003.nc which contains data relative to the
sets of atmospheric columns managed by each process.
To combine the output files from different processes into a single file
containing the full dataset, use the IOIPSL \infol{rebuild} script.
The \infol{rebuild} script is generated when installing IOIPSL
When installing IOIPSL, a  modipsl directory was created and the rebuild script
installed in \infol{modipsl/bin}\footnote{Adding this directory to your PATH,
e.g. in your {\tt\~{}/.bashrc} to avoid having to type the full path each time you want to use rebuild is advised}.\\
For each type of file, histday, histmth, etc., 
simply issue a command of the likes of\footnote{This is what the provided \infol{reb.sh} script does. You nedd only run it and specify the file to rebuild as an argument; e.g. \infol{reb.sh histday}} :
\begin{verbatim}
rebuild -o histday.nc histday_000*
\end{verbatim}
to generate the recombined output file.\\


You can check that the output files thus generated are identical to the ones
generated by the \infol{BENCH32x32x39} run. Likewise for the \infol{restart.nc} and
\infol{restartfi.nc} files. Note also that the \infol{listing} file for the
MPI run is larger than in a serial case, as most output
messages are redundantly written by all processes.



\section{Compiling and running LMDZ in OpenMP}

Compile LMDZ in OpenMP mode:
\begin{verbatim}
./makelmdz_fcm -arch local -parallel omp -d 32x32x39 -j 8 -rad rrtm gcm
\end{verbatim}
The executable, \infol{gcm\_32x32x39\_para\_mem.e}, is generated in the
\infol{bin} subdirectory.\\


Once the model has been successfully recompiled, run a new simulation in a
new subdirectory of \infol{LMDZ}, e.g. \infol{BENCH32x32x39\_omp} and copy
over boundary conditions etc. from \infol{BENCH32x32x39}, along with
the newly created excutable.\\


Before launching the run, some OpenMP environment variables must be set to
specify the (maximum)
 amount of memory privately allocated to each thread, and the number of
OpenMP threads to run with, e.g. 4:\\
\begin{verbatim}
export OMP_STACKSIZE=200M
export OMP_NUM_THREADS=4
./gcm_32x32x39_phylmd_para_mem.e > listing 2>&1
\end{verbatim}


Note that this time the generated hist*nc files are unique, but suffixed
0000.nc. Again you can check that the model outputs are the same than those
obtained in serial and pure MPI runs.


\section{Compiling and running LMDZ in mixed MPI/OpenMP}

Compile LMDZ in mixed MPI/OpenMP mode:
\begin{verbatim}
./makelmdz_fcm -arch local -parallel mpi_omp -d 32x32x39 -j 8 -rad rrtm gcm
\end{verbatim}


Again, create a subdirectory in which to run the model. All that was mentionned
in the previous sections on pure MPI and OpenMP runs combine when using the
mixed mode. So assuming you want to run using 3 MPI processes, each using
2 OpenMP threads\footnote{Note that you can request using more cores than 
available on a given machine. This is of course extremely inefficient and
one should strive to use at most the total number of available cores.}:
\begin{verbatim}
export OMP_STACKSIZE=200M
export OMP_NUM_THREADS=2
mpirun -np 3 ./gcm_32x32x39_phylmd_para_mem.e > listing 2>&1
\end{verbatim}


Since you have used 3 MPI processes, output hist files will be split in 3 
(i.e. histday\_0000.nc, histday\_0001.nc and histday\_0002.nc)
and should be recombined using the \infol{rebuild} tool. Once again, results
should match those obtained with the serial, pure MPI, and pure OpenMP runs.

\section{Exercice: Adjusting and setting the workload between MPI tasks}
In the directories where you did your MPI (or mixed MPI/OpenMP) run, you'll
find a \infol{bands} file, e.g. \infol{Bands\_32x32x39\_4prc.dat}, which 
contains information on how many columns where handled by each MPI
process\footnote{The \infol{bands} file contains information
on how many columns are handled by each MPI process, but also for each of the
four main "code steps": dynamics, tracer advection,
dissipation, and physics. A \infol{bands} file for N processes thus contains
4N lines. Each line contains two elements: the process number and number
of atmospheric columns assigned to it.}.\\


The default behaviour, for LMDZ, is to load and follow the
instructions of the bands file present in the
directory where it runs. If it cannot find such a file, it then automaticaly
generates one (which was the case in the test runs you've done so far) which
simply considers splitting evenly the work between all available tasks.
But this is rarely optimal.\\


There is an automated way of (iteratively) adjusting the workshare in LMDZ
which can be triggered by setting the \infol{adjust} parameter in file
\infol{run.def} to \infol{y}.\\
{\bf Important}: The \infol{adjust} option should only be used in pure MPI
mode, and is intended to be used to tune the bands file, and not to be
used for production runs. Once a suitable \infol{bands} file is obtained
(which typically requires a month, i.e. 30 days long run),
one should revert the \infol{adjust} option to \infol{n} and run with the
resulting \infol{bands} file.\\



Set up you experiment in a new directory. Import files (and MPI executable)
there and set \infol{adjust=y} in the run.def file, as well as
\infol{nday=30}. Run using 4 processes:
\begin{verbatim}
mpirun -np 4 ./gcm_32x32x39_phylmd_para_mem.e > listing 2>&1
\end{verbatim}
Check out the produced \infol{Bands\_32x32x39\_4prc.dat} file and compare
to the more na\"{\i}ve one that was previously generated.\\
To evaluate if this \infol{bands}
file is "converged", copy it aside for future reference and re-run LMDZ.



\end{document}
