\documentclass{article}
\usepackage[english]{babel}
\usepackage[latin1]{inputenc}

\usepackage{a4wide}
\begin{document}

\def\afaire#1{{\bf Afaire : #1}}
\title{Tutorial: Using LMD with XIOS \date{\today}}
\author{The LMDZ team}

\maketitle

This tutorial focuses on setting up, compiling 
and running LMDZ with XIOS.\\

This document can be downloaded as a pdf file:
\begin{verbatim}
wget http://www.lmd.jussieu.fr/~lmdz/Distrib/TD_XIOS.pdf
\end{verbatim}
which should ease any copy/paste of command lines to issue.\\

%\def\infol#1{#1}
%\def\infol#1{\verb+ #1 +}
\def\infol#1{{\bf #1}}
%\newcommand{\infol}[1]{\verb\+ #1 \+}

\def\resol#1#2#3{#1$\times$#2-L#3} %


This tutorial is for users who want to learn the basic
steps needed to be able to run LMDZ with the XIOS input-output library
on their computer.
Note that this implies the prerequisite that you can run in parallel
on your machine (i.e. that you have already completed the tutorial
on running LMDZ in parallel).\\

\section{Installing the required HDF5 and NetCDF4 libraries}
The XIOS library relies on HDF5 and NetCDF4 libraries (which
also rely on curl and zlib libraries) that have to be
compiled with the same compiler than the MPI library that will be used.\\

\noindent
We have a script that combines all the download and install instructions,
which you can download using:
\begin{verbatim}
wget http://www.lmd.jussieu.fr/~lmdz/Distrib/install_netcdf4_hdf5_mpi.bash
\end{verbatim}
You should then make that script executable and run it
\begin{verbatim}
chmod u=rwx install_netcdf4_hdf5_mpi.bash
./install_netcdf4_hdf5_mpi.bash > install_netcdf4_hdf5_mpi.out 2>&1
\end{verbatim}
If all goes well, then all the required libraries will be generated under
\infol{\$HOME/local} (on the provided laptops this will be
\infol{/home/util1/local}). This default is set in the script as
\infol{INSTALL\_PATH} and can be changed; in all that follows it is assumed
that the libraries have been installed under \infol{\$HOME/local}


\section{Initial setup and recompiling the IOIPSL library}
Because of the change in used NetCDF libraries, it is recommended that
you work on a copy of the modipsl directory you have used so far.
Copy over the entire \infol{LMDZtrunk} directory and remove subdirectories
\infol{LMDZtrunk/netcdf-4.0.1}, as well as the \infol{ORCHIDEE} and
\infol{ORCHIDEE\_beton} from the \infol{modeles} subdirectory
(this is only to keep things simple in this tutorial and avoid having
to also recompile Orchidee).\\

\noindent
To rebuild the IOIPSL library, you must first go to the 
\infol{modipsl/util} directory and adapt the \infol{AA\_make.gdef} file
to point NetCDF library.
Replace lines
\begin{verbatim}
#-Q- g95     NCDF_INC= ...
#-Q- g95     NCDF_LIB= ...
\end{verbatim}
With (again we assume here the NetCDF library is installed in
\infol{/home/util1/local}; adapt accordingly if it is not the case):
\begin{verbatim}
#-Q- g95     NCDF_INC= /home/util1/local/include
#-Q- g95     NCDF_LIB= -L/home/util1/local/lib -lnetcdff
\end{verbatim}
And then issue the following command:
\begin{verbatim}
./ins_make -t g95 -p I4R8
\end{verbatim}
This will generate adequate Makefile files in 
\infol{IOIPSL/tools} and \infol{IOIPSL/src}.\\

\noindent
To recompile the IOISPL library, go to the \infol{IOIPSL/tools}
subdirectory and modify the \infol{Makefile} to set
the \infol{SHELL} variable to:
\begin{verbatim}
SHELL = /bin/bash
\end{verbatim}
Note that this is mandatory, unless the (default) ksh shell is
installed on your system.
And then build the library using the \infol{make} command
\begin{verbatim}
make
\end{verbatim}
This will compile the sources and put the generated library \infol{libioipsl.a}
in directory \infol{../../../lib}, i.e. in the \infol{lib} directory under
\infol{modipsl}. You have to also manualy
move all the generated module files to that same directory:
\begin{verbatim}
mv *.mod ../../../lib
\end{verbatim}


\section{Installing the XIOS library}
First you must download the XIOS library sources, using svn, in your
\infol{modipsl/modeles} directory:
\begin{verbatim}
svn co http://forge.ipsl.jussieu.fr/ioserver/svn/XIOS/branchs/xios-1.0 XIOS
\end{verbatim}

\noindent
Then go to the \infol{XIOS/arch} directory and create the following files:\\ 
\infol{arch-local.env}:
\begin{verbatim}
export MPI_LIB="-L/usr/lib/openmpi/lib -lmpi"
export NETCDF="$HOME/local"
export NETCDFLIB=$NETCDF/lib
export NETCDFINC=$NETCDF/include
\end{verbatim}
\infol{arch-local.path}:
\begin{verbatim}
NETCDF_INCDIR="-I${NETCDFINC} "
NETCDF_LIBDIR="-L${NETCDFLIB} "
NETCDF_LIB="-lnetcdf -lnetcdff -lnetcdf_c++4"
HDF5_INCDIR="-I${NETCDFINC} "
HDF5_LIBDIR="-L${NETCDFLIB} "
\end{verbatim}
\infol{arch-local.fcm}:
\begin{verbatim}
%CCOMPILER      mpicc
%FCOMPILER      mpif90
%LINKER         mpif90
%BASE_CFLAGS    -w
%PROD_CFLAGS    -O3 -D BOOST_DISABLE_ASSERTS
%DEV_CFLAGS     -g -O2
%DEBUG_CFLAGS   -g -DBZ_DEBUG
%BASE_FFLAGS    -D__NONE__
%PROD_FFLAGS    -O3
%DEV_FFLAGS     -g -O2
%DEBUG_FFLAGS   -g
%BASE_INC       -D__NONE__
%BASE_LD        -lstdc++
%CPP            mpicc -EP
%FPP            cpp -P
%MAKE           make
\end{verbatim}

\noindent
Once these three files have been created, you can build the XIOS library.
In directory \infol{XIOS}, run the \infol{make\_xios} script as follows:
\begin{verbatim}
./make_xios --debug --arch local --job 8 > make_xios.out 2>&1
\end{verbatim}
If all went well, the \infol{XIOS/lib} directory should contain the
XIOS library, \infol{libxios.a}, and the \infol{XIOS/bin} directory should
contain (among other test programs) the XIOS server \infol{xios\_server.exe}.

\section{Compiling LMDZ with XIOS}
Return to the \infol{LMDZ5} directory and add the following two
lines to \infol{arch/arch-para.path}:
\begin{verbatim}
XIOS_INCDIR=$LMDGCM/../XIOS/inc
XIOS_LIBDIR=$LMDGCM/../XIOS/lib
\end{verbatim}
You also have to adapt the \infol{NETCDF\_LIBDIR} and \infol{NETCDF\_INCDIR}
in \infol{arch/arch-para.path}:
\begin{verbatim}
NETCDF_LIBDIR="-L/home/util1/local/lib -lnetcdff -lnetcdf -lnetcdfc++4 -lhdf5hl_fortran
-l hdf5_hl -lhdf5_fortran -lhdf5 -lz -lstdc++"
NETCDF_INCDIR="-I/home/util1/local/include"
\end{verbatim}

\noindent
You can now compile LMDZ5 as usual, but adding the "-io xios" flag, e.g.
\begin{verbatim}
./makelmdz_fcm -arch para -mem -parallel mpi -io xios -d 32x32x39 -j 8 gcm
\end{verbatim}

\section{Running a first simulation with XIOS}
Make a new simulation directory (e.g. by copying over all input \infol{.def}
and \infol{.nc} files from the \infol{BENCH32x32x39}
directory\footnote{As in this example Orchidee is not used, the flag
\infol{VEGET} in \infol{config.def} should be set to \infol{n}}).
In addition you will need the input \infol{.xml} files to manage XIOS outputs.
Examples of these can be found in the \infol{LMDZ5/Deflists} directory.
Copy over files \infol{iodef.xml}, \infol{context\_lmdz.xml}, 
\infol{field\_def\_lmdz.xml} and \infol{file\_def\_hist*.xml}.\\

\noindent
For this first simulation, we will use XIOS in attached mode (i.e.
embedded in LMDZ), so the \infol{using\_server} variable in 
\infol{iodef.xml} must be set to \infol{false}.\\
Moreover, to enable outputs via XIOS in LMDZ, the following flag:
\begin{verbatim}
ok_all_xml = y
\end{verbatim}
must be set in the \infol{run.def} file (or equivalently in the 
\infol{config.def}.\\

\noindent
As can be seen in the \infol{context\_lmdz.xml} file, many
predefined output files (mimicking what is done via the output.def file when
using IOIPSL) are defined. By default none are enabled.
Modify file \infol{file\_def\_histday\_lmdz.xml} by setting
\begin{verbatim}
<file id="histday" name="histday" output_freq="1d" output_level="2"
type="one_file" enabled=".true.">
\end{verbatim}
so that the \infol{histday.nc} file will be generated (as a single file
over the entire domain) when the model is run.\\

\noindent
The model can now be run. Note that because the NetCDF libraries are
dynamically loaded at run time, the path to these must be included in
the \infol{LD\_LIBRARY\_PATH} environment variable\footnote{Rather than having
to have to add the path to \infol{LD\_LIBRARY\_PATH} in each new shell before
running the model, it makes sense to add that command in your
\infol{\$HOME/.bashrc} file so that it is always automatically done.}:
\begin{verbatim}
export LD_LIBRARY_PATH=\home\util1\local\lib:$LD_LIBRARY_PATH
\end{verbatim}
Then run the model "as usual", e.g. in MPI mode using 4 processes:
\begin{verbatim}
mpirun -np 4 gcm_32x32x39_phylmd_para_mem.e > listing 2>&1
\end{verbatim}
And check the contents of the generated \infol{histday.nc} file.

\section{Defining the output domain}
One can ouput only a selected subset of the global domain by specifying
the appropriate \infol{domain} attributes in the
\infol{context\_lmdz.xml} file. For example to ontput a 2x3 subomain starting
at grid indexes $i=14$,$j=5$:
\begin{verbatim}
<domain id="dom_glo" data_dim="2" zoom_ni="2" zoom_ibegin="14"
zoom_nj="3" zoom_jbegin="5" />
\end{verbatim}

\noindent
To test implementing this setup, let's assume you want to output at only
one grid point, corresponding to Paris (longitude 48N ,latitude 2E)
to compare model ouput to station records.\\

\noindent
The first thing to do is to identify the global grid coordinates that will
have to be specified in the \infol{domain} attributes. This can be done
by inspecting the \infol{lat} and \infol{lon} values in the
\infol{histday.nc} file from the previous run, either via your favorite
visualization software, or simply using the \infol{ncdump} utility:
\begin{verbatim}
ncdump -ff -v histday.nc
\end{verbatim}
And adapt the \infol{context\_lmdz.xml} file accordingly.

\noindent
Since we are interested in instantaneous values, it makes sense to
enable the \infol{histins} file, so adapt the
\infol{file\_def\_histins\_lmdz.xml} so that an \infol{histins\_Paris.nc}
file (i.e.: add the \infol{enabled=.true.} and
\infol{name="histins\_Paris.nc"} in the \infol{file} attributes)
will be generated.

\noindent
Also adapt general \infol{output\_level} and/or individual \infol{level} in
\infol{file\_def\_histins\_lmdz.xml} so that \infol{histins\_Paris.nc}
will contain \infol{t2m} (temperature at 2m), \infol{precip} (precipitation
rates), \infol{psol} (surface pressure) and \infol{temp} (temperature profile).

\noindent
Run the model and check the produced files.

\section{Running in client-server mode}
When running on a small number of cores, it is advised to use XIOS in
"attached" mode. In multicore environments (i.e. more than 32) it can be
more efficient to run in client-server mode and dedicate some cores to the
XIOS server.\\

\noindent
To test this setup, make a new directory where  to
run and copy over input files from previous simulation.
Start by copying over the XIOS server
\infol{xios\_server.exe} from \infol{XIOS/bin}. Then adapt the
\infol{iodef.xml} to switch to client-server mode
by setting the \infol{using\_server} variable to \infol{true}.
The executables may now be run, where the number of processes
allocated to each is set via the \infol{mpirun} command,
for instance to run LMDZ on 3 processes and XIOS on 1:
\begin{verbatim}
mpirun -np 3 gcm_32x32x39_phylmd_para_mem.e > listing 2>&1 : -np 1 xios_server.exe
\end{verbatim}
And check that you get the same output files as before.

\end{document}
