\documentclass[a4paper,12pt]{article}

                                % Packages

\usepackage[utf8]{inputenc}
\usepackage{ifpdf}

\ifpdf
\else
\usepackage[active]{srcltx}
\fi

\usepackage{amsmath}

\usepackage[T1]{fontenc}
\usepackage{lmodern}

\usepackage{geometry}

\usepackage[np]{numprint}

\usepackage{hyperref}
\ifpdf
\hypersetup{pdftitle={Tutorial \#2}, pdfauthor={LMDZ team},
  hypertexnames=false, pdfstartview=FitBH}
\fi

\author{LMDZ team}
\title{Hands on tutorial \#2: Installing LMDZ in paralel mode ; Setting up and runing a simulation}
\date{November 2022}


\geometry{top=2cm,bottom=2cm,left=2.5cm,right=2.5cm}


                                %---------------

\begin{document}

\maketitle
%\tableofcontents
%\medskip

This tutorial shows the steps for setting up a 3D simulation to be run in parallel mode, in particular for a
zoomed configuration of LMDZ.



%This tutorial makes use of a zoomed configuration of LMDZ. Section
%\ref{sec:setting-up-case} is an initialization phase. In section
%\ref{sec:experiments}, you can either take a closer look at tracers,
%or the Orchidee model or switch some parameterizations of the physics
%part of the program. The three experiments in section
%\ref{sec:experiments} are independent. In section \ref{sec:nudging},
%you will nudge the model toward analyzed wind fields.


\textbf{Installing the model to run in parallel mode}

Download and run the script \verb+install_lmdz_par.sh+ , prepared for you to install the model with the appropriate options, in a folder named LMDZpar : 

\begin{verbatim}
wget https://lmdz.lmd.jussieu.fr/pub/Training/2022FormationMaroc/install_lmdz_par.sh
chmod +x install_lmdz_par.sh
./install_lmdz_par.sh
\end{verbatim}

\textbf{Setting up a simulation with a (regular or) zoomed grid}
\label{sec:setting-up-case}

\begin{itemize}
\item Go to the directory \verb+LMDZpar/modipsl/modeles/LMDZ+ ,
  which contains the files \verb+makelmdz_fcm+, \verb+libf+ etc. 
  Download the \verb+tutorial.tar+ archive, unpack it, then go in the resulting \verb+TUTORIAL+ folder :
\begin{verbatim}
cd LMDZpar/modipsl/modeles/LMDZ
wget https://lmdz.lmd.jussieu.fr/pub/Training/2022FormationMaroc/tutorial.tar
tar -xf tutorial.tar
cd TUTORIAL
\end{verbatim}
\item Examine the content of the \verb+TUTORIAL+ folder : there are some scripts and a DEF directory, all briefly described in the \verb+Readme+ file.
  In the \verb+DEF+ directory, edit the file \verb+gcm.def+ and examine the 
  different parameters defining the grid. 
  
  By default, the defined grid has a zoom factor = 2 both in longitude and latitude 
  
  (\verb+grossismx=2., grossismy=2. +), 
  with the zoomed area centered at (0E, 45N) : \verb+clon=0., clat=45.+. 
  If you want to set the center of the zoom at a different location, just change \verb+clon+ and \verb+clat+.

	If you want to use a regular grid, set \verb+grossismx=1. and grossismy=1. + .

\item You will run LMDZ with the simple `bucket'' surface scheme: in the \verb+init.sh+ script, check that you have the option
  \verb+veget=0+.
  (Note : \verb+veget=1+ activates the coupling with the much more complex ORCHIDEE continental surface model).
\item As you installed the model in parallel mode, in \verb+init.sh+ you must also have \verb+parallel=1+.
%\item Skip this step if you run locally :
%		
%	You may increase the stack memory that you can use, by typing the following command:
%	
%\begin{verbatim}
%ulimit -Ss unlimited
%\end{verbatim}
%  or, even better, add this command in you shell start-up file (if you
%  use Bash, your start-up file can be \verb+.profile+ or
%  \verb+.bash_profile+).
  
  
\item Run the main script  \verb+init.sh+ :
\begin{verbatim}
./init.sh
\end{verbatim}

  The script \verb+init.sh+ does the following :
  
  	- (re)compiles the model (\verb+gcm.e+) with resolution 48x36x39 ;
  
  	- compiles the program \verb+ce0l.e+, needed to create initial state and boundary conditions for the chosen grid; 
  
  	- downloads input files for \verb+ce0l.e+ (NetCDF files containing surface orography, sea-surface temperature etc.)
  
  	- runs \verb+ce0l.e+ , which creates files \verb+start.nc+, \verb+startphy.nc+ and
  \verb+limit.nc+ in a directory called \verb+INITIAL+. These files will copied in the newly-created directories \verb+SIMU1+ and \verb+PROD0+ .

Please check that these 3 files have been created in the directories \verb+INITIAL+, \verb+SIMU1+ and \verb+PROD0+. If not, ask for our help.  

% AS : pour la formation au Maroc j ai deja rajoute le flag_pres... dans DEF/config.def
%----
%\item Now edit the file \verb+SIMU1/config.def+, and look at the section "Controle des sorties" (Eng. "Output control").
%In the high-frequency NetCDF output file \#3, \verb+histhf.nc+., you'll want to include the variable \verb+pres+, 
%containing the pressure at model levels.
%To do that, add the following line :
%\begin{verbatim}
%flag_pres__00003 =           4
%\end{verbatim}


\textbf{Running a short simulation (a few days of run)}

\item You can now go in the SIMU1 directory and run the model in parallel on 2x2 processors :
\verb+./run_local.sh 2 2 gcm.e  + . 

By default it will run for 1 day : \verb+nday=1+ in \verb+DEF/run.def+ . 

The simulation should end with the message ``Everything is cool'' and the 
output files \verb+histday.nc+, \verb+histmth.nc+ and \verb+histhf.nc+ should be created. 
You can start analyzing the output variables in these files. Note that your \verb+histmth.nc+ output file is "empty" ; you need at least 5 days of run (\verb+nday=5+) to get average values in this file.

\textbf{Running a long simulation (months or years) }
\item You'll keep PROD0 as a reference, and to start other simulations. Make a copy of PROD0 as MyPROD in which you'll run your simulation, then go in MyPROD : 

\begin{verbatim}
cp -pr PROD0 MyPROD
cd MyPROD
\end{verbatim}


\item You may want to look at the DEF/*.def files, and possibly adjust them, for ex the output frequency and content. NOTE : you should NOT change to the grid parameters in gcm.def because they must be on the same grid as the start* and limit.nc files you have already produced.

\item To run the simulation, you'll use the script \verb+enchaine.sh+.
By default, the date of beginning of the run is  1 jan 2000. You can see it in the file \verb+etat+ as follows : \verb+200001 afaire+ (that is : month 01 of year 2000, to do).

The model runs automatically month after month. Each individual month is executed in a temporary folder with name starting with "WD". 

The run stops at the end of the month before \verb+stopsim+ in \verb+enchaine.sh+. By default \verb+stopsim=200101+ , so the model stops at the end of december 2000.

\item You may want to start by running for only one month, to check if you have all the output you need. Set \verb+stopsim=200002+ in \verb+enchaine.sh+ (ligne 29), then launch :
\begin{verbatim}
./enchaine.sh &
\end{verbatim}

\item At the end of the month, you'll see in the file \verb+etat+  : \verb+200001 fait ; 200002 afaire+
To continue the simulation, you just change \verb+stopsim+ in \verb+enchaine.sh+ and launch again \verb+ ./enchaine.sh &+


\end{itemize}

\end{document}
