Running LMDZ in parallel on a Linux PC
Introduction
This page contains some advice for those who want to run LMDZ in parallel on their multicore Linux PC; i.e. in a "personal" configuration, as opposed to the case of machines in HPC centers (IDRIS, CCRT, etc.) where all the environments and tools are set (and implemented in the LMDZ deployment infrasctructure).
Before moving on to the specificities implied by parallelism, it is best to first install the mode on your machine, for instance with the 'install.sh' script (as explained here) and check that all is well in the sequential mode. In all that follows, it is assumed that the compiler that is used is gfortran and that code compilation will be done via the makelmd_fcm fcm script (note that fcm was first included in the LMDZ source bundle with revision 1578 of the code).
Compiling and running in MPI mode alone
Prerequisites: An MPI library (e.g. MPICH, OpenMPI, etc.) must be installed and available.
Assuming that 'mpif90' is the wrapper for compiling (via gfortran) in MPI, that the MPI library is installed in '/my/mpi/directory/lib', and that the corresponding files to include (mpi.h) are in '/my/mpi/directory/include', the appropriate 'arch-local.fcm' file would be:
%COMPILER mpif90
%LINK mpif90
%AR ar
%MAKE make
%FPP_FLAGS -P -traditional
%FPP_DEF NC_DOUBLE
%BASE_FFLAGS -fdefault-real-8
%PROD_FFLAGS -O3 -funroll-loops
%DEV_FFLAGS -g -O1 -Wall
%DEBUG_FFLAGS -g -O0 -Wall -fbounds-check -ffpe-trap=invalid,zero,overflow
%MPI_FFLAGS -fcray-pointer -I/my/mpi/directory/include
%OMP_FFLAGS
%BASE_LD
%MPI_LD -L/my/mpi/directory/lib -lmpi
%OMP_LD
Compiling the GCM would be done with :
makelmdz_fcm -arch local -parallel mpi ......
And running, using N processes (N=4 in the example that follows) with:
mpiexec -np 4 gcm_32x24x11_phylmd_para.e
Note: When running in MPI, the IOIPSL output files from the physics are split (1 file per process) and these must be recombined using the 'rebuild' utility, as explained here.
Compiling and running in OpenMP mode alone
Assuming that 'gfortran' is the compiler you are using, the appropriate 'arch-local.fcm' file would be:
%COMPILER gfortran
%LINK gfortran
%AR ar
%MAKE make
%FPP_FLAGS -P -traditional
%FPP_DEF NC_DOUBLE
%BASE_FFLAGS -fdefault-real-8
%PROD_FFLAGS -O3 -funroll-loops
%DEV_FFLAGS -g -O1 -Wall
%DEBUG_FFLAGS -g -O0 -Wall -fbounds-check -ffpe-trap=invalid,zero,overflow
%MPI_FFLAGS
%OMP_FFLAGS -fcray-pointer -fopenmp
%BASE_LD
%MPI_LD
%OMP_LD -fopenmp
Compiling the GCM would be done with :
makelmdz_fcm -arch local -parallel omp ......
And running, using X OpenMP tasks (X=4 in the following example) with:
export OMP_NUM_THREADS=4
./gcm_48x36x19_phylmd_para.e
Notes: To avoid insufficient memory issues for the OpenMP tasks (where the latter store their private variables), it is recommended to have the maximum "stack memory" that is possible. In practice, using the command:
ulimit -s unlimited
(this line can be included in a .bashrc file to be systematically set with each session). You may also have to increase the private stack memory that is assigned to each OpenMP thread with
export OMP_STACKSIZE=200M
Compiling and running in mixted MPI/OpenMP mode
Everything mentioned above concerning the MPI and OpenMP cases combine.
An appropriate 'arch-local.fcm' file should be :
%COMPILER mpif90
%LINK mpif90
%AR ar
%MAKE make
%FPP_FLAGS -P -traditional
%FPP_DEF NC_DOUBLE
%BASE_FFLAGS -fdefault-real-8
%PROD_FFLAGS -O3 -funroll-loops
%DEV_FFLAGS -g -O1 -Wall
%DEBUG_FFLAGS -g -O0 -Wall -fbounds-check -ffpe-trap=invalid,zero,overflow
%MPI_FFLAGS -fcray-pointer -I/my/mpi/directory/include
%OMP_FFLAGS -fopenmp
%BASE_LD
%MPI_LD -L/my/mpi/directory/lib -lmpi
%OMP_LD -fopenmp
Compiling the GCM would be done with:
makelmdz_fcm -arch local -parallel mpi_omp ......
And running, using N processes (N=4 in the following example) and X tasks each (X=2 in the following example):
export OMP_NUM_THREADS=2
mpiexec -np 4 gcm_32x24x11_phylmd_para.e
Last updated: 11 December 2014