Skip to content. | Skip to navigation

Personal tools
Log in
Sections
You are here: Home Utilisateurs Guides Running LMDZ in parallel on a Linux PC

Running LMDZ in parallel on a Linux PC

Some thoughts and advice for those who want to run LMDZ in parallel on their multicore Linux PCs.

Introduction

This page contains some advice for those who want to run LMDZ in parallel on their multicore Linux PC; i.e. in a "personal" configuration, as opposed to the case of machines in HPC centers (IDRIS, CCRT, etc.) where all the environments and tools are set (and implemented in the LMDZ deployment infrasctructure).

Before moving on to the specificities implied by parallelism, it is best to first install the mode on your machine, for instance with the 'install.sh' script (as explained here) and check that all is well in the sequential mode. In all that follows, it is assumed that the compiler that is used is gfortran and that code compilation will be done via the makelmd_fcm fcm script (note that fcm was first included in the LMDZ source bundle with revision 1578 of the code).

Compiling and running in MPI mode alone

Prerequisites: An MPI library (e.g. MPICH, OpenMPI, etc.) must be installed and available.

Assuming that 'mpif90' is the wrapper for compiling (via gfortran) in MPI, that the MPI library is installed in '/my/mpi/directory/lib', and that the corresponding files to include (mpi.h) are in '/my/mpi/directory/include', the appropriate 'arch-local.fcm' file would be:

%COMPILER            mpif90
%LINK                mpif90
%AR                  ar
%MAKE                make
%FPP_FLAGS           -P -traditional
%FPP_DEF             NC_DOUBLE
%BASE_FFLAGS         -fdefault-real-8
%PROD_FFLAGS         -O3 -funroll-loops
%DEV_FFLAGS          -g -O1 -Wall
%DEBUG_FFLAGS        -g -O0 -Wall -fbounds-check -ffpe-trap=invalid,zero,overflow
%MPI_FFLAGS          -fcray-pointer -I/my/mpi/directory/include
%OMP_FFLAGS         
%BASE_LD             
%MPI_LD -L/my/mpi/directory/lib -lmpi
%OMP_LD             

Compiling the GCM would be done with :

makelmdz_fcm -arch local -parallel mpi ......

And running, using N processes (N=4 in the example that follows) with:

mpiexec -np 4 gcm_32x24x11_phylmd_para.e

Note: When running in MPI, the IOIPSL output files from the physics are split (1 file per process) and these must be recombined using the 'rebuild' utility, as explained here.

Compiling and running in OpenMP mode alone

Assuming that 'gfortran' is the compiler you are using, the appropriate 'arch-local.fcm' file would be:

%COMPILER            gfortran
%LINK                gfortran
%AR                  ar
%MAKE                make
%FPP_FLAGS           -P -traditional
%FPP_DEF             NC_DOUBLE
%BASE_FFLAGS -fdefault-real-8
%PROD_FFLAGS         -O3 -funroll-loops
%DEV_FFLAGS          -g -O1 -Wall
%DEBUG_FFLAGS        -g -O0 -Wall -fbounds-check -ffpe-trap=invalid,zero,overflow
%MPI_FFLAGS          
%OMP_FFLAGS          -fcray-pointer -fopenmp
%BASE_LD             
%MPI_LD
%OMP_LD              -fopenmp

Compiling the GCM would be done with :

makelmdz_fcm -arch local -parallel omp ......

And running, using X OpenMP tasks (X=4 in the following example) with:

export OMP_NUM_THREADS=4
./gcm_48x36x19_phylmd_para.e

Notes: To avoid insufficient memory issues for the OpenMP tasks (where the latter store their private variables), it is recommended to have the maximum "stack memory" that is possible. In practice, using the command:

ulimit -s unlimited

(this line can be included in a .bashrc file to be systematically set with each session). You may also have to increase the private stack memory that is assigned to each OpenMP thread with

export OMP_STACKSIZE=200M

Compiling and running in mixted MPI/OpenMP mode

Everything mentioned above concerning the MPI and OpenMP cases combine.

An appropriate 'arch-local.fcm' file should be :

%COMPILER            mpif90
%LINK                mpif90
%AR                  ar
%MAKE                make
%FPP_FLAGS           -P -traditional
%FPP_DEF             NC_DOUBLE
%BASE_FFLAGS         -fdefault-real-8
%PROD_FFLAGS         -O3 -funroll-loops
%DEV_FFLAGS          -g -O1 -Wall
%DEBUG_FFLAGS        -g -O0 -Wall -fbounds-check -ffpe-trap=invalid,zero,overflow
%MPI_FFLAGS          -fcray-pointer -I/my/mpi/directory/include
%OMP_FFLAGS          -fopenmp
%BASE_LD             
%MPI_LD -L/my/mpi/directory/lib -lmpi
%OMP_LD              -fopenmp

Compiling the GCM would be done with:

makelmdz_fcm -arch local -parallel mpi_omp ......

And running, using N processes (N=4 in the following example) and X tasks each (X=2 in the following example):

export OMP_NUM_THREADS=2
mpiexec -np 4 gcm_32x24x11_phylmd_para.e

 

Last updated: 11 December 2014

Add comment

You can add a comment by filling out the form below. Plain text formatting. Comments are moderated.

Enter the word