HPC systems¶

The information provided in this page is for users who intend to work on High-Performance Computing (HPC) systems, and want to use the Python 2.7 version of X-PSI. These installation instructions are system-specific. X-PSI has already been used on different systems, for some of which, we provide the instructions below. This information may also be translated to other systems by users looking for guidance.

Snellius (SURF)¶

Snellius is the Dutch National Supercomputer.

Installation¶

All of the following must be performed on a login node, in your $HOME file system.

Start by cleaning your home file system of existing versions of dependencies and move anything else to some archive in $HOME. Clean .bashrc and .bash_profile of anything related to this software. Clean .bashrc and .bash_profile of environment variables such as: LD_LIBRARY_PATH, LD_PRELOAD, RUN_PATH, PATH, and PYTHONPATH. Then logout and log back in order to get a clean environment.

To be additionally safe, run:

module purge

Load environment module and modify clean environment with Intel toolchain information:

module load 2021
module load intel/2021a

Prepare python environment:

module load Python/2.7.18-GCCcore-10.3.0-bare

We will now install the various python packages we require. We use the module /sw/arch/Centos8/EB_production/2021/modulefiles/lang/Python/2.7.18-GCCcore-10.3.0-bare.lua and its pip package manager to install packages locally in $HOME/.local/lib/python2.7/site-packages/ if they are not installed globally or are outdated.

Note that the python packages must be installed before pointing to intel compilers on Snellius to avoid child process reliability issues if using threads. matplotlib can be optionally installed for plotting purposes and is not necessary for modelling and inferencing.

pip install --user Cython==0.29.24
pip install --user wrapt

pip install --user scipy
pip install --user numpy
pip install --user matplotlib
pip install --upgrade --user setuptools

Set PYTHONPATH:

export PYTHONPATH=$HOME/.local/lib/python2.7/site-packages/:$PYTHONPATH

Point to Intel compilers:

export FC=ifort
export CC=icc
export CXX=icpc

Below we explicitly specify flag arguments to select intel instruction sets that are compatible with the AMD processors present in Snellius.

Load cmake module:

module load CMake/3.20.1-GCCcore-10.3.0

To prepare MPI from $HOME:

wget https://bitbucket.org/mpi4py/mpi4py/downloads/mpi4py-3.0.0.tar.gz
tar -xvf mpi4py-3.0.0.tar.gz
cd mpi4py-3.0.0
python setup.py install --user

To test on the login node:

mpiexec -n 8 python demo/helloworld.py

Do you see ranks 0 through 7 reporting for duty?

Note

If faced with an error message Can't locate Switch.pm ..., the Switch module can be loaded using the CPAN package manager as:

cpan

Follow the default instructions for the interactive cpan questions.

cpan[1]> install Switch
cpan[2]> exit

Note

If MPI raises a warning about missing hydra process manager, run the following code-block:

unset I_MPI_PMI_LIBRARY
export I_MPI_JOB_RESPECT_PROCESS_PLACEMENT=0

To prepare MultiNest from $HOME:

git clone https://github.com/farhanferoz/MultiNest.git ~/multinest
cd ~/multinest/MultiNest_v3.12_CMake/multinest
mkdir build
cd build
cmake -DCMAKE_{C,CXX}_FLAGS="-O3 -xAVX -axCORE-AVX2 -funroll-loops" -DCMAKE_Fortran_FLAGS="-O3 -xAVX -axCORE-AVX2 -funroll-loops" ..; make
ls ../lib/

Use the last command to check for the presence of shared objects.

Note

The Intel compilers on Snellius may run into issues with Intel Math Kernel Library (MKL) due to static linkage. These issues can be solved by setting the appropriate paths to the environment variable for the pre-load libs:

export LD_PRELOAD=/sw/arch/Centos8/EB_production/2021/software/imkl/2021.2.0-iimpi-2021a/mkl/2021.2.0/lib/intel64/libmkl_def.so.1:/sw/arch/Centos8/EB_production/2021/software/imkl/2021.2.0-iimpi-2021a/mkl/2021.2.0/lib/intel64/libmkl_avx2.so.1:/sw/arch/Centos8/EB_production/2021/software/imkl/2021.2.0-iimpi-2021a/mkl/2021.2.0/lib/intel64/libmkl_core.so:/sw/arch/Centos8/EB_production/2021/software/imkl/2021.2.0-iimpi-2021a/mkl/2021.2.0/lib/intel64/libmkl_intel_lp64.so:/sw/arch/Centos8/EB_production/2021/software/imkl/2021.2.0-iimpi-2021a/mkl/2021.2.0/lib/intel64/libmkl_intel_thread.so:/sw/arch/Centos8/EB_production/2021/software/imkl/2021.2.0-iimpi-2021a/compiler/2021.2.0/linux/compiler/lib/intel64_lin/libiomp5.so

Further details on MKL issues can be found in this thread

We also need to set the environment variable for the library path to point at MultiNest:

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HOME/multinest/MultiNest_v3.12_CMake/multinest/lib/

Now you need the Python interface to MultiNest, starting from $HOME:

git clone https://github.com/JohannesBuchner/PyMultiNest.git ~/pymultinest
cd ~/pymultinest
python setup.py install --user

If want to make sure that PyMultiNest is compatible with Python 2.7 version of X-PSI, you can e.g., checkout the following PyMultiNest commit: git checkout c8eba95, before running the setup.py file. To test the installation of MultiNest and PyMultiNest on the login node:

mpiexec -n 2 python pymultinest_demo.py

Do you obtain parameter values and evidences?

Note

We assumed above that nested sampling with MultiNest is desired. If ensemble-MCMC with emcee is desired, you need to install the Python packages emcee and schwimmbad. We assume the user can infer how to do this using the information above and on the Installation page, or using the instructions provided for the CALMIP instructions below.

Next, we need to load GSL and set the PATH environment variable:

module load GSL/2.7-GCC-10.3.0
export PATH=/sw/arch/Centos8/EB_production/2021/software/GSL/2.7-GCC-10.3.0:$PATH

To prepare X-PSI from $HOME:

git clone https://github.com/xpsi-group/xpsi.git
cd xpsi
git checkout python2
LDSHARED="icc -shared" CC=icc python setup.py install --user

This ensures that both the compiler and linker are Intel, otherwise gcc linker would be invoked. Provided the GSL <prefix>/bin is in your PATH environment variable, the X-PSI setup.py script will automatically use the gsl-config executable to link the shared libraries and give the required cflags for compilation of the X-PSI extensions. Because the library location will not change for runtime, we state the runtime linking instructions at compilation in the setup.py script.

Note

Since Snellius uses AMD processors and the Intel instruction sets are internally translated, the installation proceeds while repeating automatic CPU dispatch and icc warnings. These warnings are safe to ignore. However, as they get printed, it takes longer for the installation and can exceed the idle time on the login node, resulting in a broken pipe. In this case, it would be preferable to direct the output of the installation into an output file, and if required use a nohup or similar command.

If you ever need to reinstall, first clean to recompile C files:

rm -r build dist *egg* xpsi/*/*.c

Note

We typically do not used the PostProcessing module, but instead rsync output files to a local system to perform plotting. This circumvents any potential backend problems and permits straightforward use of IPython for interactive plotting. However, if one wishes to use it on a HPC, it would require installation of GetDist and Nestcheck. See Installation page for relevant details.

Environment variables¶

The following environment variables need to be exported in your job script script so that all relevant libraries can be located at runtime by the dynamic loader (ensure that the environment variables are only extended, and not overwritten because module loading modifies these variables).

Set runtime linking path for MultiNest:

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HOME/multinest/MultiNest_v3.12_CMake/multinest/lib/

We want to ensure that your locally installed Python packages take precedence over globally installed packages:

export PYTHONPATH=$HOME/.local/lib/python2.7/site-packages/:$PYTHONPATH

If you are to perform small tests on login nodes in your login shell, these environment variables need to be exported in your .bash_profile script, or in your .bash.rc script which can be sourced by your .bash_profile script (the default default behaviour).

The /sara/sw/python-2.7.9/ Python distribution does not seem to have numpy linked against the Intel MKL library. Instead it uses the open-source, multithreaded OpenBLAS library which still offers an optimised interface to BLAS and LAPACK. However for our purposes on distributed memory architectures, we wish to export the following environment variables in our batch job script if we do not want multithreaded libraries to spawn worker (OpenMP or POSIX) threads:

export OMP_NUM_THREADS=1
export GOTO_NUM_THREADS=1
export OPENBLAS_NUM_THREADS=1
export MKL_NUM_THREADS=1

If we instruct our likelihood evaluation object to OpenMP multithread, local multithreading regions are used which do not take instructions from the OMP_NUM_THREADS environment variable, so we can invariantly export it as above. However, the MKL_NUM_THREADS environment variable should either not be exported (in which case the OMP_NUM_THREADS variable is used), or increased so that numpy can multithread outside of the local multithreading regions in the X-PSI extension modules.

Note that OpenBLAS may not be compiled against the OpenMP library but instead use Pthreads. If numpy is linked against MKL, we have covered all possibilities because MKL whilst uses OpenMP threading but the MKL_NUM_THREADS environment variable takes precedence if set and thus we ensure it is set to one.

The GSL library we installed (see above) is not a parallel library itself, and actually supplies a low-level layer of its own as a CBLAS implementation. This may be replaced with an optimised implementation, in which case the question of nested multithreading arises. The OpenBLAS and MKL implementations can detect whether library calls are made within OpenMP-parallel regions of the X-PSI source code provided the same threading library is used: e.g., OpenBLAS compiled with USE_OPENMP=1, or X-PSI compiled with an Intel compiler and linked against MKL.

Batch usage¶

For an example job script, refer to Example job.

Lisa (SURF)¶

Lisa follows the exact installation instructions as that of Snellius, except for the paths provided to the environment variable for the pre-load libs, which is as follows:

export LD_PRELOAD=/sw/arch/Debian10/EB_production/2021/software/imkl/2021.2.0-iimpi-2021a/mkl/2021.2.0/lib/intel64/libmkl_def.so.1:/sw/arch/Debian10/EB_production/2021/software/imkl/2021.2.0-iimpi-2021a/mkl/2021.2.0/lib/intel64/libmkl_avx2.so.1:/sw/arch/Debian10/EB_production/2021/software/imkl/2021.2.0-iimpi-2021a/mkl/2021.2.0/lib/intel64/libmkl_core.so:/sw/arch/Debian10/EB_production/2021/software/imkl/2021.2.0-iimpi-2021a/mkl/2021.2.0/lib/intel64/libmkl_intel_lp64.so:/sw/arch/Debian10/EB_production/2021/software/imkl/2021.2.0-iimpi-2021a/mkl/2021.2.0/lib/intel64/libmkl_intel_thread.so:/sw/arch/Debian10/EB_production/2021/software/imkl/2021.2.0-iimpi-2021a/compiler/2021.2.0/linux/compiler/lib/intel64_lin/libiomp5.so

Helios (API)¶

Helios is a cluster of the Anton Pannekoek Institute for Astronomy. Here we present two possible approaches to install X-PSI. We can use either a conda environment or install locally in the user home directory using python --user (but do not mix them). Note, however, that using the conda environment is safer if there ever will be need for conflicting auxiliary installations for the user in the cluster:

Helios (using conda)¶

Let’s start by loading the necessary modules and creating a conda environment:

git clone https://github.com/xpsi-group/xpsi.git
cd xpsi
git checkout python2
module load anaconda2/2019-10
conda env create -f basic_environment.yml
conda activate xpsi

module load openmpi/3.1.6

Let’s then install mpi4py:

cd; wget https://bitbucket.org/mpi4py/mpi4py/downloads/mpi4py-3.0.3.tar.gz
tar zxvf mpi4py-3.0.3.tar.gz
cd mpi4py-3.0.3
python setup.py build   --mpicc=/zfs/helios/filer0/sw-astro/api/openmpi/3.1.6/bin/mpicc
python setup.py install
mpiexec -n 4 python demo/helloworld.py

Let’s then install MultiNest and PyMultiNest (and checkout a version that is known to be Python 2.7 compatible):

cd; git clone https://github.com/farhanferoz/MultiNest.git multinest
cd multinest/MultiNest_v3.12_CMake/multinest
mkdir build
cd build
CC=gcc FC=mpif90 CXX=g++ cmake -DCMAKE_{C,CXX}_FLAGS="-O3 -march=native -funroll-loops" -DCMAKE_Fortran_FLAGS="-O3 -march=native -funroll-loops" ..
make

cd; git clone https://github.com/JohannesBuchner/PyMultiNest.git pymultinest
cd pymultinest
git checkout c8eba95
python setup.py install

Let’s then install GSL:

cd; wget -v http://mirror.koddos.net/gnu/gsl/gsl-latest.tar.gz
cd gsl-latest
mkdir build
cd build
../configure CC=gcc --prefix=$HOME/gsl
make
make check
make install
make installcheck
make clean
export PATH=$HOME/gsl/bin:$PATH

Let’s then finally install X-PSI:

cd; cd xpsi;
CC=gcc python setup.py install

Helios (using `python --user`)¶

Let’s start by loading the necessary modules and upgrading the setup-tools:

module load anaconda2/python2.7.16
module load openmpi/3.1.6
pip install --upgrade --user setuptools

Let’s then install mpi4py:

cd; wget https://bitbucket.org/mpi4py/mpi4py/downloads/mpi4py-3.0.3.tar.gz
tar zxvf mpi4py-3.0.3.tar.gz
cd mpi4py-3.0.3
python setup.py install --user
mpiexec -n 4 python demo/helloworld.py

Let’s then install MultiNest and PyMultiNest (and checkout a version that is known to be Python 2.7 compatible):

cd; git clone https://github.com/farhanferoz/MultiNest.git multinest
cd multinest/MultiNest_v3.12_CMake/multinest
mkdir build
cd build
CC=gcc FC=mpif90 CXX=g++ cmake -DCMAKE_{C,CXX}_FLAGS="-O3 -march=native -funroll-loops" -DCMAKE_Fortran_FLAGS="-O3 -march=native -funroll-loops" ..
make

cd; git clone https://github.com/JohannesBuchner/PyMultiNest.git pymultinest
cd pymultinest
git checkout c8eba95
python setup.py install --user

Let’s then install GSL:

cd; wget -v http://mirror.koddos.net/gnu/gsl/gsl-latest.tar.gz
cd gsl-latest
mkdir build
cd build
../configure CC=gcc --prefix=$HOME/gsl
make
make check
make install
make installcheck
make clean
export PATH=$HOME/gsl/bin:$PATH

Let’s then finally install X-PSI:

cd; git clone https://github.com/xpsi-group/xpsi.git
cd xpsi
git checkout python2
CC=gcc python setup.py install --user

Batch usage¶

For example job scripts, see Helios using conda environment or python --user in Example job.

CALMIP¶

CALMIP is the supercomputer of Université Fédérale de Toulouse

Installation¶

In your $HOME file system, from the login node, start by loading the necessary modules:

module purge
module load python/2.7.14
module load cmake
module load intel/18.2.199
module load intelmpi/18.2
module load gsl/2.5-icc

Then, install/update the required python packages:

pip install emcee==3.0.2  --user
pip install --upgrade numpy --user
pip install --upgrade Cython --user
pip install schwimmbad --user

Install MPI4PY in your $HOME:

wget https://bitbucket.org/mpi4py/mpi4py/downloads/mpi4py-3.0.0.tar.gz
tar -xvf mpi4py-3.0.0.tar.gz
cd mpi4py-3.0.0
python setup.py install --user

Download the MultiNest package in your $HOME:

git clone https://github.com/farhanferoz/MultiNest.git ~/multinest
cd ~/multinest/MultiNest_v3.11_CMake/multinest
mkdir build
cd build

Compile MultiNest in your $HOME, following recommendation from CALMIP support:

cmake -DCMAKE_INSTALL_PREFIX=~/multiNest \
        -DCMAKE_{C,CXX}_FLAGS="-O3 -xCORE-AVX512 -mkl" \
        -DCMAKE_Fortran_FLAGS="-O3 -xCORE-AVX512 -mkl" \
        -DCMAKE_C_COMPILER=mpiicc    \
        -DCMAKE_CXX_COMPILER=mpiicpc \
        -DCMAKE_Fortran_COMPILER=mpiifort  ..
make

Set up your library paths:

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HOME/multiNest/MultiNest_v3.12_CMake/multinest/lib
export PYTHONPATH=$HOME/.local/lib/python2.7/site-packages/:$PYTHONPATH
export LD_PRELOAD=$MKLROOT/lib/intel64/libmkl_core.so:$MKLROOT/lib/intel64/libmkl_sequential.so

Note that the module commands, and the library path commands above will have to be added in your SBATCH script (see Example job) to execute a run.

HPC systems¶

Snellius (SURF)¶

Installation¶

Environment variables¶

Batch usage¶

Lisa (SURF)¶

Helios (API)¶

Helios (using conda)¶

Helios (using python --user)¶

Batch usage¶

CALMIP¶

Installation¶

Helios (using `python --user`)¶