Example job

Snellius

The following job script job.sh is an example job script used by the X-PSI core team for NICER data analysis on the Snellius system (see HPC systems).

#!/bin/bash
#SBATCH -N 5
#SBATCH --tasks-per-node=128
#SBATCH -t 1-00:00:00
#SBATCH -p thin
#SBATCH --job-name=run1

echo start of job in directory $SLURM_SUBMIT_DIR
echo number of nodes is $SLURM_JOB_NUM_NODES
echo the allocated nodes are:
echo $SLURM_JOB_NODELIST

module load 2021
module load intel/2021a
module load Python/2.7.18-GCCcore-10.3.0-bare

cp -r $HOME/NICER_analyses/J0437/CST_PDT/ "$TMPDIR"

export PYTHONPATH=$HOME/.local/lib/python2.7/site-packages/:$PYTHONPATH

export OMP_NUM_THREADS=1
export OPENBLAS_NUM_THREADS=1
export GOTO_NUM_THREADS=1
export MKL_NUM_THREADS=1
export LD_PRELOAD=/sw/arch/Centos8/EB_production/2021/software/imkl/2021.2.0-iimpi-2021a/mkl/2021.2.0/lib/intel64/libmkl_def.so.1:/sw/arch/Centos8/EB_production/2021/software/imkl/2021.2.0-iimpi-2021a/mkl/2021.2.0/lib/intel64/libmkl_avx2.so.1:/sw/arch/Centos8/EB_production/2021/software/imkl/2021.2.0-iimpi-2021a/mkl/2021.2.0/lib/intel64/libmkl_core.so:/sw/arch/Centos8/EB_production/2021/software/imkl/2021.2.0-iimpi-2021a/mkl/2021.2.0/lib/intel64/libmkl_intel_lp64.so:/sw/arch/Centos8/EB_production/2021/software/imkl/2021.2.0-iimpi-2021a/mkl/2021.2.0/lib/intel64/libmkl_intel_thread.so:/sw/arch/Centos8/EB_production/2021/software/imkl/2021.2.0-iimpi-2021a/compiler/2021.2.0/linux/compiler/lib/intel64_lin/libiomp5.so
export LD_LIBRARY_PATH=$HOME/multinest/MultiNest_v3.12_CMake/multinest/lib:$LD_LIBRARY_PATH

srun python main_run1.py @config.ini > out_run1 2> err_run1

cp run1* out_run1 err_run1 $HOME/NICER_analyses/J0437/CST_PDT/CST_PDT_outputs/init/.
#end of job file

A corresponding resume script would look like:

#!/bin/bash
#SBATCH -N 30
#SBATCH --tasks-per-node=128
#SBATCH -t 2-00:00:00
#SBATCH -p thin
#SBATCH --job-name=run1_r1

echo start of job in directory $SLURM_SUBMIT_DIR
echo number of nodes is $SLURM_JOB_NUM_NODES
echo the allocated nodes are:
echo $SLURM_JOB_NODELIST

module load 2021
module load intel/2021a
module load Python/2.7.18-GCCcore-10.3.0-bare

cp -r $HOME/NICER_analyses/J0437/CST_PDT/ "$TMPDIR"

export PYTHONPATH=$HOME/.local/lib/python2.7/site-packages/:$PYTHONPATH

export OMP_NUM_THREADS=1
export OPENBLAS_NUM_THREADS=1
export GOTO_NUM_THREADS=1
export MKL_NUM_THREADS=1
export LD_PRELOAD=/sw/arch/Centos8/EB_production/2021/software/imkl/2021.2.0-iimpi-2021a/mkl/2021.2.0/lib/intel64/libmkl_def.so.1:/sw/arch/Centos8/EB_production/2021/software/imkl/2021.2.0-iimpi-2021a/mkl/2021.2.0/lib/intel64/libmkl_avx2.so.1:/sw/arch/Centos8/EB_production/2021/software/imkl/2021.2.0-iimpi-2021a/mkl/2021.2.0/lib/intel64/libmkl_core.so:/sw/arch/Centos8/EB_production/2021/software/imkl/2021.2.0-iimpi-2021a/mkl/2021.2.0/lib/intel64/libmkl_intel_lp64.so:/sw/arch/Centos8/EB_production/2021/software/imkl/2021.2.0-iimpi-2021a/mkl/2021.2.0/lib/intel64/libmkl_intel_thread.so:/sw/arch/Centos8/EB_production/2021/software/imkl/2021.2.0-iimpi-2021a/compiler/2021.2.0/linux/compiler/lib/intel64_lin/libiomp5.so
export LD_LIBRARY_PATH=$HOME/multinest/MultiNest_v3.12_CMake/multinest
export LD_LIBRARY_PATH=$HOME/MultiNest_v3.11_CMake/multinest/lib:$LD_LIBRARY_PATH

srun python main_run1_resume1.py @config.ini> out_run1_resume1 2> err_run1_resume1

cp run1* out_run1_resume1 err_run1_resume1 $HOME/NICER_analyses/J0437/CST_PDT/CST_PDT_outputs/resume1/.
#end of job file

Note how srun is aware of the MPI World, so there is no need specify the number of processes to spawn as a flag argument. Also note that the number of processes (which we set to equal the number of physical cores per node) specified in the top directives is far higher than for the initial run. This is because parallelisation efficiency scales with the local rejection fraction during a nested sampling iteration.

Finally, note that only the root process will generate output for inspection.

Helios

If X-PSI is installed using the conda environment in the Helios system (see HPC systems), we can use the following type of job script:

#!/bin/bash
#SBATCH -N 2
#SBATCH --tasks-per-node=126
#SBATCH -t 3-00:00:00
#SBATCH --partition=neutron-star
#SBATCH --job-name=run1
#SBATCH --o out
#SBATCH --e err
#SBATCH --mem 160000

module purge
module load anaconda2/2019-10
conda activate xpsi
module load openmpi/3.1.6

export OMP_NUM_THREADS=1
export OPENBLAS_NUM_THREADS=1
export GOTO_NUM_THREADS=1
export MKL_NUM_THREADS=1
export LD_LIBRARY_PATH=$HOME/multinest/MultiNest_v3.12_CMake/multinest/lib/:$LD_LIBRARY_PATH
export PATH=$HOME/gsl/bin:$PATH

srun -mca btl_tcp_if_include ib0 python main_run1.py > out_run1 2> err_run1

#end of job file

However, note that the above example is not making use of the faster communication between different nodes by using the scratch space called hddstore.

If X-PSI is installed using the python --user approach, we can use the following type of job script (making now also use of the scratch space):

#!/bin/bash
#SBATCH -N 2
#SBATCH --tasks-per-node=126
#SBATCH -t 3-00:00:00
#SBATCH --partition=neutron-star
#SBATCH --job-name=run1
#SBATCH --o out
#SBATCH --e err
#SBATCH --mem 160000

module purge
module load anaconda2/python2.7.16
module load openmpi/3.1.6

export OMP_NUM_THREADS=1
export OPENBLAS_NUM_THREADS=1
export GOTO_NUM_THREADS=1
export MKL_NUM_THREADS=1
export PYTHONPATH=$HOME/.local/lib/python2.7/site-packages/:$PYTHONPATH
export LD_LIBRARY_PATH=$HOME/multinest/MultiNest_v3.12_CMake/multinest/lib/:$LD_LIBRARY_PATH
export PATH=$HOME/gsl/bin:$PATH
export JOB_DIR=$HOME/NICER_analyses/J0030_ST_PST/STPST_modules

#Save output only in the primary node
export OUTPUT_FOLDER=$(mktemp -d -p /hddstore/$USER)
echo $OUTPUT_FOLDER $SLURMD_NODENAME
mkdir $OUTPUT_FOLDER/run1
cd $OUTPUT_FOLDER

#Copy the input data to be visible for all the nodes (and make sure your paths point to hddstore):
srun -n $SLURM_JOB_NUM_NODES --ntasks-per-node=1 mkdir -p /hddstore/$USER/data
sleep 1
srun -n $SLURM_JOB_NUM_NODES --ntasks-per-node=1 cp -r $HOME/NICER_analyses/data/* /hddstore/$USER/data/
sleep 1

mpiexec -n 252 -mca btl_tcp_if_include ib0 python $JOB_DIR/main_run1.py @$JOB_DIR/config.ini > out1 2> err1
sleep 1

#Move your output from scratch to storage space.
mkdir -p /zfs/helios/filer0/$USER/
cp -r $OUTPUT_FOLDER/* /zfs/helios/filer0/$USER/

#Clean the scratch automatically here.
#But remember to remove manually in each node, if the main program ends by crashing.
rm -rf $OUTPUT_FOLDER
srun -n $SLURM_JOB_NUM_NODES --ntasks-per-node=1 rm -rf /hddstore/$USER/data

#end of job file