Installation

X-PSI is an open-source software package that is available on GitHub and can be cloned as:

git clone https://github.com/xpsi-group/xpsi.git </path/to/xpsi>

In this page, we lay down the instructions for installing X-PSI and all the necessary prerequisites on your local self-administered system.

Note

For installation on a high-performance computing system, we direct the reader to the HPC systems page for guidance since the instructions on this page are either not applicable or do not target performance.

Prerequisite Python Packages

X-PSI was originally developed in Python 2.7 and was ported to Python 3 as of X-PSI v2.0. We recommend creating a conda virtual environment with anaconda3 as per instructions below so as to not disrupt your Python ecosystem.

Conda Environment

In the source directory we provide a dependency file basic_environment.yml that installs the Python packages required for basic functionality of X-PSI. Its contents are:

name: xpsi_py3
channels:
    - defaults
dependencies:
    - numpy < 2.0.0
    - cython ~= 3.0.11
    - matplotlib
    - scipy
    - wrapt

The core packages required for likelihood functionality are numpy, cython, matplotlib, scipy, and wrapt.

To create a virtual environment from this file:

conda env create -f <path/to/xpsi>/basic_environment.yml

If conda does not solve the environment dependencies, you may need to create an environment manually via

conda create -n xpsi_py3

and then install the core dependencies listed in basic_environment.yml via conda.

Activate the environment as:

conda activate xpsi_py3

Note

ALL THE FOLLOWING STEPS SHOULD BE PERFORMED IN THIS NEWLY CREATED ENVIRONMENT. Pay special attention to reactivate the environment if you ever have to restart the kernel.

We start by installing the GNU Scientific Library (GSL):

conda install gsl

Next, install mpi4py which is required for nested sampling:

conda install -c conda-forge mpi4py

We also need PyMultiNest (the interface to the MultiNest library) for nested sampling. However, conda install -c conda-forge pymultinest might try to install dependencies in the environment, including binaries for MPI, BLAS/LAPACK, and a Fortran compiler, all in order to install MultiNest. Moreover, the MultiNest version listed is a minor release too low to satisfy all our needs. Thus, see the PyMultiNest instructions below.

Then, install optional packages getdist, h5py, nestcheck, and fgivenx which are required for post-processing:

conda install -c conda-forge getdist h5py nestcheck fgivenx

Note

However, to get the most updated versions of getdist and nestcheck (which may be needed by some of the X-PSI post-processing features), they should be installed from the source (https://github.com/cmbant/getdist and https://github.com/ejhigson/nestcheck) by cloning the repositories and running python setup.py install in them.

In addition, some optional miscellaneous packages are:

  1. jupyter if you want to run X-PSI in a notebook. You may also need the ipywidgets that can be installed with conda install -c conda-forge ipywidgets.

  2. pytest if you want to run functionality tests for X-PSI.

  3. emcee for optional ensemble-MCMC functionality.

  4. UltraNest as alternative sampler.

  5. sbi for performing Simulation-Based Inference (SBI). This additionally requires installation of torch and torchvision.

Prerequisite Non-Python Packages and PyMultiNest

X-PSI has dependencies that are not Python packages, or which are Python packages but need to be installed from source (PyMultiNest). Build and install guidelines are given below.

Note

The next steps require an OpenMP-enabled C compiler (known compatibility with icc, gcc, and clang). Most linux systems come with GCC built-in. To find out the GCC path-executable on your system, run which gcc.

MultiNest

Although production sampling runs need to be performed on a high-performance system and X-PSI can be installed locally without sampling functionality, it is advisable to install MultiNest on your personal machine to gain experience in application to inexpensive test problems. In addition, to leverage some capabilities of sample post-processing software you require MultiNest v3.12. To build the MultiNest library, you require an MPI-wrapped Fortran compiler (e.g., openmpi-mpifort from Open MPI).

Prerequisites for MultiNest are c and fortran compilers (e.g. gcc and gfortran), cmake, blas, lapack, and atlas. In case missing them, they can be installed by:

sudo apt-get install cmake libblas-dev liblapack-dev libatlas-base-dev

To have MPI-wrapped compilers, one should also install mpich if not installed already:

sudo apt install mpich

Assuming these libraries are available, first clone the repository, then navigate to it and build:

git clone https://github.com/farhanferoz/MultiNest.git <path/to/clone>/multinest
cd <path/to/clone>/multinest/MultiNest_v3.12_CMake/multinest/
mkdir build
cd build
CC=gcc FC=<path/to/working/mpifortran/compiler/>mpif90 CXX=g++ cmake -DCMAKE_{C,CXX}_FLAGS="-O3 -march=native -funroll-loops" -DCMAKE_Fortran_FLAGS="-O3 -march=native -funroll-loops" ..
make
ls ../lib/

Note

We note that new default mpif90 created by mpi4py conda installation may not work here. Thus, one needs to point the path to the native mpif90 compiler of the system (e.g. CC=gcc FC=/usr/bin/mpif90 CXX=g++ ...) or install mpi4py only after MultiNest has been installed and use then FC=mpif90.

Now you need the Python interface to MultiNest:

git clone https://github.com/JohannesBuchner/PyMultiNest.git <path/to/clone>/pymultinest
cd <path/to/clone>/pymultinest
python setup.py install [--user]

The package will be installed in your conda environment, if the environment is activated. In that case, the optional --user flag should be omitted. We also need PyMultinest to interface with MultiNest. To do so, add the following line to ~/.bashrc:

export LD_LIBRARY_PATH=/my/directory/MultiNest/lib/:$LD_LIBRARY_PATH

It’s also good to check whether this has worked. In a new kernel, try

python -c 'import pymultinest'

which should import without any errors. If you get ERROR:   Could not load MultiNest library "libmultinest.so", that means either MultiNest was not successfully installed or could not be found. While X-PSI will run properly, the nested-sampling capabilities (requiring MultiNest) will crash. The user can use emcee as the back-up sampler (see example in Modeling). Note however that the post-processing tutorials have only been implemented for the outputs of MultiNest.

X-PSI

Finally, to build and install from the X-PSI clone root, execute:

CC=<path/to/compiler/executable> python setup.py install [--user]

The --user flag is optional and specifies where the package is installed; if you want to install the package in a virtual environment (as recommended), omit this flag.

For icc, you may need to prepend this command with LDSHARED="icc -shared". This ensures that both the compiler and linker are Intel, otherwise the gcc linker might be invoked.

A quick check of the X-PSI installation can be done with import xpsi, which should print to screen something like the following:

/=============================================\
| X-PSI: X-ray Pulse Simulation and Inference |
|---------------------------------------------|
|                Version: 2.2.0               |
|---------------------------------------------|
|      https://xpsi-group.github.io/xpsi      |
\=============================================/

Imported GetDist version: 1.4
Imported nestcheck version: 0.2.1

Note

Importing X-PSI should not be done in the X-PSI root directory (where the setup.py file locates). Otherwise, a following type of error is expected: ImportError: cannot import name 'set_phase_interpolant' from 'xpsi.tools' (unknown location)

For a more complete verification of the X-PSI installation, you can execute the following:

cd examples/examples_fast/Modules/
python main.py

This module performs a likelihood check. If the likelihood value calculated matches the given value, X-PSI is functioning as expected, else it will raise an error message. The following part of this module requires a functioning MultiNest installation. It initiates sampling using MultiNest, and given the settings, it should take ~5 minutes. To cancel mid-way press ctrl + C.

Note

Note that in X-PSI versions before 2.1.0 the selection of the atmosphere extension needed to be done when installing X-PSI using appropriate flags:

CC=<path/to/compiler/executable> python setup.py --help
CC=<path/to/compiler/executable> python setup.py install [--NumHot] [--NumElse] [--user]

This installed the numerical atmosphere for the hot regions and/or for the rest of the surface (elsewhere). To (re-) install the default blackbody surface emission model, the following command without the flags was used:

CC=<path/to/compiler/executable> python setup.py install [--user]

For X-PSI versions newer than 2.1.0 atmosphere selection is done without reinstalling X-PSI.

If you ever need to reinstall, first clean to recompile the C files:

rm -r build dist *egg* xpsi/*/*.c xpsi/include/rayXpanda/*.o

Alternatively, to build X-PSI in-place:

CC=<path/to/compiler/executable> python setup.py build_ext -i

This will build extension modules in the source code directory. You must in this case ensure that the source code directory is on your PYTHONPATH environment variable, or inserted into sys.path within a calling module.

Documentation

If you wish to compile the documentation you require Sphinx and extensions. To install these, run the following command:

conda install "sphinx<7.0"
conda install -c conda-forge nbsphinx
conda install decorator
conda install sphinxcontrib-websupport
conda install sphinx_rtd_theme

Now the documentation can be compiled using:

cd xpsi/docs; [make clean;] make html

To rebuild the documentation after a change to source code docstrings:

[CC=<path/to/compiler/executable>] python setup.py install [--user]; cd
docs; make clean; make html; cd ..

The .html files can then be found in xpsi/docs/build/html, along with the notebooks for the tutorials in this documentation. The .html files can naturally be opened in a browser, handily via a Jupyter session (this is particularly useful if the edits are to tutorial notebooks).

Note that if you require links to the source code in the HTML files, you need to ensure Sphinx imports the xpsi package from the source directory instead of from the ~/.local/lib directory of the user. To enforce this, insert the path to the source directory into sys.path in the conf.py script. Then make sure the extension modules are inside the source directory – i.e., the package is built in-place (see above).

Note

To build the documentation, all modules need to be imported, and the dependencies that are not resolved will print warning messages.

Tips for installing on Mac OS

Most of the aforementioned instructions for linux are also applicable for Mac OS. Here we note some of the changes required.

After creating the environment using the environment.yml file, install xcode or xcode tools. Be mindful of the sequence of programs to be installed hereafter. Use pip install to download and install h5py and emcee (and maplotlib, numpy, scipy and cython ~= 3.0.11 if not using the environment.yml. You may use the file as a reference of the packages required).

On Mac OS, it’s preferable to use llvm clang rather than gcc. The homebrew version of clang works, but some users may face potential issues (see below for the MacOS native clang). To use homebrew version of clang, first install homebrew:

/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

Install llvm with homebrew, even if weird messages appear, saying llvm is already present in the Mac OS:

brew install llvm

Install GSL (see above).

Install fortran before MPI. If faced with issues when specifying or using gfortran (and it “does not pass simple tests”) specify the compiler as being gfortran in the mpif90 wrapper files and delete the files that were already in the build directory. Once MPI is installed, export the following environment variables:

export LD_LIBRARY_PATH="/Users/<your_path>/openmpi/lib:$LD_LIBRARY_PATH"
export PATH=$PATH:/Users/<your_path>/mpi/bin/
export LDFLAGS="-L/usr/local/opt/llvm/lib"
export CPPFLAGS="-I/usr/local/opt/llvm/include"
export KMP_DUPLICATE_LIB_OK=TRUE

Consider adding these lines directly in your bashrc (or equivalent file for a different shell e.g. zshrc).

Install X-PSI using:

CC=/usr/local/opt/llvm/bin/clang python setup.py install [--user]

If you are facing problems with this installation (e.g., linker problems, or –fopenmp libraries missing), you may try the following:

CC=/usr/local/opt/llvm/bin/clang python setup.py install --noopenmp [--user]

You may also try to use the MacOS native version of clang:

CC=/usr/bin/clang python setup.py install --noopenmp [--user]

If you encounter any problems with permissions when installing X-PSI, use the --user option (This will install X-PSI globally, and not just within your virtual environment).

Note

We are encountering issues with installing MultiNest on Mac and we are working on proposing a solution.

Tips for installing on Windows

Note

We do not recommend installing and running X-PSI on windows. However, if you must, this section details some of the relevant procedures.

X-PSI was successfully installed and run on Windows in the year 2020, at least for the purpose of likelihood functionality, using the following user-contributed procedure.

  • Clone the X-PSI repository to a directory on your Windows computer (see above).

  • Download Ubuntu for Windows.

  • Install a Anaconda or Miniconda virtual Python environment in an Ubuntu shell.

  • Follow the instructions of this page to install all the python and non-python packages.