Installation
X-PSI is an open-source software package that is available on GitHub and can be cloned as:
git clone https://github.com/xpsi-group/xpsi.git </path/to/xpsi>
In this page, we lay down the instructions for installing X-PSI and all the necessary prerequisites on your local self-administered system.
Note
For installation on a high-performance computing system, we direct the reader to the HPC systems page for guidance since the instructions on this page are either not applicable or do not target performance.
Prerequisite Python Packages
X-PSI was originally developed in Python 2.7 and was ported to Python 3 as of X-PSI v2.0. We recommend creating a conda virtual environment with anaconda3 as per instructions below so as to not disrupt your Python ecosystem.
Conda Environment
In the source directory we provide a dependency file basic_environment.yml
that
installs the Python packages required for basic functionality of X-PSI. Its
contents are:
name: xpsi_py3
channels:
- defaults
dependencies:
- numpy < 2.0.0
- cython ~= 3.0.11
- matplotlib
- scipy
- wrapt
The core packages required for likelihood functionality are numpy, cython, matplotlib, scipy, and wrapt.
To create a virtual environment from this file:
conda env create -f <path/to/xpsi>/basic_environment.yml
If conda does not solve the environment dependencies, you may need to create an environment manually via
conda create -n xpsi_py3
and then install the core dependencies listed in basic_environment.yml
via
conda.
Activate the environment as:
conda activate xpsi_py3
Note
ALL THE FOLLOWING STEPS SHOULD BE PERFORMED IN THIS NEWLY CREATED ENVIRONMENT. Pay special attention to reactivate the environment if you ever have to restart the kernel.
We start by installing the GNU Scientific Library (GSL):
conda install gsl
Next, install mpi4py which is required for nested sampling:
conda install -c conda-forge mpi4py
We also need PyMultiNest (the interface to the MultiNest library) for nested sampling. However, conda install -c conda-forge pymultinest might try to install dependencies in the environment, including binaries for MPI, BLAS/LAPACK, and a Fortran compiler, all in order to install MultiNest. Moreover, the MultiNest version listed is a minor release too low to satisfy all our needs. Thus, see the PyMultiNest instructions below.
Then, install optional packages getdist, h5py, nestcheck, and fgivenx which are required for post-processing:
conda install -c conda-forge getdist h5py nestcheck fgivenx
Note
However, to get the most updated versions of getdist and nestcheck (which may be needed by
some of the X-PSI post-processing features), they should be installed from the source
(https://github.com/cmbant/getdist and https://github.com/ejhigson/nestcheck)
by cloning the repositories and running python setup.py install
in them.
In addition, some optional miscellaneous packages are:
jupyter if you want to run X-PSI in a notebook. You may also need the
ipywidgets
that can be installed withconda install -c conda-forge ipywidgets
.pytest if you want to run functionality tests for X-PSI.
emcee for optional ensemble-MCMC functionality.
UltraNest as alternative sampler.
sbi for performing Simulation-Based Inference (SBI). This additionally requires installation of torch and torchvision.
Prerequisite Non-Python Packages and PyMultiNest
X-PSI has dependencies that are not Python packages, or which are Python packages but need to be installed from source (PyMultiNest). Build and install guidelines are given below.
Note
The next steps require an OpenMP-enabled C
compiler (known compatibility with icc
, gcc
, and clang
). Most
linux systems come with GCC built-in. To find out
the GCC path-executable on your system, run which gcc
.
MultiNest
Although production sampling runs need to be performed on a high-performance
system and X-PSI can be installed locally without sampling functionality, it is
advisable to install MultiNest on your personal machine to gain experience in
application to inexpensive test problems. In addition, to leverage some
capabilities of sample post-processing software you
require MultiNest v3.12
.
To build the MultiNest library, you require an MPI-wrapped Fortran compiler
(e.g., openmpi-mpifort
from Open MPI).
Prerequisites for MultiNest are c and fortran
compilers (e.g. gcc
and gfortran
), cmake
, blas
, lapack
, and
atlas
. In case missing them, they can be installed by:
sudo apt-get install cmake libblas-dev liblapack-dev libatlas-base-dev
To have MPI-wrapped compilers, one should also install mpich
if not installed already:
sudo apt install mpich
Assuming these libraries are available, first clone the repository, then navigate to it and build:
git clone https://github.com/farhanferoz/MultiNest.git <path/to/clone>/multinest
cd <path/to/clone>/multinest/MultiNest_v3.12_CMake/multinest/
mkdir build
cd build
CC=gcc FC=<path/to/working/mpifortran/compiler/>mpif90 CXX=g++ cmake -DCMAKE_{C,CXX}_FLAGS="-O3 -march=native -funroll-loops" -DCMAKE_Fortran_FLAGS="-O3 -march=native -funroll-loops" ..
make
ls ../lib/
Note
We note that new default mpif90 created by mpi4py conda installation may not work here. Thus, one needs to point the path to the native mpif90 compiler of the system (e.g. CC=gcc FC=/usr/bin/mpif90 CXX=g++ ...
) or install mpi4py only after MultiNest has been installed and use then FC=mpif90
.
Now you need the Python interface to MultiNest:
git clone https://github.com/JohannesBuchner/PyMultiNest.git <path/to/clone>/pymultinest
cd <path/to/clone>/pymultinest
python setup.py install [--user]
The package will be installed in your conda environment, if the environment is activated.
In that case, the optional --user
flag should be omitted.
We also need PyMultinest to interface with MultiNest. To do so, add the
following line to ~/.bashrc
:
export LD_LIBRARY_PATH=/my/directory/MultiNest/lib/:$LD_LIBRARY_PATH
It’s also good to check whether this has worked. In a new kernel, try
python -c 'import pymultinest'
which should import without any errors. If you get ERROR: Could not load
MultiNest library "libmultinest.so"
, that means either MultiNest was not
successfully installed or could not be found. While X-PSI will run properly,
the nested-sampling capabilities (requiring MultiNest) will crash. The user can
use emcee as the back-up sampler (see example in Modeling).
Note however that the post-processing tutorials have only been implemented
for the outputs of MultiNest.
X-PSI
Finally, to build and install from the X-PSI clone root, execute:
CC=<path/to/compiler/executable> python setup.py install [--user]
The --user
flag is optional and specifies where the package is installed;
if you want to install the package in a virtual environment (as recommended), omit this flag.
For icc
, you may need to prepend this command with
LDSHARED="icc -shared"
. This ensures that both the compiler and linker
are Intel, otherwise the gcc
linker might be invoked.
A quick check of the X-PSI installation can be done with import xpsi
, which
should print to screen something like the following:
/=============================================\
| X-PSI: X-ray Pulse Simulation and Inference |
|---------------------------------------------|
| Version: 2.2.0 |
|---------------------------------------------|
| https://xpsi-group.github.io/xpsi |
\=============================================/
Imported GetDist version: 1.4
Imported nestcheck version: 0.2.1
Note
Importing X-PSI should not be done in the X-PSI root directory (where the setup.py
file locates).
Otherwise, a following type of error is expected:
ImportError: cannot import name 'set_phase_interpolant' from 'xpsi.tools' (unknown location)
For a more complete verification of the X-PSI installation, you can execute the following:
cd examples/examples_fast/Modules/
python main.py
This module performs a likelihood check
. If the likelihood value calculated
matches the given value, X-PSI is functioning as expected, else it will raise
an error message. The following part of this module requires a functioning
MultiNest installation. It initiates sampling using MultiNest, and given the
settings, it should take ~5 minutes. To cancel mid-way press ctrl + C
.
Note
Note that in X-PSI versions before 2.1.0 the selection of the atmosphere extension needed to be done when installing X-PSI using appropriate flags:
CC=<path/to/compiler/executable> python setup.py --help
CC=<path/to/compiler/executable> python setup.py install [--NumHot] [--NumElse] [--user]
This installed the numerical atmosphere for the hot regions and/or for
the rest of the surface (elsewhere
). To (re-) install the default
blackbody surface emission model, the following command without the flags
was used:
CC=<path/to/compiler/executable> python setup.py install [--user]
For X-PSI versions newer than 2.1.0 atmosphere selection is done without reinstalling X-PSI.
If you ever need to reinstall, first clean to recompile the C files:
rm -r build dist *egg* xpsi/*/*.c xpsi/include/rayXpanda/*.o
Alternatively, to build X-PSI in-place:
CC=<path/to/compiler/executable> python setup.py build_ext -i
This will build extension modules in the source code directory. You must in
this case ensure that the source code directory is on your PYTHONPATH
environment variable, or inserted into sys.path
within a calling module.
Documentation
If you wish to compile the documentation you require Sphinx and extensions. To install these, run the following command:
conda install "sphinx<7.0"
conda install -c conda-forge nbsphinx
conda install decorator
conda install sphinxcontrib-websupport
conda install sphinx_rtd_theme
Now the documentation can be compiled using:
cd xpsi/docs; [make clean;] make html
To rebuild the documentation after a change to source code docstrings:
[CC=<path/to/compiler/executable>] python setup.py install [--user]; cd
docs; make clean; make html; cd ..
The .html
files can then be found in xpsi/docs/build/html
, along with the
notebooks for the tutorials in this documentation. The .html
files can
naturally be opened in a browser, handily via a Jupyter session (this is
particularly useful if the edits are to tutorial notebooks).
Note that if you require links to the source code in the HTML files, you need
to ensure Sphinx imports the xpsi
package from the source directory
instead of from the ~/.local/lib
directory of the user. To enforce this,
insert the path to the source directory into sys.path
in the conf.py
script. Then make sure the extension modules are inside the source directory
– i.e., the package is built in-place (see above).
Note
To build the documentation, all modules need to be imported, and the dependencies that are not resolved will print warning messages.
Tips for installing on Mac OS
Most of the aforementioned instructions for linux are also applicable for Mac OS. Here we note some of the changes required.
After creating the environment using the environment.yml
file,
install xcode
or xcode tools
. Be mindful of the sequence of programs to
be installed hereafter. Use pip install
to download and install h5py
and emcee
(and maplotlib
, numpy
, scipy
and cython ~= 3.0.11
if not using the environment.yml
. You may use the file as a reference of the
packages required).
On Mac OS, it’s preferable to use llvm clang
rather than gcc
. The
homebrew
version of clang
works, but some users may face potential
issues (see below for the MacOS native clang
). To use homebrew
version
of clang
, first install homebrew
:
/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
Install llvm
with homebrew, even if weird messages appear, saying llvm is
already present in the Mac OS:
brew install llvm
Install GSL
(see above).
Install fortran
before MPI
. If faced with issues when specifying or
using gfortran (and it “does not pass simple tests”) specify the compiler as
being gfortran in the mpif90
wrapper files and delete the files that were
already in the build directory. Once MPI
is installed, export the following
environment variables:
export LD_LIBRARY_PATH="/Users/<your_path>/openmpi/lib:$LD_LIBRARY_PATH"
export PATH=$PATH:/Users/<your_path>/mpi/bin/
export LDFLAGS="-L/usr/local/opt/llvm/lib"
export CPPFLAGS="-I/usr/local/opt/llvm/include"
export KMP_DUPLICATE_LIB_OK=TRUE
Consider adding these lines directly in your bashrc (or equivalent file for a different shell e.g. zshrc).
Install X-PSI
using:
CC=/usr/local/opt/llvm/bin/clang python setup.py install [--user]
If you are facing problems with this installation (e.g., linker problems, or –fopenmp libraries missing), you may try the following:
CC=/usr/local/opt/llvm/bin/clang python setup.py install --noopenmp [--user]
You may also try to use the MacOS native version of clang
:
CC=/usr/bin/clang python setup.py install --noopenmp [--user]
If you encounter any problems with permissions when installing X-PSI, use the
--user
option (This will install X-PSI globally, and not just within your
virtual environment).
Note
We are encountering issues with installing MultiNest on Mac and we are working on proposing a solution.
Tips for installing on Windows
Note
We do not recommend installing and running X-PSI on windows. However, if you must, this section details some of the relevant procedures.
X-PSI was successfully installed and run on Windows in the year 2020, at least for the purpose of likelihood functionality, using the following user-contributed procedure.
Clone the X-PSI repository to a directory on your Windows computer (see above).
Download Ubuntu for Windows.
Install a Anaconda or Miniconda virtual Python environment in an Ubuntu shell.
Follow the instructions of this page to install all the python and non-python packages.