Installation¶
X-PSI is an open-source software package that is available on GitHub and can be cloned as:
git clone https://github.com/xpsi-group/xpsi.git </path/to/xpsi>
In this page, we lay down the instructions for installing X-PSI and all the necessary pre-requisites on your local self-administered system.
Note
For installation on a high-performance computing system, we direct the reader to the HPC systems page for guidance since the instructions on this page are either not applicable or do not target performance.
Python environment¶
X-PSI was developed in Python 2.7, and is in the process of being ported to Python 3. Fortunately, there are several ways to create a virtual environment with a different version of Python, without disrupting your Python ecosystem. In this page, we present the installation procedure using Python 2.7. To use Python 2.7 compatible X-PSI version, we need to switch from the main branch to a branch called python2:
cd </path/to/xpsi>
git checkout python2
This section is divided into two subsections. We recommend that the user follow the instructions in the Basic Conda environment subsection to begin with. If faced with installation issues, the user may refer to the Environment duplication for diagnosis subsection.
Basic Conda environment¶
In the source directory we provide a basic dependency file that installs the core Python packages required for likelihood functionality. These packages are:
If you want to run X-PSI in a
Jupyter
notebook, you can add this as an entry (e.g., - jupyter=1.0
) to the
environment file or you can install it via Conda (or pip) after environment
creation.
To create a virtual environment from file:
conda env create -f <path/to/xpsi>/basic_environment.yml
If Conda does not solve the environment dependencies, you may need to create an environment manually via
conda create -n xpsi python=2.7
and then install the core dependencies listed in basic_environment.yml, such as NumPy and Cython.
All the following steps need to be performed in this newly created environment which can be activated as:
conda activate xpsi
Environment duplication for diagnosis¶
In the source repository we provide another dependency file that can facilitate
exact duplication of the environment from which X-PSI v0.6
was
released. This information may be useful if trying to diagnose installation
problems, but can only be expected to be compatible with the same platform. We therefore recommend users to instead try and follow the steps mentioned in the previous subsection, and use this section as more of a guidance if required.
The development environment:
- Ubuntu 14.04
- Installed globally via
apt
: GCC 4.8.4; Open MPI 1.6.5 (“ancient”); BLAS; LAPACK; ATLAS.- Miniconda2 (Python 2.7; 64-bit)
- Conda environment exported to
xpsi/environment.yml
When inspecting the xpsi/environment.yml
file, note that most of the
entries were installed via automatic resolution of a strict dependency chain
when core packages were specified. Also note the packages that
were installed into a Conda environment via pip. There are a few reasons
for these choices, but the main one is that pip is purely for Python
packages and will not install unwanted non-Python libraries. To be clear, such
libraries would be dependencies that could have been installed via Conda,
if we had not already satisfied them as listed above in this instance.
The Python packages below can be installed straightforwardly from source or via a package manager (Conda, pip, or a combination), via the instructions native to the packages. When searching for an open-source package you may need to add conda-forge package channel.
To duplicate from file:
conda env create -f <path/to/xpsi>/environment.yml
Dependencies¶
Note
For installing X-PSI on a Mac OS or Windows, please look at the tips below before proceeding with the installation of the various depnedencies.
Python dependencies¶
The following Python packages are required for nested sampling (see below how to install PyMultiNest and mpi4py from source):
- PyMultiNest (the interface to the MultiNest library)
- mpi4py (for parallelisation)
- mpifort (MPI-wrapped Fortran compiler for building library)
Note
That conda install -c conda-forge pymultinest
might try to install
dependencies in the environment, including binaries for MPI, BLAS/LAPACK,
and a Fortran compiler, all in order to install MultiNest. Moreover, the
MultiNest version listed is a minor release too low to satisfy all our
needs. Although production sampling runs need to be performed on a
high-performance system and X-PSI can locally be installed without sampling
functionality, it is advisable to install MultiNest on your
personal machine to gain experience on application to inexpensive test
problems. Below we offer from source instructions.
Running the tests requires:
The following Python packages are required for full functionality of the post-processing module:
- GetDist (posterior KDE corner plotting)[1]
- h5py (storage of X-ray signals computed from posterior samples; also used by emcee)
- nestcheck (posterior error analysis, plotting, run combination, etc.)[2]
- fgivenx (conditional posterior plotting; also required by nestcheck)
Note that post-processing can generally be done on a desktop computer and thus these packages are not necessary for running sampling processes on a high-performance system. If they are not installed, a warning message is printed or an exception is raised (by the root process if MPI world size >1).
The emcee Python package for ensemble-MCMC is optional.
Note
That pip install emcee==3.0.2 [--user]
installs a version working with Python 2.
Footnotes
[1] | The version of GetDist currently compatible with X-PSI, and used in Riley et al. (2019), is v0.3.1. It may be cloned as follows: git clone [--single-branch] -b customisation \
https://github.com/ThomasEdwardRiley/getdist.git
|
[2] | The version of nestcheck currently compatible with X-PSI, and used in Riley et al. (2019), is v0.2.0. It may be cloned as follows: git clone [--single-branch] -b feature/getdist_kde \
https://github.com/ThomasEdwardRiley/nestcheck.git
|
From source¶
X-PSI has several dependencies that are not Python packages. Build and install guidelines are given below.
GSL¶
To obtain the latest GSL source code (otherwise v2.5
works):
wget -v http://mirror.koddos.net/gnu/gsl/gsl-latest.tar.gz
Note
The next steps require an OpenMP-enabled C compiler (known compatibility with icc
, gcc
, and
clang
). Most linux systems come with GCC built-in. To find out the GCC path-executable on your system, run which gcc
.
Untar, navigate to the build directory (e.g., cd gsl-latest/build
), and
then build and install:
../configure CC=<path/to/compiler/executable> --prefix=$HOME/gsl
make
make check
make install
make installcheck
make clean
This will install the library in your $HOME
, as an example. You can check
the prefix and version of GSL on your path:
gsl-config --version
gsl-config --prefix
MultiNest¶
To leverage some capabilities of sample post-processing software you require
MultiNest v3.12
. To build the MultiNest library,
you require an MPI-wrapped Fortran compiler (e.g., mpifort
from Open MPI).
Note
The following assumes an environment similar to that summarised in the in the Python environment section above, specifically to emphasise where an MPI compiler wrapper is required.
First clone the repository, then navigate to it and build:
git clone https://github.com/farhanferoz/MultiNest.git <path/to/clone>/multinest
cd <path/to/clone>/multinest/MultiNest_v3.12_CMake/multinest/
mkdir build
cd build
CC=gcc FC=mpif90 CXX=g++ cmake -DCMAKE_{C,CXX}_FLAGS="-O3 -march=native -funroll-loops" -DCMAKE_Fortran_FLAGS="-O3 -march=native -funroll-loops" ..
make
ls ../lib/
Use the last command to check for the presence of shared objects. There is
no need to make install
as suggested in the source code documentation.
Note
If prompted about missing cmake
and gfortran
, they can simply be installed as sudo apt-get install cmake gfortran
If you have not already installed mpi4py using pip (or Conda assuming a
different environment setup to that summarised in Python environment), then here
is how to do it from source (e.g., on some path such as $HOME
):
wget https://bitbucket.org/mpi4py/mpi4py/downloads/mpi4py-3.0.0.tar.gz
tar -xf mpi4py-3.0.0.tar.gz
python setup.py build --mpicc=mpicc
python setup.py install
The package will be installed in your Conda environment (if activated).
To test:
mpiexec -n 4 python demo/helloworld.py
Do you see ranks 0 through 3 reporting for duty? The number of MPI processes
might be best set to somewhere between the number of physical cores and
logical cores in your machine for test sampling applications. For a typical
laptop that might be up to -n 4
.
You also need to set the environment variable for the library path to point at MultiNest:
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HOME/multinest/MultiNest_v3.12_CMake/multinest/lib/
Now you need the Python interface to MultiNest:
git clone https://github.com/JohannesBuchner/PyMultiNest.git <path/to/clone>/pymultinest
cd <path/to/clone>/pymultinest
python setup.py install [--user]
The package will be installed in your Conda environment (if activated).
Note
Here we clone the latest PyMultiNest repository. However, to make sure that PyMultiNest is compatible with Python 2.7 version of X-PSI, you can e.g., checkout the following PyMultiNest commit: git checkout c8eba95
, before running the setup.py file. For Riley et al. (2019),
working with X-PSI v0.1
, we used the repository as frozen in a fork.
To clone this version instead:
git clone https://github.com/ThomasEdwardRiley/PyMultiNest.git <path/to/clone>
and then simply follow the same installation procedure.
X-PSI¶
Finally, to build and install from the X-PSI clone root, execute:
CC=<path/to/compiler/executable> python setup.py install [--user]
The --user
flag is optional and specifies where the package is installed;
if you want to install the package in a virtual environment, omit this flag.
For icc
, you may need to prepend this command with
LDSHARED="icc -shared"
. This ensures that both the compiler and linker
are Intel, otherwise the gcc
linker might be invoked.
Provided the GSL <prefix>/bin
is in your PATH
environment variable, the X-PSI setup.py
script will automatically use the
gsl-config
executable to link the shared libraries and give the required
C flags for compilation of the X-PSI extensions. Because the library location
will not change for runtime, we state the runtime linking instructions at
compilation in the setup.py
script.
To check whether installation proceeded correctly and the software is functioning as expected, execute the following:
cd examples/examples_fast/Modules/
python main.py
This module performs a likelihood check
. If the likelihood value calculated matches
the given value, X-PSI is functioning as expected, else it will raise an error message.
The module will then initiate sampling using MultiNest (assuming that it’s installed),
and given the settings, it should take ~5 minutes. To cancel mid-way press ctrl + C
.
Note
The default X-PSI is installed with an analytical blackbody surface emission model extension. If you want to use alternative models for the surface radiation field, you will need to (re-)install / (re-)compile XPSI with the appropriate flags:
CC=<path/to/compiler/executable> python setup.py --help
CC=<path/to/compiler/executable> python setup.py install [--NumHot] [--NumElse] [--user]
This will install the numerical atmosphere for the hot regions and/or for the rest of the surface (elsewhere
). To (re-) install the default blackbody surface emission model, run the command again without the flags:
CC=<path/to/compiler/executable> python setup.py install [--user]
If you ever need to reinstall, first clean to recompile the C files:
rm -r build dist *egg* xpsi/*/*.c
Alternatively, to build X-PSI in-place:
CC=<path/to/compiler/executable> python setup.py build_ext -i
This will build extension modules in the source code directory. You must in
this case ensure that the source code directory is on your PYTHONPATH
environment variable, or inserted into sys.path
within a calling module.
Documentation¶
If you wish to compile the documentation you require Sphinx:
To install sphinx, run the following command in the X-PSI environment:
conda install sphinx=1.8.5
You then need the relevant extensions and need to ensure versions compatible with python2. Make sure to run each line individually and not copy-paste the whole block into your terminal for proper installation.
conda install -c conda-forge nbsphinx=0.5.1
conda install decorator=4.4.1
pip install sphinxcontrib-websupport==1.1.2
pip install sphinx_rtd_theme==0.4.3
Now the documentation can be compiled using:
cd xpsi/docs; [make clean;] make html
To rebuild the documentation after a change to source code docstrings:
[CC=<path/to/compiler/executable>] python setup.py install [--user]; cd docs; make clean; make html; cd ..
The .html
files can then found in xpsi/docs/build/html
, along with the
notebooks for the tutorials in this documentation. The .html
files can
naturally be opened in a browser, handily via a Jupyter session (this is
particularly useful if the edits are to tutorial notebooks).
Note that if you require links to the source code in the HTML files, you need
to ensure Sphinx imports the xpsi
package from the source directory
instead of from the ~/.local/lib
directory of the user. To enforce this,
insert the path to the source directory into sys.path
in the conf.py
script. Then make sure the extension modules are inside the source directory
– i.e., the package is built in-place (see above).
Note
To build the documentation, all modules need to be imported, and the dependencies that are not resolved will print warning messages.
Tips for installing on Mac OS¶
Most of the aforementioned instructions for linux are also applicable for Mac OS. Here we note some of the changes required.
After creating the environment using the basic_environment.yml
file, install xcode
or xcode tools
. Be mindful of the sequence of programs to be installed hereafter.
Use pip install
to download and install h5py
and emcee
(and maplotlib
, numpy
, scipy
and cython
if not using the basic_environment.yml
. You may use the file as a reference of the packages required).
On Mac OS, it’s preferable to use llvm clang
rather than gcc
. In order to do so, first install homebrew
:
/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
Install llvm
with homebrew, even if weird messages appear, saying llvm is already present in the Mac OS:
brew install llvm
Install GSL
(see above).
Install fortran
before MPI
.
If faced with issues when specifying or using gfortran (and it “does not pass simple tests”) specify the compiler as being gfortran in the mpif90
wrapper files and delete the files that were already in the build directory.
Once MPI
is installed,
export the following environment variables:
export LD_LIBRARY_PATH="/Users/<your_path>/openmpi/lib:$LD_LIBRARY_PATH"
export PATH=$PATH:/Users/<your_path>/mpi/bin/
export LDFLAGS="-L/usr/local/opt/llvm/lib"
export CPPFLAGS="-I/usr/local/opt/llvm/include"
export KMP_DUPLICATE_LIB_OK=TRUE
Consider adding these lines directly in your bashrc (or equivalent file for a different shell e.g. zshrc).
Install X-PSI
using:
CC=/usr/local/opt/llvm/bin/clang python setup.py install [--user]
If it gives problem, remove the tools
and surface_radiation_field
entires from setup.py
of X-PSI
.
The line in the setup.py file would then look like:
packages = ['xpsi', 'xpsi/PostProcessing']
If you encounter any problems with permissions when installing X-PSI, use the --user
option (This will install X-PSI globally, and not just within your virtual environment).
For compatibility, install the specified fgivenx
, GetDist
and nestcheck
(see above).
Tips for installing on Windows¶
Note
We do not recommend installing and running X-PSI on windows. However, if you must, this section details some of the relevant procedures.
X-PSI was successfully installed and run on Windows in the year 2020, at least for the purpose of likelihood functionality, using the following user-contributed procedure.
- Clone the X-PSI repository to a directory on your Windows computer (see above).
- Download Ubuntu for Windows.
- Install Python 2.7.
- Create a virtual Python environment in an Ubuntu shell.
- Install supporting packages
pip install matplotlib numpy cython scipy
followed bysudo apt-get install libgsl-dev
. - Ensure you are in the X-PSI directory and install X-PSI
CC=gcc python setup.py install
. - Install any missing packages that you need, e.g.,
pip install h5py
for post-processing functionality if you have posterior sample sets available. - Install Jupyter notebook using
pip install notebook
. - Start the kernel with the command
Jupyter notebook
.