Installing software

The INSY cluster runs the CentOS 7 Linux distribution, which provides the general Linux software. Most common software, including programming languages, libraries and development files for compiling your own software, is installed on the cluster. However, a not-so-common program that you need might not be installed. Or your research requires a state-of-the-art program that is not (yet) available as a package for CentOS. This page will help you get the software installed on the cluster.

On a cluster, it's important that software is available and identical on all nodes, both login and compute. For self-installed software, it's easier to install the software in one shared location than installing and maintaining the same software separately on every single node. You should therefore install your software on one of the network shares (your home folder or a bulk folder) that are accessible from all nodes. The installation recipe below will show you how to do this.

As a regular Linux user you don't have administrator rights. In Linux you are able to do your normal work, including installing software in a personal folder, without needing administrator rights. Consequently, you don't need (nor are you allowed) to use the sudo or su commands that are often shown in manuals, as the installation recipe below shows.

Important: although both Linux flavors Red Hat Enterprise Linux (RHEL, CentOS, Scientific Linux, Fedora) and Debian (Ubuntu) can run the same Linux software, they use completely different package systems for installing software. The available software, packages names and package versions might differ, and the package formats and package management tools are incompatible; it is not possible to install Ubuntu or Debian (.deb) packages in CentOS or use apt-get to install software. So when installing software, use a manual for CentOS, Red Hat or Fedora. If you can only find a manual for Ubuntu, you have to substitute the CentOS versions for any Ubuntu-specific packages or commands.

General software

To check if the program that you need is installed, you can simply try to start it:

$ python
Python 2.7.5 (default, Nov 20 2015, 02:00:19) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> quit()
$ python3
-bash: python3: command not found

Alternatively, you can try to locate the program or library, or check if the package is installed:

$ whereis python
python: /usr/bin/python /usr/bin/python2.7 /usr/lib/python2.7 /usr/lib64/python2.7
/usr/include/python2.7 /usr/share/man/man1/python.1.gz
$ rpm -q python
python-2.7.5-34.el7.x86_64
$ whereis python3
python3:
$ rpm -q python3
package python3 is not installed

When a program is not installed you should check if a package containing the program is available:

$ yum search python3
================= N/S matched: python3 ==================
python3.x86_64 : Version 3 of the Python programming language aka Python 3000
$ yum search python4
Warning: No matches found for: python4
No matches found

When a package is available you can simply request it's installation by sending an e-mail to the INSY cluster administrators.

Ready-to-use software

Some programs come as precompiled binaries or are written in a scripting language such as Perl, PHP, Python or shell script. Most of these programs don't actually need to be "installed" since you can simply run these programs directly (you may need to make the program executable first):

$ ./program
-bash: ./program: Permission denied
$ chmod +x program
$ ./program
Hello world!

Installing from source

When a package for your software is not available, you'll have to install the software yourself from the source. This is not too difficult following the recipe below.

Installation environment

When you are installing software for the very first time, you need to set up your environment. If you have already done this before , you can skip this and go directly to the installation instructions below.

To set up your environment, run:

$ curl -s http://insy.ewi.tudelft.nl/sites/default/files/bash_profile.txt >> ~/.bash_profile
$ source ~/.bash_profile
$ mkdir -p "$PREFIX"

This appends the following lines (download) to your ~/.bash_profile:

# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
	. ~/.bashrc
fi

# User specific environment and startup programs

export PREFIX="$HOME/.local"
export ACLOCAL_PATH="$PREFIX/share/aclocal${ACLOCAL_PATH:+:$ACLOCAL_PATH}"
export CPATH="$PREFIX/include${CPATH:+:$CPATH}"
export LD_LIBRARY_PATH="$PREFIX/lib64:$PREFIX/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"
export LIBRARY_PATH="$PREFIX/lib64:$PREFIX/lib${LIBRARY_PATH:+:$LIBRARY_PATH}"
export MANPATH="$PREFIX/share/man${MANPATH:+:$MANPATH}"
export PATH="$HOME/bin:$PREFIX/bin:$PATH"
export PERL5LIB="$PREFIX/lib64/perl5:$PREFIX/share/perl5${PERL5LIB:+:$PERL5LIB}"
export PKG_CONFIG_PATH="$PREFIX/lib64/pkgconfig:$PREFIX/share/pkgconfig${PKG_CONFIG_PATH:+:$PKG_CONFIG_PATH}"
export PYTHONPATH="$PREFIX/lib/python${PYTHONPATH:+:$PYTHONPATH}"

Note: if you already have some of these settings in your ~/.bash_profile (or elsewhere), you should combine them so they don't duplicate the paths.

The line export PREFIX="$HOME/.local" sets your software installation directory to /home/nfs/netid/.local (which is the default and accessible on all nodes). This is in your personal home directory where you have a default space quota of 2GB. If your programs will need more space, you need to instead use a larger share, for example bulk:

export PREFIX="/tudelft.net/staff-bulk/ewi/insy/group/$USER/.local"

The other variables will let you use your self-installed programs. You are now ready to install your software!

Installation recipe

Software installation usually just requires you to follow the general installation recipe described below, but you always need to consult the documentation for your software.

  1. Place the source of the software in a folder under /tmp:

    $ mkdir /tmp/$USER
    $ cd /tmp/$USER
    $ wget http://host/path/software.tar.gz
    $ tar -xzf software.tar.gz
              Or from github: git clone https://github.com/software
    $ cd software
    

    Note: .tgz is the same as .tar.gz, for .tar.bz2 files use tar -xjf software.tar.bz2.

  2. If the software provides a configure script, run it:

    $ ./configure --prefix="$PREFIX"
    

    If configure complains about missing software, you'll either have to install that software, tell configure where it is (--with-feature=path) or disable the feature (--disable-feature).

    If your software provides a CMakeLists.txt file, run cmake (note: the trailing two dots on the last line are needed exactly as shown):

    $ mkdir -p build
    $ cd build
    $ cmake -DCMAKE_INSTALL_PREFIX="$PREFIX" ..
    

    Again, if cmake complains about missing software, you'll either have to install that software or tell cmake where it is (-DCMAKE_SYSTEM_PREFIX_PATH="/usr/local;/usr;$PREFIX;path").

    If neither is provided, consult the documentation for dependencies and configuration (specifically for the installation directory).

    There is no point in continuing until all reported problems have been fixed.

  3. Compile the software:

    $ make
    

    If compilation is aborted due to an error, Google the error for possible solutions. Again, there is no point in continuing until all reported problems have been fixed.

  4. Install the software. When you used configure or cmake, you can simply run:

    $ make install
    

    When you used neither, you need to use:

    $ make prefix="$PREFIX" install
    
  5. Your software should now be ready to use, so check it:

    $ cd
    $ program
    
  6. When the program works, clean up /tmp/netid:

    $ rm -r /tmp/$USER
    

Python modules

After setting up the installation environment (above) you are also able to install Python modules by yourself, by using the --user option. The easiest way is when the module is available through pip:

$ pip search module
$ pip install --user module

When you only have the source code for the module, follow the installation instructions for the module, but make sure to use --user in the installation step:

$ python setup.py install --user