Install Applications and Libraries in the Home Directory

We are adding more content to this webpage. Let us know your thoughts via emailing to sys-help@loni.org

1. Introduction

Typically, a user has the permission to compile and install extra libraries and applications in his/her home, work or project directory, where there is enough space to store such tools and libraries. However, at first please check our blacklist within this webpage. If a library or an application on the blacklist was installed without permission, it may be harmful to the HPC envrionment and the user will be warned or even have his/her account disabled.

Please direct questions to sys-help@loni.org if you are unsure about whether an installation is appropriate or need help for the installation.

Back to top

2. Blacklist

2.1 glibc

We do not allow any users to install their version of glibc in their own directory on any HPC clusters, because glibc is integrated into the operating system and the installation will cause security issues. Therefore, we do not support using a self-installed version of glibc. If you need to link your package to glibc, please link it to the default system-wide installation of glibc; if you have source code, please recompile it using the system-wide installation of glibc; if only a binary is availabe, please specify what the code is, where it comes from and if it is licensed, then ask the developer to compile the code with the same glibc version as our system-wide installation of glibc.

Back to top

3. Application

3.1 FFTW

The latest official version of FFTW can be obtained at www.fftw.org. If you are still considering to use FFTW 2.x, please note that FFTW 2.x was last updated in 1999 and it is obsolete. So please install FFTW 3.x.

To install FFTW3, copy the FFTW3 tarball to the home directory and unpack it. Then rename the new directory to tmp and change to it.

$ tar -xvzf fftw-3.3.4.tar.gz
$ mv fftw-3.3.4 tmp
$ cd tmp

Next, in the configure step, specify the install destination with the --prefix option. Meanwhile, the customization of FFTW should be well checked during the configuration step (e.g. whether a serial or paralell version should be installed). For the full description of the FFTW, please carefully read the introduction at www.fftw.org/fftw2_doc/fftw.html. For example, the following configure program will install a parallel single-precision version of FFTW with shared libraries:

$ ./configure --prefix=$HOME/fftw-3.3.4 --enable-float --enable-shared --enable-mpi -mpicc --enable-sse2
      

Here the --enable-float flag produces a single-precision version of FFTW; --enable-shared creates shared libraries; --enable-mpi flag enables compliation and installation of FFTW MPI library with a specified compiler.The version of MPI compiler can be obtained with which command (which mpicc). The last --enable-sse2 flag is to compile FFTW to support SIMD (single instruction multiple data) instructions. This is asked by some self-installed software such as Gromacs for better performance when linking your self-installed FFTW.

After the configuration, use GNU make and make install to finish the installation.

   
$ make
$ make install
      

To testing FFTW for correctness, make check can be used to put the FFTW test programs through their paces. For instance, the following command line combines the error message (if any) with standard output, and then dumps the output to a log file named check.log. Failures are not necessarily problems as they might be caused by missing functionality, but a careful look at any reported discrepancies is required.

$ make check 2>&1 | tee check.log
      

Alternatively, examine the binaries in the fftw-3.3.4/bin with ldd command, as well as check if the library files in the fftw-3.3.4/lib has suitable extensions (.so for shared library; .a for static library).

If problems occured during configuration or compilation, use make distclean before trying again; this ensures that you don't have any stale files left over from previous compilation attempts.

Finally, if the installation looks good, add relative paths to the shell initialization script under home directory. For a bash shell, after openning the .bash_profile , type:

#added path to fftw
export PATH=$HOME/fftw-3.3.4/bin:$PATH
export LD_LIBRARY_PATH=$HOME/fftw-3.3.4/lib:$LD_LIBRARY_PATH
export LIBRARY_PATH=$HOME/fftw-3.3.4/lib:$LIBRARY_PATH
export PKG_CONFIG_PATH=$HOME/fftw-3.3.4/lib/pkgconfig:$PKG_CONFIG_PATH
#end 
      

Once finished, remember source this file or relogin to the cluster to take effect the change. The tmp directory may be deleted.

Back to top

3.2 R

The 2016 version of R has removed the versions of several compression libraries requried by the installastion that used to be included, such as zlib, bzip2xz, curl and pcre. The R package assumes those libraries are all up-to-date in the operating system. Unfortunately, on the LONI and LSU HPC clusters, which is still running RedHat 6, it is a serious problem as the required libraries are out of date. When configuring a R package released in 2016 or later, it will give error message like this:

   
 checking if zlib version >= 1.2.5... no
 checking whether zlib support suffices... configure: error: zlib library and headers are required
      

From the HPC administrator's point of view, we are unwilling to update the support libraries system-widely very often, because the defects (bugs) of the new libraries may be harmful to other existing software. But it is possible for the user to complile and install all support libraries (zlib, bzip2xz etc.) in the home directory without system-wide intervention. To do so, please refer to an article written by Prof. Johnson at here.

Alternatively, the versions of R released in 2015 can be installed without problem. The last release version of R in 2015 is 3.2.3, and this version is still compatible with most of the R packages. If Intel compiler is used, please note the default version (14.0.2) of Intel compiler does not support _Alignof in C11, which will fatally terminate the make process. Therefore, a newer version of Intel complier is required. Intel 15.0.0 on SuperMic and Intel 16.0.3 on QB2 have been used to compile R respectively, and the tests of installation were successful. Before the installation of R, unload the default Intel compiler and load the newer one with module:

   
$ module unload intel/14.0.2
$ module load intel/16.0.3
      

Accordign to an article at R-bloggers and other online resources, it is recommended to use Math Kernel Library (MKL) optimized for Intel processors, with performance far superior to traditional libraries, as well as to have the capabilities of multithreading.

To install R, copy the R tarball to the home directory and unpack it. Then rename the new directory to tmp and change to it.

$ tar -xvzf R-3.2.3.tar.gz
$ mv R-3.2.3 tmp
$ cd tmp
      

Specify the install destination (the home directory), the use of Intel compiler and MKL in the configuration:

fast="-ip -O3 -opt-mem-layout-trans=3 -xHost -mavx -fp-model precise"
./configure --prefix="$HOME/r-3.2.3" CC=icc CFLAGS="$fast -wd188" CXX=icpc \ CXXFLAGS="$fast" FC=ifort FCFLAGS="$fast" F77=ifort FFLAGS="$fast" \
 --with-blas='-mkl=parallel' --with-lapack
      

The last two options is for MKL support. The --with-blas option compiles a special version of BLAS library supported by MKL. If more precise control other than -mkl=parallel is desired, please read the MKL User's Guide or see the website of the Intel MKL Link Line Advisor. Some good examples of detailed --with-blas flag options can be found here, here, here and here. The --with-lapack option compiles a special version of LAPACK library supported by MKL.

Then use GNU make and make install to finish the installation.

   
$ make
$ make install
      

After make install, to check if MKL library was dynamically linked in the R, you may use ldd command to verify:

   
$ ldd $HOME/r-3.2.3/lib64/R/bin/exec/R
linux-vdso.so.1 =>  (0x00007fffd87ff000)
	libmkl_intel_lp64.so => /usr/local/compilers/Intel/cluster_studio_xe_2013.1.046/composer_xe_2013_sp1.2.144/mkl/lib/intel64/libmkl_intel_lp64.so (0x00002ba734fe5000)
......
      

In order to test R for correctness, make check can be used to put the R test programs through their paces.

$ make check
      

Finally, if the installation looks good, add relative paths to the shell initialization script under home directory. For a bash shell, after openning the .bash_profile , type:

#added path to R
export PATH=$HOME/r-3.2.3/bin:$PATH
#end 
      

Once finished, remember source this file or relogin to the cluster to take effect the change. The tmp directory may be deleted.

Back to top

4. Library

4.1 R

Installation

If a system-wide installed R is used, the default R library path is /home/packages/r/3.x.x/INTEL-14.0.2/lib64/R/library, which is not writable if a user do not have root privilege. To install a R package, the environment variable R_LIBS_USER needs to be pointed to a desired location in your own directory. For example, when using bash shell you would issue a command similar to the following:

$ export R_LIBS_USER=$HOME/packages/R/libraries
      

The above command line can be added to the default shell initialization environment by pasting the line to the script such as .bash_profile under your home directory. If the R is installed in your home directory, the above step is unnecessary but can be considered if you wish to have two directories for the default and new libraries respectively.

Type R to enter the console, then run R commands there.

 
$ R
(introduction section)
...
> 

To see libraries that R currently searching, use:

> .libPaths ()
      

Use install.packages("(package name)") function (double quotation is mandatory) to install package. For example, to install the package glmnet:

> install.packages ("glmnet",repos="http://cran.r-project.org")
      

Then select a CRAN mirror (e.g. 144 USA (TX)) to install the package glmnet.

Load, update and remove packages

When in the R console, the library() function checks all of the previously installed packages. the library()function can also be used to load a specified installed package:

> library ("glmnet")
      

Packages can be updated, as well as removed in the R session:

> update.packages ("glmnet")
#
> remove.packages ("glmnet")
      
Back to top

4.2 Perl

Installation

Perl modules can be treated as libraries. Each module comes with reusable codes. The Comprehensive Perl Archive Network (CPAN) has tons of modlues developed by the Perl community at www.cpan.org. Therefore, before settng out to write something serious, check CPAN first.

Perl modules can be installed manually, which includes:1. download the tarball and extract the content; 2. create a Makefile; 3. make, make test and make install. And "make Makefile.Pl prefix=" is required for users who don't have root privilege.

Alternatively, the module installation can be done with the module cpan, which provides a console to search, download and install modules and their dependencies automatically. Note the module cpan is not provided by the default Perl (5.10.1) on our supercomuter clusters, so a newer version of Perl has to be added by softenv or module to use cpan.

Once the right version of Perl (check our current versions and availability) has been loaded into the user environment, to install a module, for example, to install the module Inline you may type:

$ perl -MCPAN -e 'install Inline'
      

If this is your first time to use module cpan to install other Perl modules, after typing the command line above, it will appear an interactive dialogue:

Would you like to configure as much as possible automatically?
      

It is expecting you to type yes or no. Type yes to have the configuration automatically unless you are really knowing how to configure. Then the next question is:

To install modules, you need to configure a local Perl library directory or escalate your privileges. What approach do you want?  (Choose 'local::lib', 'sudo' or 'manual')
      

Type local::lib as you need to configure a local Perl library without root privilege. Then the next question is:

Would you like me to automatically choose some CPAN mirror sites for you? 
      

Type yes. The last question is:

Would you like me to append that to /home/ychen64/.bashrc now?
      

Again, yes. Then several lines aiming to change certain environment variables will be written to the .bashrc in your home directory to tell Perl where to install and load modules in the local Perl library. Remember to restart your command line shell (e.g. source your .bashrc if using bash ) if you wish to install another Perl module or load the newly installed module immediately, because your user environment hasn't been changed yet.

After the first use, no more setup is required in your next Perl module installations with cpan. For example, after installed the module Inline, the module Inline::Files can be installed by simply typing:

$ perl -MCPAN -e 'install Inline::Files'
      

If everything goes well, it will be installed automatically in seconds. If you got permission denied error, most likely your forgot to source your .bashrc and/or did not logout and login to execute the new lines written in .bashrc.

Locate module hiding places

When Perl is installed, it creates a list, @INC, with the names of all the includes directories. The contents can be reviewed from the command line:

$ perl -le 'print foreach @INC'
/home/ychen64/perl5/lib/perl5/5.16.3/x86_64-linux-thread-multi
/home/ychen64/perl5/lib/perl5/5.16.3
/home/ychen64/perl5/lib/perl5/x86_64-linux-thread-multi
/home/ychen64/perl5/lib/perl5
/usr/local/packages/perl/5.16.3/INTEL-14.0.2/lib/site_perl/5.16.3/x86_64-linux-thread-multi
/usr/local/packages/perl/5.16.3/INTEL-14.0.2/lib/site_perl/5.16.3
/usr/local/packages/perl/5.16.3/INTEL-14.0.2/lib/5.16.3/x86_64-linux-thread-multi
/usr/local/packages/perl/5.16.3/INTEL-14.0.2/lib/5.16.3
      

The /home/ychen64/perl5/lib/perl5 is the local directory saving the installed modules.

Load module

Users have multiple ways of telling Perl where to search for their locally installed modules. For example, PERL5LIB is an envrionment variable saving the name of local Perl directories. It is searched before the contents of @INC. Setting up PERL5LIB will help Perl the search the locally installed modules.

If the module is installed by cpan, PERL5LIB has already been setup in .bashrc. If the module is manually installed (i.e. without cpan), use

$ export PERL5LIB=/your_path_to_the_perl_lib
      

to specify the local directory saving the installed modules.

Alternatively, in the Perl script, the location can be specified by:

#! /usr/bin/perl
use lib "/your_path_to_the_perl_lib"
      

After letting Perl know the local library path, the installed module can be loaded in script. For example, to load Inline::Files, add the following line into the Perl script:

use Inline::Files
      
Start over cpan

Suppose now you have more confidence to manually configure cpan settings by saying "no" in the first question in the cpan dialogue mentioned above (e.g. you wish to assign your perferred installation directory rather than the default /home/your_username/perl5/lib/perl5), you may think changing the environment viriable setting, which is written in your .bashrc or the MyConfig.pm file in ~/.local/share/.cpan/CPAN. But both changes are limited and requires you remember those variable definitions. On the other hand, in the cpan dialogue, you will be told how to setup those environment variables interactively. Thus, I recommend that you make use of the cpan dialogue to configure your setting.

However, you will find the cpan configuration dialogue only appears at the first time you use it. To make the dialogue appear again, you need to start over cpan by deleting the related files of cpan in the directory .local/share in your home directory.

Back to top

4.3 Python

Installation

Python modules can be treated as libraries. Each module comes with reusable codes. On our supercomputer clusters several well-known Pyhon modules such as NumPy have been installed system-widely (globally). However, if you prefer to use modules that haven't been installed, or wish to use a certain version of the module rather than the system-wide version on the cluster, you may find the default Python module path is not writable for module installation as you do not have root privilege.

Therefore, you may consider to intall your desired Python modules in your home directory, and the module installation can be done using a Python module called pip. Note the module pip is not provided by the default Python (2.6) on some of our clusters, so a newer version of Python (check our current versions and availability) has to be loaded by softenv or module to use pip. After loading Python, make sure pip is available by some commands such as which pip or pip --version.

Once pip is available, it is important to know the directory saving your local-installed modules. The default location is explained in the Python documentation for the site.USER_SITE and site.USER_BASE variables. The default value of the former is ~/.local/lib/pythonX.Y/site-packages which can be modified by the PYTHONPATH environment variable; while the default value of latter is ~/.local variable which can be modified by the PYTHONUSERBASE enrironment variabe.

Therefore, by updating PYTHONPATH and PYTHONUSERBASE at the same time, you may specify your desired install location in your home directory. For example, if using Python 2.7, to specify a module install location ~/packages/python, following command (when using bash) should be used:

 $ export PYTHONPATH=$HOME/packages/python/lib/python2.7/site-packages:$PYTHONPATH
 $ export PYTHONUSERBASE=$HOME/packages/python
      

The above command lines should be added to the default shell initialization environment by pasting the line to the script such as .bash_profile under your home directory. Even you choose to use the default ~/.local/lib/pythonX.Y/site-packages, you should add export PYTHONPATH=$HOME/.local/lib/python2.7/site-packages:$PYTHONPATH to your default shell initialization environment (.bash_profile). This is because when the location of globally installed modules is also in the PYTHONPATH, sometimes it has precedence over the local path. If that is the case, Python will load the globally installed modules instead of your local modules. In other words, even you have already installed your desired module locally, Python will choose to load the globally installed module, which is the one you don't like, over the local one if you did not add the export PYTHONPATH=$HOME/.local/lib/python2.7/site-packages:$PYTHONPATH to your .bash_profile. Thus, please be careful and make sure the PYTHONPATH contains your local module installation path and it is before the system-wide path.

Once the pip is setup, the newest version of pip should be installed:

 $ pip install --user --upgrade pip
      

The --user option can turn on the site.USER_SITE variable and ask pip to install any new modules (including itself here) into your home directory.

You may wish to use pip to list all of the current installed modules, or just list one or several modules. Using the following commands respectively:

 $ pip list      # list all moudles available
 $ pip show module_name # list a module called "module_name" 
      

Now you may have a clear view of the modules currently installed. If you cannot find the module you need, or you wish to use a different version, you may use one of the following methods, and remember to include the --user option:

 $ pip install --user module_name           # latest version
 $ pip install --user module_name==x.y.z    # specific version x.y.z
 $ pip install --user 'module_name>=x.y.z'  # minimum version
      

Sometimes after typing pip install module_name, you will got the following message:

 $ pip install --user module_name
Requirement already satisfied: module_name in ./.local/lib/python2.7/site-packages
Requirement already satisfied: python-dateutil in ./.local/lib/python2.7/site-packages
...
...
      

And the installation will be terminated. This is because Python and pip apply a mechanism to manage the modules install. With the mechanism, pip sometimes will consider the globally installed modules already satisfy the installation requirements and does nothing, and reports that "requirement is satisfied". If you need more precise control, you may take a look a the pip user guide about the requirement files. If not, just use the second method (pip install module_name==x.y.z) to specify the module version. If still got this issue, consider to add --upgrade option.

Listing modules

pip list and pip show module_name can be used to list all of the current installed modules and list one or several modules, respectively. For more information, please see pip list and pip show reference pages.

Upgrade/downgrade modules

Add --upgrade option to upgrade module. The module may be downgraded with the same option. In the following the second method is preferred as it is more precise and can be used to downgrade modules:

 $ pip install --upgrade --user module_name           # latest version
 $ pip install --upgrade --user module_name==x.y.z    # specific version x.y.z
      
Uninstallation

Simply use:

 $ pip uninstall module_name
      
Back to top