Hardware and software information

Hardware

All CPUs throughout the cluster are quad-core Intel 2.53GHz E5540 Xeon (Nehalem) CPUs. Each node has two of these, ie 8 cores.

The head node is called mosura, (12GB RAM, 4 CPU cores). You can ssh/rlogin into this from anywhere on the mountain (or via VPN). The compute nodes themselves can only be accessed from the head node, as they are on a private InfiniBand network.

Compute nodes

The small-memory nodes are mosura01 through mosura12, 8 CPU cores each, 6GB/core = 48GB total RAM The large-memory nodes are mosura13 through mosura16, 8 CPU cores each, 9GB/core = 72GB total RAM

Each node has a separate /export/scratch/ disk of several hundred GB.

Software

Openmpi 1.4 is installed and is in the default $PATH. The openmpi libraries and include files are in /usr/local/openmpi/{lib,include}.

HDF5 is installed in /usr/local/hdf (v1.8) and /usr/local/hdf5-1.6 (v1.6). Double check which version your code needs.

To run MPI jobs under ssh (the default), you'll need to configure ssh to be able to ssh login to the compute nodes without a password. This needs only to be done once. The steps to do this are:

  • login to the COALA head node
  • run ssh-keygen -t dsa, accept the default file locations and use an empty passphrase
  • run cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys2

Alternatively, you can run MPI jobs with rsh, by adding "-mca plm_rsh_agent /usr/bin/rsh" to your mpirun command line arguments.

There is no batch queuing system installed.

All the usual network disks (/home, /priv etc) are available on all the nodes. All the usual school linux software is available from /pkg/linux.

Compilers

  • The Intel C/C++ and Fortran compilers are installed. Run: source /pkg/linux/intel/composerxe-2011.3.174/bin/compilervars.csh intel64 to set up your environment for it (64bit). The compiler names are icc and ifort. If you need to use LAPACK/BLAS routines, link with -lmkl_intel_lp64 -lmkl_sequential -lmkl_core , which will provide highly optimised versions of these libraries.
  • The Sun Studio 12 Update 2 compilers are installed. These are useful for those porting code from Solaris systems, as the compiler flags are the same as the solaris compilers. They are good compilers and give reasonable performance. To use them you will need to add this to your .cshrc:

    set path = ( /pkg/linux/SS12/sunstudio12.2/bin $path )

    These compilers provide good performance on the COALA "Nehalem" cpus with these compiler optimisation flags:

    "-fast -xtarget=nehalem -xipo=2 -xvector=simd".

    If you need to use LAPACK/BLAS routines, with the Sun compilers you should link with -lsunperf, which will provide highly optimised versions of these routines.

  • gcc/gfortran 4.1.2 is installed too but is not likely to produce good performance.
  • Unfortunately, g77 is installed. Do not use it. Every other choice is better.