HISO Cluster Documentation

Cluster Name

The name of the cluster (following the "place names in Alberta North of Calgary" tradition) is Heisler (similar to HISO). There is only one of the nodes on our network--heisler (or if you are inside the cluster Heisler-gw is slightly more efficient). Heisler has identical hardware to the computational nodes, but I suggest that it not be used too heavily as it also acts as a relay for all the network traffic into and out of the cluster. The cluster nodes are named: heisler-1, heisler-2, ...heisler-15.

Hardware Description

The cluster consists of 16 nodes (15 computational plus the head node) interconnected with a giga-bit Ethernet. Each node is identical and has the following resources:

Dual AMD Opteron 248 64-bit CPU's. The CPU's run at 2.2 GHz, dissipate 89w, have a 800 MHz HyperTransport bus, and have a 1 MB L2 cache. The SPEC.org published performance measures of this system are: SPECint2000=1229 SPECfp2000=1553, SPECint_rate2000=29.3, and SPECfp_rate2000=35.8 (all numbers listed are the "Base" ratings, not the "Peak"). More information about the AMD Opteron's can be found on AMD's web site.
5GB of memory. There are 2.5GB on each processor, although the HyperTransport Bus makes the two memory banks look like one memory space that both CPU's can access.
two 80GB IDE disks. As usual, the system is on /dev/hda1. There is 10GB of swap on the hda disk, and then the remainder (about 60GB) of it and the second disk (/dev/hdb) are appended together to make a large (127GB) /usr/scratch space. As is the norm, /usr/scratch is writable by any user, but only they can delete their files (unless they give write permission on the file and directory to others). The scratch "disk" is not backed up--it is intended for transient files that should be local to reduce network traffic. Please be kind to your colleagues (and the systems administrators) and delete your files when you are finished using them.
Giga-bit Ethernet.

Cluster Software Systems Information

Here are a few notes about the software systems configuration on the Heisler computational nodes.

Access

The only access to the cluster nodes is through the head-node:
The HISO Faculty Group (Amaral, Mueller, GHLin, ...) are the authority for who is allowed access to the computational nodes.
Access is controlled by having a separate "password" file from the Department. The cluster is in a different NIS domain from the Department. When a person is allowed on the cluster their "password" entry is copied from the Department NIS to the cluster NIS. This means that you will have the same password both places. If you change your password on the Department domain though the change will not be propagated to the cluster (and likely there currently is no way for user's to change their password on the cluster (except by asking the System Administrators to copy the Departmental record to the cluster domain).
The user's should see all of their normal files when on the cluster. The Network Address Translator (NAT) is set up to essentially be transparent (more about this later).

Network Address Translator

Programs cannot generally connect to servers on the internal network. There are no proxies in place to assist this. Programs running on the cluster nodes can connect to services on other computers in the Department (and the world). This means that the following work:
- Printing through "paf", lpr, etc.
- X sessions initiated from the inside of the cluster. For instance after doing "xhost +heisler" on your local workstation you can be logged onto one of the nodes and do "DISPLAY=your_computer:0.0; export DISPLAY" and then can start X applications (like 'xterm') and they will open a window on your workstation.
- Mail does NOT work (I have not investigated to see if it could be made to work--not high enough on my priority list).

Operating System

The cluster nodes were built from "Red Hat Enterprise Linux WS release 3 (Taroon Update 3)" using "Kernel 2.6.8.1 on a 2-processor x86_64".
This is a 64-bit operating system. This means that most of the utilities run in 64-bit space.
One notable implication of this is that the compilers generate 64-bit code. For instance:
main(){printf("Size of pointer is %d-bytes\n", sizeof (int *));}
produces:
Size of pointer is 8-bytes
This may not be what your code expects. If you have 32-bit dependencies you might be able to avoid them with a -m32 flag on gcc, but it can be tricky to use the right libraries (there are actually two library stacks--{/lib,/usr/lib} and {/lib64,/usr/lib64}).
If all else fails then you can compile your code on a 32-bit computer and just run the binary on the cluster. The AMD Opteron architecture and the Linux system fully support doing this.
The Intel C compiler (icc) is installed on all of the cluster nodes. It is not 64-bit aware.

Cluster Scheduling and Status

Currently there is no scheduler implemented on the cluster. The current scheduling model is one of "be kind to your colleagues":

Do not run unnecessary jobs.
Monitor the jobs that you are running.
Look before you leap--don't start up jobs on heavily used nodes.
Talk to your colleagues to understand each others needs and decide what resources you are all going to use.
Don't run several jobs in parallel on a node when they could be run sequentially just as well (or better--there is O/S overhead in doing context switches).

To facilitate the load balancing and monitoring two sub-systems have been installed:

"ruptime" (and "rwho") are a command line, text based, system that will give a one line per node summary of the load. If you run the "ruptime" command on Heisler or any of the interior nodes it will report the cluster data. If you ran it on other computers in the Department it will show all of the other hosts, but not the cluster nodes.
Ganglia is a web based tool to look at an overview of the cluster usage, or several measures of an individual node. The Ganglia view of the Heisler cluster is at http://www.cs.ualberta.ca/~steve/Heisler/.

Where to go for help

If you want an ID on the cluster then ask the HISO administrator (currently Dr. M. Mueller, email=mmueller) to approve the application. If they approve the request then they will send a note to Gordon Atwood and/or Steve Sutphen to add the ID to the cluster password file.

For other problems or requests send a note to "steve" and/or "gordon" (currently Steve is more familiar with the system, but that will be changing over time).

University of Alberta

Computing Science

Steve Sutphen