Opt⁶⁴ Cluster ⇔ 16+1 nodes, 32 dies, 64 cores, 64 bit (x86_64)

Hardware

All machines are 2-way SMPs with dual-core AMD Opteron 265 processors, per core:

64KB split I+D L1 caches, 2-way associative, 64B/line
1MB shared L2 cache, 16-way associative, 64B/line

Tyan Thunder K8SD Pro Motherboard (2882-D)
Transport GX28 (B2882)
Tyan M3289 IPMI
server: opt (has 750GB RAID5 array NFS mounted to nodes, UPS)
nodes: optXX, XX=00..17
experimental nodes: opt16, opt17

Software

All software is 64 bit unless marked otherwise.

Monitor the System Status with Ganglia
Monitor the Server Room Temperature (should be < 80F)
htop to monitor per-processor activity on a node
Fedora Core 13 Linux x86_64
gcc/gfortran (C compiler)
gcc34/g77 (32 bit C and Fortran compilers)
MPICH2 (also MPI API)
locally: man MPI (after adding export MANPATH="$MANPATH:/usr/share/man/mpich2")
OpenMPI
OpenMP v2.5 (via gcc)
ATLAS (BLAS)
locally: /usr/share/doc/atlas-devel-3.6.0/doc
PAPI4 (using perfctr)
locally: man PAPI

Reservation

Before you use any node, check the calendar if someone needs exclusive access to some nodes. Do NOT use such nodes. Reserve your own nodes if you need access. (Password obtained from Operating Systems Research group.)

Access

Request a user id from Vince Freeh (server "long") or Frank Mueller (server "opt"). Please indicate your unity ID and student ID.
Accounts on long will differ in name from your unity ID, accounts on opt use your unity ID but do not share any unity file space.
Install ~/.ssh/config on your local Linux machine. If you're not using Linux, skip this step.
On Linux, ssh into any optXX machine (XX=00..17).
```
ssh <your-username›@optXX
```
Your initial password is your student ID.
On windows, use your favorite ssh client to log in to opt.csc.ncsu.edu port 22XX (X=00..17). This will log you in to one of the nodes (same as above).
Please change your password. If you got your account through Vincent Free, use one of his servers; otherwise ssh to one of the optXX nodes. Then change you password:
```
yppasswd
```
The new password will be effective no later than at the full hour.

Using OpenMP

The "#pragma omp" directive in C programs works.
```
gcc -fopenmp -o fn fn.c
```
Gcc 4.1 for FC5 is back-patched with the 4.2 OpenMP support. It may not be the fastest but it works.

Using PAPI

Compile:
```
gcc -o fn fn.c -lpapi
```
Make sure your LD_LIBRARY_PATH is set. See below.

Interactively running MPI programs with MPICH

Create the file ~/.rhosts and set the access rights:
```
chmod 600 .rhosts
```
Create a file ~/.mpd.conf and set the access rights:
```
chmod 600 .mpd.conf
```

Append to your file ~/.bashrc:

export PATH=".:~/bin:/usr/bin:/usr/local/bin:/usr/lib64/mpich2/bin:$PATH"
export LD_LIBRARY_PATH="/usr/local/lib64:/usr/local/lib:/usr/lib64/mpich2/lib:$LD_LIBRARY_PATH"

Log out and back in to optXX to activate the new settings.

Check if MPD is running:
```
mpdtrace
```
This should return a list of available nodes to run MPI jobs on. If not, then first start the user-level MPD (see below)

Compile MPI programs:

mpicc -O3 -o pi pi.c

If you're using BLAS/ATLAS:

mpicc -O3 -o pi pi.c -L/usr/lib64/atlas -lcblas -latlas

Execute the program on 2 processors (using MPICH2):
```
mpirun -np 2 pi
```
Try again with a different number of processors.
Use the new MPICH2 run/execute interface (more options, see man page):
```
mpiexec -n 2 pi
```
Try again with a different number of processors.

User-level MPD

If mpdtrace is not running yet, the start your MPD ring:

Figure out which nodes are alive. You can either Monitor the System Status with Ganglia or do this: ping optXX for XX=00..17. Still, they may not work properly. Try: rsh optXX pwd -- maybe working?
Create a file like ~/mpd.hosts but only include the nodes that are working!
If you have 2 working nodes and you are logged in to optYY, issue:
```
mpdboot --ifhn=optmpiYY -n 2 -r rsh --ncpus=4
```
You can now run MPI jobs (see mpirun/mpiexec above).
Inquire about running jobs:
```
mpdlistjobs
```
Stop MPD daemons:
```
mpdallexit
```
Clean up MPD daemons and files:
```
mpdcleanup
```
Vary you mpdboot parameters. In the above case, you'll run 4 jobs per node (since we have 2-way SMPs, each with a dual core, i.e., 4 processor cores per node). Check out the man pages. Examples below:
1 processor per node, 18 nodes, all in 1 mpd ring:
```
mpdboot --ifhn=optmpiYY -n 18 -r rsh --ncpus=1 --maxbranch=20 --verbose -d
```
More cleanup: sometimes mpdboot does not work because of left-over processes; issue:
```
rsh optYY killall -9 python2.6
rsh optYY mpdcleanup
```

Interactively running MPI programs with OpenMPI

add to your ~/.bashrc (or just enter if you don't want to make it permanent):

export PATH="/usr/lib64/openmpi/bin:$PATH"
export LD_LIBRARY_PATH="/usr/lib64/openmpi/lib"
module load openmpi-x86_64

Compile MPI programs:
```
mpicc -O3 -o pi pi.c
```
create .rhosts (and optionally a machinefile w/ valid hosts) as above

Execute the program on 2 processors (using OpenMPI, exclude eth0 interface):

mpirun -mca plm_rsh_agent rsh -mca btl_tcp_if_exclude lo,eth0,virbr0 -machinefile machinefile -np 2 pi2

To run on 4 processors per node, modify you machinefile:
```
opt00 slots=4 max_slots=4
opt01 slots=4 max_slots=4
```

MPI batch job submission

To reduce the per-node activity and avoid competition of jobs for resources, larger MPI jobs should be submitted as batch jobs. They can also be scheduled at certain times of the day (reserved in the calendar).

Batch submission is realized via Torque over OpenPBS (coming later: Maui Cluster Scheduler).

On opt, issue:

submit: qsub ...
check: qstat
remove: qdel ...
documentation: "man pbs" and Torque link above

Power monitoring

Advanced topics

For all other topics, access is restricted. Request a root password. Also, read this documentation, which is only accessible from selected NCSU labs.

This applies to:

IPMI
diskless booting
disk imaging
booting your own kernel
booting xen (non-root access in the future)
installing your own OS

Known Problems

Please let us know if you have solutions to these problems.

Xen kernel incompatible with perfctr and our powernow patch

References:

A User's Guide to MPI by Peter Pacheco
Debugging: Gdb does not work with MPI, and we don't have totalview (an MPI-aware debugger). Hence, you are down to good old printf debugging, unfortunately.
Linux Cluster Info (add cluster tools to the wish list)

Opt64 Cluster ⇔ 16+1 nodes, 32 dies, 64 cores, 64 bit (x86_64)

Hardware