ARC: A Root Cluster for Research into Scalable Computer Systems

Useful Information

Pin threads to cores (only with Gnu compilers): export GOMP_CPU_AFFINITY="0-15:1", see GOMP_CPU_AFFINITY
More explicit thread pinning: use sched_setaffinity, see man sched_setaffinity and Take charge of processor affinity
Pin Open MPI tasks to cores (can be used in conjunction with OpenMP threads): Open MPI core pinning: mpirun -rankfile rankfile ...
Pin Open MPI tasks to cores (should NOT be used in conjunction with OpenMP threads): Open MPI core pinning: mpirun --bind-to-core ...
Note: MVAPICH2 binds threads automatically
Monitor thread/process pinning: ps -L -eo pid,tid,nlwp,tty,comm,psr
PVFS2 storage network (1Gbps on eth1): use compute-storage-0-XXX as hostname
(similarly login-storage-0-0, oss-storage-0-XXX, mds-storage-114)
IP over Infiniband (IPoIB): use compute-ibnet-0-XXX as hostname
SDP over Infiniband (faster than IPoIB): use compute-ibnet-0-XXX as hostname.
Then execute your a.out binary as: LD_PRELOAD=libsdp.so a.out ...
Or for with MPI: LD_PRELOAD=libsdp.so mpirun -mca btl_tcp_if_include ib0 -mca btl tcp,self env LD_PRELOAD=libsdp.so ... (works with MPI_THREAD_MULTIPLE)
not working: ~~all communication via TCP over Inifiniband with Open MPI:~~
- ~~sed 's/compute-/compute\-ibnet-/' $PBS_NODEFILE >hostfile~~
- ~~mpirun -mca btl_tcp_if_include ib0 -mca btl tcp,self -hostfile hostfile ...~~
MPI via TCP over Inifiniband, MPI runtime via TCP over Ethernet with Open MPI (works with MPI_THREAD_MULTIPLE):
- mpirun -mca btl_tcp_if_include ib0 -mca btl tcp,self ...
all communication via TCP over Ethernet with Open MPI (required for MPI_THREAD_MULTIPLE as Infiniband natively does not work with it):
- mpirun -mca btl_tcp_if_include br0 -mca btl tcp,self ...
Tripwire Reports
System Info
IB monitoring (user: monitor, password: monitor)

Additional Hardware

64 cores (128 hyper-threaded cores) on 4 compute nodes integrated by Advanced HPC. All machines are 2-way SMPs with Xeon Sandy Bridge EP E5-2650 processors with 8 cores per socket (16 cores per node, 32 hyper-threaded cores). E5-2650 Xeon Processor (single core):

64KB split (32KB+32KB) I+D L1 caches, 8-way associative, 64B/line (private)
256KB L2 cache, 8-way associative, 64B/line (private, non-inclusive)
20MB L3 cache, 20-way associative, 64B/line (shared, inclusive)
Data TLB0: 2-MB or 4-MB pages, 4-way associative, 32 entries
Data TLB: 4-KB Pages, 4-way set associative, 64 entries
Instruction TLB: 4-KB pages, 4-way set associative, 64 entries
L2 TLB: 1-MB, 4-way set associative, 64-byte line size
Shared 2nd-level TLB: 4 KB pages, 4-way set associative, 512 entries
800MHz-2GHz Core speed / 2.8GHz turbo
8GHz bus speed (QPI)
95 Watts peak power (0.6-1.35V)
32nm SOI CMOS fab
FCLGA2011 socket

Nodes:

Supermicro X9DRG-HF Motherboard with IPMI 2.0
Transport 1027G-TQF with 4 GPU slots + 1 half-size PCI-E slot
16GB DDR3 1600 ECC DRAM
nodes: compute-0-XXX, XXX=116..119 (1TB HDD)

32 cores (64 hyper-threaded cores) on 2 compute nodes integrated by Advanced HPC. All machines are 2-way SMPs with Ivy Bridge E5-2667v2 processors with 8 cores per socket (16 cores per node, 32 hyper-threaded cores). E5-2667v2 Xeon Processor (single core):

64KB split (32KB+32KB) I+D L1 caches, 8-way associative, 64B/line (private)
256KB L2 cache, 8-way associative, 64B/line (private, non-inclusive)
25MB L3 cache, 20-way associative, 64B/line (shared, inclusive)
Data TLB0: 2-MB or 4-MB pages, 4-way associative, 32 entries
Data TLB: 4-KB Pages, 4-way set associative, 64 entries
Instruction TLB: 4-KB pages, 4-way set associative, 64 entries
L2 TLB: 1-MB, 4-way set associative, 64-byte line size
Shared 2nd-level TLB: 4 KB pages, 4-way set associative, 512 entries
800MHz-3.3GHz Core speed / 4.0GHz turbo
8GHz bus speed (QPI)
130 Watts peak power (0.65-1.3V)
22nm SOI CMOS fab
FCLGA2011 socket

Nodes:

Supermicro X9DRG-HF Motherboard with IPMI 2.0
Transport 1027GR-TSF with 3 GPU slots + 1 half-size PCI-E slot
16GB DDR3 1600 ECC DRAM
nodes: compute-0-XXX, XXX=121..122 (1TB HDD)

64 cores (256 hyper-threaded cores) on 2 compute nodes Intel Xeon Phi X200 KNL (64 cores, 4X hyperthreaded)Ninja Developer platform (Supermicro SuperServer 5038K-i pedestal: liquid cooled, 96GB DRAM, 223.6GB Intel SSDSC2BB24, 4TB Hitachi HGST Ultrastar 7k6000 SAS 128MB cache 12Gbps, CentOS 7.2, Linux kernel 3.10.0-327.18.2, Intel® Parallel Studio XE Cluster Edition through the Developer Access Program, see slides, installed under /opt/intel/...

To access icc/ifort/openmpi/advixe-cl/amplxe-cl/inspxe-cl/etc.
export PATH=/opt/intel/bin:/opt/intel/compilers_and_libraries/linux/mpi/bin64:/usr/lib64/openmpi/bin:/opt/intel/advisor/bin64:/opt/intel/vtune_amplifier_xe/bin64:/opt/intel/inspector/bin64:$PATH
export LD_LIBRARY_PATH=/opt/intel/lib/mic:/opt/intel/lib/intel64:/opt/intel/compilers_and_libraries/linux/mpi/lib64:/usr/lib64/openmpi/lib:/opt/intel/advisor/lib64:/opt/intel/vtune_amplifier_xe/lib64:/opt/intel/inspector/lib64:$LD_LIBRARY_PATH
export MANPATH=/opt/intel/man/common:/opt/intel/compilers_and_libraries/linux/mpi/man:/opt/intel/advisor/man:/opt/intel/vtune_amplifier_xe/man:/opt/intel/inspector/man:$MANPATH
Also check other components under /opt/intel
nodes: compute-0-XXX, XXX=124..125 (use ssh -Y... for X11)