ARC: A Root Cluster for Research into Scalable Computer Systems

module switch mvapich2 openmpi4

Running MPI Programs with Open MPI and gcc/g++/gfortran (Alternative)

• Compile MPI programs:

  mpicc -O3 -o pi pi.c
  mpic++ -O3 -o pi pi.cpp
  mpifort -O3 -o pi pi.f
  

• Execute the program on 2 processors (using Open MPI):

  prun ./pi
  

• Execute the program on 32 (virtual) processors using 16 (physical) cores (using Open MPI):

  mpirun --oversubscribe -np 32 ./pi
  

Using the NVHPC/PGI compilers (V23.7 for CUDA 12.3)

(includes OpenMP and CUDA support via pragmas, even for Fortran)

• Issue

  module unload cuda
  module load nvhpc
  

  • For Fortran 77, use: pgf77 -V x.f
  • For Fortran 90, use: pgf90 -V x.f
  • For Fortran 95, use: pgf95 -V x.f
  • For C++, use: pgc++ -V x.c
  • For ANSI C, use: pgcc -V x.c

  • For more compile output, add option: -Minfo=all
  • For AMD 64-bit, add option: -tp=barcelona-64
  • For OpenMP, add option: -mp
  • For OpenACC/CUDA (default: 12.3), add options: -acc
  • For OpenACC/CUDA 12.3: -acc -Mcuda=cuda12.3,rdc
  • For OpenACC/CUDA on specific GPUs, run: pgaccelinfo, then use the respective -ta option in the output for compilation, e.g., -acc -ta=tesla,cc20
    • with filename.f: supports Fortran ACC pragmas (for CUDA), e.g., !$acc parallel
    • with filename.c: supports C ACC pragmas (for CUDA), e.g., #pragma acc parallel
• Slides and excercises on MPI+GPU programming with CUDA and OpenACC
• NVHPC/PGI Documentation
  • PGI Accelerator Quick Reference
  • Full OpenACC Support since V13.4
• OpenAcc Documentation
  • OpenAcc Quick Reference
  • OpenAcc V2.0
• NVHPCPGI+Open MPI (loaded with nvhpc):

  module unload openmpi4
  module load nvhpc
  #compile for C (similar for C++/Fortran)
  mpicc ...
  #use mpirun or prun to execute
  prun ./a.out
  mpirun ./a.out
  

  Notice that this OpenMPI version has support for CUDA pointers, RDMA, and GPU Direct

• NVHPC/PGI+MVAPICH: not supported

Dynamic Voltage and Frequency Scaling (DVFS)

• Change the frequency/voltage of a core to save energy (without any of with minor loss of performance, depending on how memory-bound an application is)
• Use cpupower and its utilities to change processor frequencies. Notice: When hyperthreading (ht) is off, you can change one core at a time; but when ht if on, you will need to change correspodning pairs of cores, e.g., (0,16) or (1,17) etc. for 32 visible cores (adapt as needed).
• Example 1 for core 0 without hyperthreading (requires sudo rights):
  • grep " ht " /proc/cpuinfo #should not produce any output; o/w see example 2
  • cpupower frequency-info
  • sudo cpupower -c 0 frequency-set -f 1500Mhz #set to userspace 1.5GHz
  • watch cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq #should be nearly constant
  • cpupower frequency-info
  • sudo cpupower frequency-set -g ondemand #revert to original settings
• Example 2 for core (0,16) pair with hyperthreading (requires sudo rights):
  • grep " ht " /proc/cpuinfo #should produce output
  • cpupower frequency-info
  • sudo cpupower -c 0 frequency-set -f 1500Mhz #set to userspace 1.5GHz
  • sudo cpupower -c 16 frequency-set -f 1500Mhz #set to userspace 1.5GHz
  • watch cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq #should be nearly constant
  • cpupower frequency-info
  • sudo cpupower frequency-set -g ondemand #revert to original settings

Power monitoring

• export PATH=".:~/bin:/usr/local/bin:/usr/bin:$PATH"
• export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/local/lib"
• for serial meters, use:

  mlogger -p 0 -o

• for USB meters, use:

  wattsup ttyUSB0 watts 

• Linux support for Watts Up serial version
  • man page / Power Aware Tools
  • Watts Up Pro Manual (serial)
  • Watts Up Pro Data Protocol (serial)
• Linux support for Watts Up USB version
  • Watts Up Pro Manual (USB) (local copy)

Virtualization

• OpenHPC-specific documatention for Charliecloud (read this first!)
• Charliecloud
  • module load charliecloud
• Singularity
  • module load singularity

Storage: BeeGFS and Burst Buffers

• BeeGFS /mnt/beegfs (for large data, slow)
  • cd /mnt/beegfs #to access it from compute nodes
  • mkdir $USER #to create your subdirectory (only needs to be done once)
  • chmod 700 $USER #to ensure others cannot access you data (only done once)
  • cd $USER #go to directory where you should place your large files
  • about 145TB of storage over 16 servers (9TB each)
  • via 100Gpbs IB switch RDMA connection (ib0)
  • Not protected by RAID, not backed up!
• Burst Buffers /mnt/local (for node-local storage, fast)
  • cd /mnt/local #to access it from compute nodes (uses some local SSD)
  • mkdir $USER #to create your subdirectory (only needs to be done once)
  • chmod 700 $USER #to ensure others cannot access you data (only done once)
  • cd $USER #go to directory where you should place your node-local files

PAPI

• module load papi
• Reads hardware performance counters
• Check supported counters: papi_avail
• Edit your source file to define performance counter events, read them and then print or process them, see PAPI API
• Add to the Makefile compile options: -I${PAPI_INC}
• Add to the Makefile linker options: -L${PAPI_LIB} -lpapi

likwid

• module load likwid
• Pins threads to specific cores, avoids Linux-based thread migration and may increase NUMA performance, see likwid project for a complete list of tools (power, pinning etc.)
• print NUMA core topology: likwid-topology -c -g
• Use likwid-pin to pin threads to specific cores
  • Example: likwid-pin myapp
  • Example: mpirun -np 2 /usr/local/bin/likwid-pin ./myapp
• Use likwid-perfctr or likwid-mpirun mearure performance counters, optionally with pinned threads
• Others: likwid-mpirun, likwid-powermeter, likwid-setfreq, ...

Python

• python/python3 (version 3.6), pip/pip3 install -U --user...
• python3.8 (version 3.8), pip3.8 install -U --user...
• python3.9 (version 3.9), pip3.9 install -U --user...
• mpi4py: Due to OpenHPC packing problem, currently requires setting below instead of pmix/ucx, use OpenMPI4 and

      export OMPI_MCA_pml="^ucx"
      export OMPI_MCA_btl_openib_if_include="mlx5_0:1"
      mpirun python3.6 your-code.py
    

• Option 1: Install hadoop in your directory. Create a subdirectory. From that subdirectory, run:

  salloc -N 4
  source hadoop-setup.sh
          

  Every time you do a new salloc, you should re-run the hadoop-setup script! Always make sure to use source to run it, otherwise the environment variables that the script sets will not get picked up by your current shell.

  Inspect the script's output. You should see the Hadoop components being started, and the HDFS filesystem will be formatted. You can check with jps to see which components are running, they should be NodeManager, SecondaryNameNode, DataNode, ResourceManager, NameNode (and Jps). On any other node of your allocation, it would just be NodeManager, DataNode (and Jps), you can check with ssh cXX jps (where XX depends on the nodes you have in echo $SLURM_NODELIST).

  If you do NOT see these components active, then something went wrong. (There is also a log in logs/main.log you can check, lots of log messages.) To check to see if everything was set up correctly, run:

  hdfs dfs -ls /user

  This should print out something like:

  Found 1 items
  drwxr-xr-x   - UNITYID supergroup          0 TIMESTAMP /user/UNITYID
          

  The above hadoop-setup script creates a "hadoop" directory. It then links common hadoop files to this directory and sets some environment variables. Next, it creates some personalized files in the hadoop/etc/hadoop directory, which contain your unity ID and the nodes you currently have reserved (this is why you have to re-run the script for every new salloc). These files tell Hadoop where to store the Hadoop Distributed File System (HDFS). Temporary folders on each node are also created to hold the HDFS data. Finally, we configure a blank HDFS (starting the NameNode with -format argument), start the HDFS (starting the name/data/secondary nodes), and create a /user/UNITY-ID directory inside the HDFS (the one from the hdfs dfs -ls command).

  Before running YOUR code, you must copy the input to the HDFS:

  hdfs dfs -put input0 /user/$USER/input0

  hdfs dfs -put input1 /user/$USER/input1

  Compile and run YOUR code:

  javac YOUR.java
  jar cf YOUR.jar YOUR*.class
  hadoop jar YOUR.jar YOUR input0 input1 &> hadoop_output.txt
  rm -rf output*
  hdfs dfs -get /user/$USER/output* .
          

  The hadoop_output.txt file will contain the hadoop output. Inspect this output to find any runtime errors/exceptions. As you are developing your code, you can repeatedly run the above five commands to compile/run your code and get the output.

  When you are done, before releasing your reserved nodes, run the following to shut down the hdfs file system:

  hadoop/sbin/stop-yarn.sh

  hadoop/sbin/stop-dfs.sh

• Spark uses HDFS as well, we need to set it up in a similar fashion, except some extra variables for Spark:

  salloc -N4
  tar xvf YOUR.tar
  cd YOUR
  source spark-hadoop-setup.sh
  hdfs dfs -put input /user/UNITYID/input
              

  javac YOUR.java
  jar cf YOUR.jar YOUR*.class
  spark-submit --class YOUR YOUR.jar input &> spark_output.txt
  grep -v '^24\|^(\|^-' spark_output.txt > output.txt
  diff -s solution.txt output.txt
  

  When you are done, before releasing your reserved nodes, run the following to shut down the hdfs file system:

  hadoop/sbin/stop-dfs.sh

• Option 2: module load magpie
  documentation

Tensorflow

• Tensorflow
• Option 1: install locally
  • python3 -m pip install --upgrade --user pip #to upgrade pip
  • pip3 install tensorflow
• Option 2: see OpenHPC Exercise 4 -- Tensorflow under Horovod with Charliecloud
• for jupyter-notebook to work, issue

  pip3 install jupyter seaborn pydot pydotplus graphviz -U --user
  #set a password for your sessions (for security!!)
  jupyter notebook password
  #start the server
  jupyter-notebook --NotebookApp.token='' --no-browser --ip=cXX
  #from your VPN/campus machine, assuming a port 8888 in the printed URL, issue:
  ssh <your-unity-id>@arc.csc.ncsu.edu -L 8889:cXX:8888
  #point your local browser at http://localhost:8889 and enter the password
      

  • Notice: Anyone on ARC can now connect to cXX:8888, not just you, and potentially write to your notebook, open other files in this directory and subdirectories!
  • Either use passwords (see above) or read about secure authentification for jupyter-nookbook for stronger protection.

PyTorch

• PyTorch
• pip3.8 install -U --user torch torchvision torchaudio
• python3.8

  import torch
  print(torch.__version__)
  torch.cuda.is_available()
  print(torch.cuda.current_device())
  torch.cuda.get_device_name(0)
  print(torch._C._cuda_getCompiledVersion())
  print(torch.rand(2,3).cuda()) #fails on sm 3.5 GPUs or earlier
  

• no support for older GPUs, error message: "The NVIDIA driver on your system is too old". (You could build an older version from source, but we don't bother.)

Other Packges

  module avail # show which modules are available
  module load X
  export |grep X #shows what has been defined
  gcc/mpicc -I${X_INC} -L{X_LIB} -lx #for a library
  ./X #for a tool/program, may be some variant of 'X' depending on toolkit
  module switch X Y #for mutually exclusive modules if X is already loaded
  module unload X
  module info #learn how to use modules

------------------- /opt/ohpc/pub/moduledeps/gnu12-openmpi4 --------------------
   adios/1.13.1        netcdf-fortran/4.6.0    scalapack/2.2.0
   boost/1.80.0        netcdf/4.9.0            scalasca/2.5
   dimemas/5.4.2       omb/6.1                 scorep/7.1
   extrae/3.8.3        opencoarrays/2.10.0     sionlib/1.7.7
   fftw/3.3.10         petsc/3.18.1            slepc/3.18.0
   hypre/2.18.1        phdf5/1.10.8            superlu_dist/6.4.0
   imb/2021.3          pnetcdf/1.12.3          tau/2.31.1
   mfem/4.4            ptscotch/7.0.1          trilinos/13.4.0
   mumps/5.2.1         py3-mpi4py/3.1.3
   netcdf-cxx/4.3.1    py3-scipy/1.5.4

------------------------ /opt/ohpc/pub/moduledeps/gnu12 ------------------------
   R/4.2.1         mpich/3.4.3-ofi        pdtoolkit/3.25.1
   gsl/2.7.1       mpich/3.4.3-ucx (D)    plasma/21.8.29
   hdf5/1.10.8     mvapich2/2.3.7         py3-numpy/1.19.5
   likwid/5.2.2    openblas/0.3.21        scotch/6.0.6
   metis/5.1.0     openmpi4/4.1.4  (L)    superlu/5.2.1

-------------------------- /opt/ohpc/pub/modulefiles ---------------------------
   EasyBuild/4.6.2                nvhpc-hpcx-cuda12/23.7
   autotools               (L)    nvhpc-hpcx/23.7
   charliecloud/0.15              nvhpc-nompi/23.7
   cmake/3.24.2                   nvhpc/23.7
   cuda                    (L)    ohpc                   (L)
   gnu12/12.2.0            (L)    os
   gnu9/9.4.0                     papi/6.0.0
   hwloc/2.7.0             (L)    prun/2.2               (L)
   libfabric/1.13.0        (L)    singularity/3.7.1
   magpie/2.5                     ucx/1.11.2             (L)
   nvhpc-byo-compiler/23.7        valgrind/3.19.0
   nvhpc-hpcx-cuda11/23.7

Advanced topics (pending)

This applies to:

• booting your own kernel
• installing your own OS

Known Problems

References:

• A User's Guide to MPI by Peter Pacheco
• Debugging: Gdb only works on one task with MPI, you need to "attach" to other tasks on the respective nodes. Another trick is to spawn terminals for each MPI rank: mpirun -np 2 /usr/bin/xterm -e gdb ./[my-a.out] or even start running with mpirun -np 2 /usr/bin/xterm -e gdb -ex run --args ./[my-a.out] [pgm-args]. We don't have totalview (an MPI-aware debugger), but it's available for free for students, though you would have to install it yourself. You can also use printf debugging, of course. If your program SEGVs, you can set ulimit -c unlimited and run the mpi program again, which will create one or more core dump files (per rank) named "core.PID", which you can then debug: gdb binary and then core core.PID.

Additional references:

• Intro to UNIX at NCSU
• Emacs reference card
• Gdb reference card
• Gnu manuals (make, gdb and many more)