HW 3

Homework 3

Deadline: see web page
Assignments: All parts are to be solved individually (turned in electronically, written parts in ASCII text, NO Word, postscript, etc. permitted unless explicitly stated).

All programs have to be written in C, translated with mpicc/gcc and turned in with a corresponding Makefile.

(50 points) Analyze MPI call statistics and desive a method through PMPI instrumentation to generate a communication matrix.
- (0 points) Familiarize yourself briefly with the documentation of the NAS Parallel Benchmark suite and download the source code of NPB V3.3.
- Please use the henry2 cluster to compile the CG benchmark and run it:
  - ssh login.hpc.ncsu.edu -l unity-id
  - add intel
  - cp
  - tar xzf NPB-3.3.tar.gz
  - cd NPB-3.3/NPB-3.3-MPI
  - cp config/make.def.template config/make.def
  - edit config/make.def:
    - MPIF77=mpif77
    - MPICC=mpicc
    - FMPI_LIB =
    - CMPI_LIB =
  - make cg NPROCS=4 CLASS=A
  - mpirun -np 4 bin/cg.A.4
- (5 points) Familiarize yourself briefly with the documentation of the mpiP tool and download the source code of V3.1.2.
- Compile the mpiP sources, bind them into CG and analyze the output:
  - cd ../..
  - tar xzf mpiP-3.1.2.tar.gz
  - cd mpiP-3.1.2
  - ./configure
  - make
  - cd ../NPB-3.3/NPB-3.3-MPI
  - add to CG/Makefile: FMPI_LIB += -L../../../mpiP-3.1.2/ -lmpiP -lbfd -liberty
  - make cg NPROCS=16 CLASS=A
  - mpirun -np 16 bin/cg.A.16
  Analyze the out file cg.A.16.*.mpiP to answer (in README) the following questions:
  1. What is the dominating communication call in terms of time?
  2. How many Send calls account for over 90% of the message volume?
  3. Which communication routine had the highest duration? Why did this happen? (Consider input sizes, distribution of MPI tasks over nodes etc.)
  4. What was the average data size sent in reductions?
- (40 points) Write PMPI wrappers in a module named pmpi.c (in the CG subdirectory) to record the frequency of MPI_Send events in a communication matrix indexed by MPI task (rank ID) for source, destation of the message (see slide 55 of MPI lecture).
  Hints:
  - Write a wrappers for Init, Send and Finalize.
  - Use PMPI_XXX calls to (a) call the respective MPI routines...
  - but also to set up your instrumentation environment (to inquire about your rank etc.)
  - and to collect all frequencies from remote nodes.
  - Rank 0 should print out the entire matrix (plain data as integers separated by spaces without row/column headings sent to stdout).
  - Update the CG/Makefile.
- (5 points) Generate a heat plot.
  - Capture the communication matrix for CG with 64 MPI tasks and class A input in a file matrix.data on henry2.
  - Transfer the file to one of the osXX (XX=40..55) nodes.
  - run: gnuplot heatmap.plot
  - visualize your output: xv meatmap.jpg
  The results should look like this:
Turn in pmpi.c, matrix.data, README

(10 points) Modify the pi program from HW1 (see lecture slides) to parallelize for OpenMP (when called with option "o") and MPI+OpenMP (default). Obtain runtimes on henry2 for the Intel compiler for 100,000,000 intervals using gettimeofday() for OpenMP with 8 threads and OpenMP+MPI with 8 threads and 2 MPI tasks. Report results in README2. Sample runs:

./pi o
Process 0 on localhost with 8 threads
Enter the number of intervals: (0 quits) 1000000
pi is approximately 3.1415926535898708, Error is 0.0000000000000777
wall clock time = 0.  6964 sec.usecs

mpirun -np 2 pi
Process 0 on login01 with 8 threads
Enter the number of intervals: (0 quits) 100000
pi is approximately 3.1415926535981256, Error is 0.0000000000083324
wall clock time = 0.  2846 sec.usecs
Process 1 on login01 with 8 threads

Turn in pi.c, README2

(0 points) Learn how to compile and execute a CELL BE program on our Sony PS3 Cluster
- Install ~/.ssh/config on your local Linux machine. If you're not using Linux, skip this step.
- On Linux, ssh into any psXX machine (XX=00..07).
```
        ssh <your-unity-id›@psXX
      On windows, use your favorite ssh client to log in to
	optout.csc.ncsu.edu, port 22XX (XX=00..07). There may be a way to
        create aliases as well, check the manual of you ssh client. 
```
- export CELL_TOP=/opt/cell/sdk (and add to your ~/.bashrc)
- tar xf /opt/cell/sdk/src/tutorial_source.tar
- cd tutorial/simple
- make INSTALL_DIR=bin
- ./simple
(40 points) Modify the pi program from HW1 (see lecture slides) to parallelize for 6 Cell SPEs. Follow the steps below.
- Start out with the "simple" program from the tutorial but change the files to "ppu_pi.c" (host program for pi) and "pi_spu.c" (kernel of pi calculation).
- Create a data structure to pass parameters to the SPEs and initialize it on the host, including the number of intervals, the thread ID and a pointer to the result data. Make sure the data structure is an integer multiple of 16 bytes in size and aligned on a 16-byte boundary (for correct DMA).
- From the host, pass a pointer to this structure to the SPE thread as an argument.
- In the SPE kernel, receive the pointer to the structure. Then DMA the entire structure into a structure inside the SPE's local store (declared as "volatile ... __attribute__ ((aligned (128)));)".
  Hints: See the "euler" example in the tutorial and use spu_writech(), spu_mfcdma32(), and spu_mfcstat().
- After the kernel calculations, DMA the resulting partial sum to the host.
  Hints: Use spu_mfcdma32() and spu_mfcstat() to copy into the host's memory. Make sure your destination pointer in host memory is alligned on a 16-byte boundary.
- In the host, aggregate the partial results from the 6 SPE threads to obtain your PI approximation.
- Measure the overall time using gettimeofday() and compare it to an execution of the kernel on the host without SPE parallelization (optional parameters 's' for serial).
Sample output:
```
./pi
Enter the number of intervals: (0 quits) 2000000
pi is approximately 3.1415926535898016, Error is 0.0000000000000084
elapsed wall time 0.035496 sec.usec
Enter the number of intervals: (0 quits) 0

./pi s
Enter the number of intervals: (0 quits) 2000000
pi is approximately 3.1415926535899938, Error is 0.0000000000002007
elapsed wall time 0.108270 sec.usec
```
Turn in ppu_pi.c, Makefile, spu_pi.c, spu_Makefile, any local header files (if applicable)

What to turn in for programming assignments:

commented program(s) as source code, comments count 15% of the points (see class policy on guidelines on comments)
Makefiles (if required)
test programs as source (and input files, if required)
README (documentation to outline solution and list commands to install/execute)
in each file, include the following information as a comment at the top of the file, where "username" is your unity login name and the single author is the person who wrote this file:

Single Author info:

username FirstName MiddleInitial LastName