Parallel and Distributed Computing (WS16/17)

Parallel and Distributed Computing

Most content is taken from Prof. Moore's lecture of Parallel and Distributed Computing in the previous years. I have added new content for GPGPU programming (Cuda and OpenCL) and removed some older stuff from the lecture and lab assignments. So be sure to use my slides (available in Moodle) and not Prof. Moore's slides from the previous years.


Belegnummer: 41.4874 Modulbeschreibung


A master's level lecture course with lab ("Praktikum"), in English.

Topics covered include:

  • Concepts and architectures for parallel and distributed computing
  • Shared Memory Programming Paradigm with Java, Posix threads and OpenMP
  • Message-Passing Programming with MPI
  • 21st century trends such as (for example):
  • Grid & Cloud Computing (particularly Apache Hadoop & Co.),
  • GPGPUs (General Purpose Graphic Processing Units)
  • 21st century programming languages such as Golang and Scala/Akka -- as time permits (!).


The Lecture starts on 05.10.2016.

The lab starts on TBA (24.10.2016) and TBA (31.10.2016).

Attendance at the lab is mandatory! If you do not appear at the first meeting, your seat in the lab can be given to another student.

Corollary: This also means that if you did not get a seat in the lab thorugh the OBS, come to the meeting anyway! You may be able to take over a seat from someone who didn''t attend the first meeting!

Please read the explanation of "Exercise 0" below before coming to the first lab meeting!

Exercise 0("Zero")


WARNING: This material is left over from last year and is subject to change!

Material: Exercise0.tar.gz. (coming soon) Warning: Changes are possible in this material in the days before the first meeting of the Lab (please check back here from time to time).

This file contains three CPU example programs (and one Makefile), as well as two GPU example programs (and no Makefile). All three CPU programs calculate the value of pi (3.1415926...) using the same algorithm, but very different parallel technologies, namely:

  1. MPI - Message Passing Interface
  2. C++11 Threads - the Thread library from the new C++11 standard.
  3. OpenMP.

(MPI is a library that must be installed; C++11 and OpenMP are supported by modern C++ Compilers).

Preparation for Exercise 0 - Prepare Your Team and Your Development Environments

  • Consider who you would like to be in a team with. The lab exercises will be performed by 2-person teams (3-person teams will be allowed only if we have an odd number of participants).
  • Set up your Development Environment (mpich - coming soon):
  • Every group needs at least one development computer for software development between appointments (whenever the laboratory is not available).
  • It is highly recommended (but not strictly speaking required) that development be done under Linux. Any Linux Distribution (e.g. Ubuntu, SUSE, etc.) should do. (Linux will be used in the lab).
  • Once you have Linux installed, you should have the C++ Compilers installed, and thus you should have C++11 and OpenMP.
  • You will also need to install an implementation of MPI. We recommend MPICH2. See (Binary packets are available for almost every Linux Distribution; Compiling from source is also highly recommended).
  • Test your Development Environment:
  • Download and run the three example programs (see above).
  • Enter "make tests" (after unpacking the file).
  • If all goes well, all three programs will be compiled, linked and run.
  • If not, fix them as well as you can.
  • Whether the programs run "out of the box" or not, familiarize yourself with the code.

In the lab, we will:

1. Determine who is present.

  • Attendance is mandatory!
  • Students who do not attend, and who have not sent the instructor an email explaining their absence (in advance!) will lose their seat in the lab!
  • Students who have not received a seat in the lab from the OBS should attend as well - extra seats will be distributed where possible.

2. Build teams.

3. Introduce Moodle

  • After this Exercise, all subsequent Exercises (i.e. Exercise n for all n where 0 < n <= 4) are described in Moodle s. Course Nr. TBA.
  • Moodle will be used for submitting Lab Reports (for all exercises, including this one) for grading.

4. Introduce the Lab Workstations

  • We will make sure everyone can log in to the computers in the laboratory.
  • We will make sure every group can build and run all three example programs (see above).

5. Benchmark the Three Sample Programs

  • Using the three example programs supplied (Exercise0.tar.gz, see above), you should measure the performance of the computers in the laboratory (and if you wish, your development computers).
  • To get interesting measurements, the programs will have to be modified! They are currently written to finish very quickly. The programs must be modified to run longer, so that the measurments are interesting!
  • You will then collect and analyze empirical results.
  • Finally, each team will produce and upload a short lab report (document) detailing your analysis, meeting minimal scientific standards (e.g. with a title, the names of all authors, an introduction, description of your methods, presentation of the empirical results, a conclusion, perhaps a list of literature, etc.). The lab report is a mandatory part of every exercise, including Exercise 0.