MPI Program Sample on Extreme

This page contains an example of Message Passing Interface (MPI) program and how to run a MPI program as a job on Extreme. For more information on compiling programs on Extreme, please see the section of Compiling your program on Extreme on FAQ page. For more information on managing your job on Extreme, please see the section of Submit and manage jobs on Extreme on FAQ page.

 

Example of a MPI Program mpitest.c

/* MPI program sample, HPCC, Sept. 2015 */

#include <mpi.h>

#include <stdio.h>

#include <unistd.h>

int main(int argc, char **argv) {

int world_size;

int rank;

char hostname[256];

char processor_name[MPI_MAX_PROCESSOR_NAME];

int name_len;

// Initialize the MPI environment

MPI_Init(&argc, &argv);

// get the total number of processes

MPI_Comm_size(MPI_COMM_WORLD, &world_size);

// get the rank of current process

MPI_Comm_rank(MPI_COMM_WORLD, &rank);

// get the name of the processor

MPI_Get_processor_name(processor_name, &name_len);

// get the hostname

gethostname(hostname,255);

printf(“Hello world!  I am process number %d from processor %s on host %s out of %d processors\n”, rank, processor_name, hostname, world_size);

MPI_Finalize();

return 0;

}

The first step to building an MPI program is to include the MPI header files with #include <mpi.h>. After this, the MPI environment must be initialized with:

MPI_Init(&argc, &argv);

During MPI_Init, all of MPI’s global and internal variables are constructed. Here, MPI_Init takes two arguments that are not necessary, and the extra parameters are simply left as extra space in case future implementations might need them.

After MPI_Init, two main functions are called, which are used in almost every single MPI program.
MPI_Comm_size returns the size of a communicator. Since MPI_COMM_WORLD encloses all of the processes in the job, this call should return the amount of processes that were requested for the job.
MPI_Comm_rank returns the rank of a process in a communicator, which is usually used for identification purposes when sending/receiving messages.

The next function MPI_Get_processor_name is usually used to obtain the actual name of the processor. The exact returned format is implementation defined and it may or may not be the same as might be returned by gethostname, which is mentioned below.

gethostname is a useful non-MPI function, which is actually a system call used to access the hostname of the current processor.

The final call in this program is:

MPI_Finalize()

MPI_Finalize is used to clean up the MPI environment. No more MPI calls can be made after this one.

 

Example of MPI Program Execution

The preferred means of starting MPI programs on Extreme is mpiexec. It is much simpler to write the PBS script file using mpiexec than mpirun. The example MPI program above, mpitest.c, could be executed with the following PBS script that uses mpiexec.

Sample Script mpitest.pbs to Run mpitest.c on Extreme

#!/bin/bash

#PBS -l nodes=2:ppn=20,walltime=1:00

#PBS -N mpiTestJob

#PBS -q edu_shared

#PBS -m abe

#PBS -M netid@uic.edu

#PBS -e mpitest.err

#PBS -o mpitest.out

#PBS -d /export/home/netid/path/to/mpitest/directory

module load tools/mpich2-1.5-gcc

mpiexec -n 40 ./mpitest

“#PBS -d” specifies the directory the program begin executing. Make sure it contains your executable file “mpitest”. The standard output file mpitest.out and standard error file mpitest.err will be created in the same directory after the job terminates.

In this example, the script will spawn 40 processes on two nodes with 20 processes on each.

How to Start for CS-ECE566 Students

For education purpose, the above mpitest.c and mpitest.pbs have been stored in class shared directory of CS-ECE566 at /export/share/classes/cs-ec566/MPIsample/ on Extreme. You have to copy them over to your home directory and from there you need to change the .pbs script accordingly to get stared.

Example of Building and Running MPI

0. Make sure load MPICH module before compiling your MPI program.

module load tools/mpich2-1.5-gcc

1. Compile the program

      

mpicc -o mpitest mpitest.c
 

2. Submit the .pbs file to the queue.

      

qsub mpitest.pbs

When job submission is successful, the qsub command returns the job’s process number:

143796.admin.extreme.uic.edu

Output Files

If you specifies the output file with “#PBS -O” as the above script shows, the output file is mpitest.out stored in the same directory as where your mpitest executable is stored. If not, by default the output files are created using the same filename as your PBS script filename with an extension that includes “e” for “error” or “o” for “output” followed by the process number.

The output of the example program can be viewed by listing the output file.

Output Snippet of the Sample mpitest Program

Hello world! I am process number: 36 from processor compute-1-32.local on host compute-1-32.local out of 40 processors

Hello world! I am process number: 37 from processor compute-1-32.local on host compute-1-32.local out of 40 processors

Hello world! I am process number: 38 from processor compute-1-32.local on host compute-1-32.local out of 40 processors

Hello world! I am process number: 39 from processor compute-1-32.local on host compute-1-32.local out of 40 processors

Hello world! I am process number: 22 from processor compute-1-32.local on host compute-1-32.local out of 40 processors

Hello world! I am process number: 25 from processor compute-1-32.local on host compute-1-32.local out of 40 processors

Hello world! I am process number: 1 from processor compute-0-34.local on host compute-0-34.local out of 40 processors

Hello world! I am process number: 2 from processor compute-0-34.local on host compute-0-34.local out of 40 processors

Hello world! I am process number: 3 from processor compute-0-34.local on host compute-0-34.local out of 40 processors

Hello world! I am process number: 4 from processor compute-0-34.local on host compute-0-34.local out of 40 processors

Hello world! I am process number: 7 from processor compute-0-34.local on host compute-0-34.local out of 40 processors

Hello world! I am process number: 13 from processor compute-0-34.local on host compute-0-34.local out of 40 processors

Hello world! I am process number: 15 from processor compute-0-34.local on host compute-0-34.local out of 40 processors

Hello world! I am process number: 16 from processor compute-0-34.local on host compute-0-34.local out of 40 processors

Hello world! I am process number: 19 from processor compute-0-34.local on host compute-0-34.local out of 40 processors

Hello world! I am process number: 0 from processor compute-0-34.local on host compute-0-34.local out of 40 processors

As it is shown, 40 processes are created on 2 nodes (compute-1-32 and compute-0-34) and each process have a unique rank number associated with it.