Message Passing Programming with CMMD

(CS 267, Feb 2 1995)

Distributed address space SPMD, or message passing, is a programming style suitable for distributed address space MIMD machines. We begin by examining the CMMD solution of the first Sharks and Fish problem, fish swimming in a current. The solutions are written in C with calls to the CM-5 message passing library CMMD. They could also have been written in Fortran with no change in the use of CMMD. (Documentation for CMMD is located in Volume 1 of the class reference material, or by typing cmview while logged into rodin or moore.) There are actually two source files to examine, which you should pop up in separate windows: the control processor code, called fish.cp.c, and the processor node code, called The control processor code runs on the "front end" of the CM-5, the Sparc you log in to (rodin or moore), and the processor node code runs in parallel on all the processors.

The host and the nodes all begin by initializing the programming environment, by calling CMMD_enable_host() (on the nodes) and CMMD_enable() (on the host). All CMMD procedures are prefixed by CMMD_. These calls are synchronous, or blocking, meaning that all CMMD_enable_host() calls wait until all the others have been called, before simultaneously returning to continue execution.

The host proceeds to read input arguments, and computes

      myfish = fish/CMMD_partition_size(), 
the number of fish each node will own, which is equal to the total number of fish divided by the number of processors.

Then the host broadcasts the input data to all the processors by calling

      CMMD_bc_from_host(&host_data, sizeof(host_data)). 
The nodes receive this broadcast data by calling
      CMMD_receive_bc_from_host(&node_data, sizeof(node_data)). 
These calls are also synchronous: only after all nodes and the host call their respective routines and the broadcast is completed, do all subroutines simultaneously return. If one node failed to call CMMD_receive_bc_from_host, the other processors would hang. Also, all the processors should agree on the number of bytes being sent.

At this point the nodes compute myfish, the number of fish they will each simulate (by the same formula as above). Then they malloc the space needed to store the fish, and initialize them to some reasonable values. Each processor initializes its fish somewhat differently by having it depend on MYPROC = CMMD_self_address(), the number of each processor, which runs from 0 to CMMD_partition_size()-1.

Now the nodes start looping over the time steps of the simulation, updating their local fish positions and velocities. This is straightforward sequential C code. Periodically, every steps_per_display time steps, the nodes send their local fish positions back to the host for graphical display. To do this, each node calls

      CMMD_send(CMMD_host_node(), tag=0, 
           FISH = pointer to local fish array, 
           size of local fish array)
to send a message to the host containing the local fish data. The second argument is used to label messages, and is a kind of "return address" that can be used by the receiving processor to sort its incoming mail. The host in turn must receive a message from each processor by calling
      CMMD_receive (j, 0, &allfish[j*myfish], 
             size of incoming message)
in a loop from j=0 to CMMD_partition_size()-1. To receive a message, not only must the destination node specified in CMMD_send() and the source node specified in CMMD_receive() match, but the tags must match. One can also specify the source node as CMMD_ANY_NODE to match any sending node, and specify the tag as CMMD_ANY_TAG to match any incoming tag. The size of the incoming message must be at least as big as the size of the message sent. (One can query how many bytes actually arrived.)

These sends and receives are also synchronous, or blocking. (Later releases of CMMD use CMMD_send_block instead of CMMD_send, and CMMD_receive_block instead of CMMD_receive, to distinguish from other kinds of send and receive we will discuss shortly.) This means that the sending node does not continue executing until the receive has been executed and the transmission completed. Similarly, the receive does not return until the transmission is completed. In other words, blocking sends and receives must come in matching pairs, or else the program will hang.

Now we discuss the computation of the root-mean-square fish velocity, which requires computing a global sum of local sum-of-squared-velocities. The local sums computed on each processor are called vsum. The global sum is computed by having each node call

      vsum = CMMD_reduce_to_host_float(vsum, CMMD_combiner_fadd);
and the host call
      vsum = CMMD_reduce_from_nodes_float (0, CMMD_combiner_fadd);
These synchronous calls compute the global floating point sum (as indicated by the second argument CMMD_combiner_fadd) of the first argument (vsum), returning the same result everywhere. The global maximum of the velocities (maxv) and accelerations (maxa) are computed similarly. The final operation is a global maximum of the local CPU times to get the overall running time.

We wish to illustrate one other CMMD routine, which is used in the solution of the second Sharks and Fish problem: fish with gravity. As before, there are actually two source files to examine, which you should pop up in separate windows: the control processor code, called fish.cp.c, and the processor node code, called Recall that in our current solution, each fish position must be combined with every other fish position in order to compute the gravitational force. The approach taken in is to have each node call

in order to make a copy of all fish data on all processors. Here Fish is the local fish array, AllFish is another array large enough to contain all fish data, and the last argument is the size of the local fish data. This synchronous subroutine returns with a copy of each local Fish occupying part of AllFish. This is not intended to be a particularly efficient solution, especially in memory: there needs to be a completely redundant copy of all the data on each processor.

Let us think about improving this solution. The solution analogous to the CM Fortran and Matlab solutions is to make a second copy of the data, call it FISHp, and rotate it among the processors. We now have a choice: As in the CM Fortran, we could compute the contribution to each fish force from one other fish, and then rotate the second copy FISHp by one fish. Most rotating fish in FISHp would remain on the same processor, with those at the "boundary" requiring communication. Alternatively, we could loop over all the local fish, computing all myfish^2 local force contributions of FISHp to FISH, and then rotate by myfish, getting an entirely new subset of FISHp. The first solution does number_of_fish rotations (communications) of one fish each. The second solution does p-1 rotations of myfish = number_of_fish / p fish each. Which is faster?

To predict which is faster requires a performance model for communication. The simplest model we will use for communication is the following one: The time it takes to send n bytes from one processor to one other processor is

        alpha + beta n
where alpha is the latency (in units of seconds) and beta is the inverse of bandwidth, where bandwidth is in units of bytes/second. We will supply numbers for these later, but on the CM-5, you can expect a bandwidth of about 5 Megabytes/second, or beta = .2 microseconds/byte. Alpha is much larger, tens or hundreds of microseconds. Thus, the first solution, which sends number_of_fish-1 messages each of size 1, costs
  (number_of_fish-1)*alpha + (number_of_fish-1)*beta    seconds.
The second solution, which sends p-1 messages each of size myfish = number_of_fish/p, costs
    (p-1)*alpha + (p-1)*myfish*beta = 
       (p-1)*alpha + number_of_fish*(p-1)/p*beta  seconds.
Comparing the two timings, the coefficients of beta are nearly the same: all the fish still have to visit all the processors. But the coefficients of alpha are quite different: the second solution is faster, because the number of fish should be much greater than p (otherwise we would not bother with a parallel computer!), and alpha is much larger than beta.

Another way to think about this result is in terms of the memory hierarchy of the machine. Accessing remote memory requires explicit communication and is much more expensive than local memory. The second solution above reduces the number of times we have to access this remote memory (though the total number of words accessed remains about the same). This is analogous to the first assignment, where we saw how important it was to use an entire cache line, since it was as expensive to fetch one word as every byte in the line.

Now let us think about implementing the second solution, using the block send and receive we have just discussed. The most obvious solution is to try having each processor execute

    CMMD_send( (MPROC+1) mod p, ... )         
                  ! send to neighbor on right
    CMMD_receive( (MPROC-1) mod p, ... )      
                  ! receive from neighbor on left
But this solution deadlocks, because no CMMD_send is allowed to return until a matching receive has occurred. Since everyone is waiting, no progress is made.

A simple fix is to take two communication steps instead of one: do the odd processors, and then the even processors:

    if ( MPROC is even )
       CMMD_send( (MPROC+1) mod p, ... )       
                     ! send to odd neighbor on right
       CMMD_receive( (MPROC-1) mod p, ... )    
                     ! receive from odd neighbor on left
    else ! ( MPROC is odd )
       CMMD_receive( (MPROC-1) mod p, ... )    
                     ! receive from even neighbor on left
       CMMD_send( (MPROC+1) mod p, ... )       
                     ! send to even neighbor on right
However, no such simple solution suffices if instead of a simple rotation, we wanted to do an arbitrary permutation of our data.

To support these operations, there is another synchronous CMMD routine: CMMD_send_and_receive, which lets the user specify both a message to be sent (to MPROC+1, say) and a message to be received (from MPROC-1, say). However, it does not fix all our problems. In particular, it still requires all processors to synchronize, so that if one processor is "running late", all other processors must wait for it, lowering efficiency. This is analogous to not having any conversations at a party until all latecomers have arrived, rather than letting those already present talk independently.

The solution offered by CMMD (and other similar libraries like PVM and MPI) is to have versions of send and receive which expose more machine details and have less strict semantics than the synchronous versions just described.

Nonblocking send, or

   CMMD_send_noblock( destination_node, tag, &data, sizeof(data) )
sends a message "immediately" by either putting it in the communication network, or saving it in a buffer if the network is busy, and returns right away. There is no guarantee that the message is ever received by any other processor. Asynchronous send, or
   CMMD_send_asynch( destination_node, tag, &data, sizeof(data),
                     handler, handler_argument) 
has even weaker guarantees. CMMD_send_asynch returns a pointer to a message control block or MCB, a structure in which the system records the status of the message, and can be queried by the user. The routine returns immediately, even if the network is not ready to receive the data. But in the interest of efficiency, no copy is made of the data in this case, (copying takes time and space), and it is the user's responsibility not to overwrite the data before it has been sent. The user can discover whether the message has been sent in two ways. First, the user can examine the MCB, by calling either CMMD_msg_done(MCB) or CMMD_all_sends_done(). Second, the system will automatically call the user-supplied handler procedure (with arguments MCB and handler_argument) when the message is finally sent. This is done asynchronously, typically with an interrupt. The handler procedure can set a flag to indicate the send is done, increment a message counter, or most anything else not involving blocking communication. It should also make sure to free the MCB, which was malloced by the system at the time of the send; otherwise memory fills up with dead MCBs.

Either send may be paired either with CMMD_receive_block or an asynchronous receive

   CMMD_receive_asynch( source_node, tag, &data, sizeof(data), handler )
This routine returns immediately after allocating an MCB to keep track of the requested incoming message. As above, the user may query the MCB (via CMMD_msg_done(MCB) or CMMD_all_messages_done()) to see if the message has arrived, or let the handler do it, which as before is called when an interrupt announces the arrival of the requested message. Again, the handler must make sure to free the MCB created by the CMMD_receive_asynch call.

The use of nonblocking or asynchronous send and receive permits more flexibility in the design of parallel algorithms, allowing communication in the network to proceed in parallel with computation, without requiring both sending and receiving processors to be at the same place at the same time. A simple analogy is using the phone system, where both parties need to be present, and (e)-mail, which runs asynchronously. Indeed, using synchronous send is like using a telephone where, having dialed, you can't do anything other than wait for someone to pick up. Using synchronous receive is like having a telephone that does not ring, and having picked up the phone in anticipation of a call, you can't do anything other than wait for someone to call. (One can do slightly better by calling CMMD_msg_pending, which says whether an incoming message is pending, before calling CMMD_receive_block.)

Let us think about how to write a general permutation routine in the three styles discussed so far. Suppose each processor has an integer perm(i), where perm(0) through perm(p-1) form a permutation of 0 through p-1. First we will try synchronous send and receive, for which the relevant routine is

        source_node, source_tag, &source_array, sizeof(source_array), 
        dest_node, dest_tag, &dest_array, sizeof(dest_array) )
The first solution one might try is
        CMMD_any_node, 0, &local_data, sizeof(local_data), 
        perm(MYPROC), 0, &local_data, sizeof(local_data) )
The problem with this is that local_data may be overwritten by new data before it is sent out. So one needs to have a second copy:
        CMMD_any_node, 0, &new_local_data, sizeof(local_data), 
        perm(MYPROC), 0, &local_data, sizeof(local_data) )
This will work, but as we said, it requires all processor to synchronize, possibly wasting time if they are at different places in the computation.

Now let us try a solution using asynchronous or nonblocking send. Our first attempt is

   CMMD_send_asynch( perm(MYPROC), 0, &local_data, sizeof(local_data),
                     handler, handler_argument) 
   CMMD_receive_block( CMMD_any_node, 0, &local_data, sizeof(local_data) )
This permits every processor to send as soon as it can, and proceed as soon as its own data arrives (alternatively, we could do CMMD_receive_asynch if we did not need to use local_data immediately). But the trouble with this is that local_data could be overwritten before it is sent out, since CMMD_send_asynch assumes local_data will be available whenever it needs it. There are two ways to fix this. The first is to explicitly have a second copy of the local_data:
   CMMD_send_asynch( perm(MYPROC), 0, &local_data, sizeof(local_data),
                     handler, handler_argument) 
   CMMD_receive_block( CMMD_any_node, 0, &new_local_data, sizeof(local_data) )
and the second is to have an implicit copy, which is needed only if local_data cannot immediately be pushed into the communication network:
   CMMD_send_noblock( perm(MYPROC), 0, &local_data, sizeof(local_data) )
   CMMD_receive_noblock( CMMD_any_node, 0, &new_local_data, sizeof(local_data) )

There are some other important synchronous communication primitives in CMMD. One node may call

and all the other nodes may call
in order for the first node to broadcast data to all the others. All nodes may call
    CMMD_reduce_add (or CMMD_reduce_max or ...) 
in order to compute the global sum (or max or ...) of the arguments, and send this value to all the processors. All the nodes may call
    CMMD_scan_add (or CMMD_scan_max or ...) 
in order to compute the scan or parallel prefix of the indicated operations. These routines are the CMMD analogues of CMF_SCAN_ADD, etc., in CM Fortran. As we described in the solution of Sharks & Fish 2,
    CMMD_concat_with_nodes(&local_data, &all_data, sizeof(local_data) 
makes a copy of every processor's local_data in all_data on each processor.
may be used to set the number of processors equal to any q which is no larger than the actual number of physical processors. This may be used to help compute speedups for varying numbers of processors.

By this time, you may have the (accurate) impression that there are many different kinds of message passing routines, and may be wondering whether there is a common denominator to which they all may be reduced. There are two approaches to this problem. The first is a cooperative effort by the parallel programming community to write a standard for message passing libraries, called the Message Passing Interface or MPI (See Volume 5 of the class reference material, or click here). This long document standardizes many variations on message passing, and is being implemented by various companies and research groups. A second approach is being taken at Berkeley. Active Messages form a yet lower communication layer in which all the others may be implemented. Being lower level, it permits more efficient communication, as well as more general communication patterns (which we will see in our discussion of Split-C), but still provides a machine independent layer to help guarantee portability.