[mvapich-discuss] Performance of MPI_Iallgatherv

[Zehan Cui]

Hi,

I'm testing the non-blocking collective of MVAPICH2-2.0rc1.

I have two nodes with Infiniband to perform allgather on totally 128MB data.

I split the 128MB data into eight pieces, and perform computation and
MPI_Iallgatherv() on one piece of data each iteration, hoping that the
MPI_Iallgatherv() of last iteration can be overlapped with computation of
current iteration. A MPI_Wait() is called at the end of last iteration.

However, the total communication time (including the final wait time) even
exceeds the traditional blocking MPI_Allgatherv.


Following is the test pseudo-code and result.

===========================

Using MPI_Allgatherv:

for( i=0; i<8; i++ )
{
  // computation
    mytime( t_begin );
    computation;
    mytime( t_end );
    comp_time += (t_end - t_begin);

  // communication
    t_begin = t_end;
    MPI_Allgatherv();
    mytime( t_end );
    comm_time += (t_end - t_begin);
}

result:
    comp_time = 7,454,219 us
    comm_time = 399,854 us

--------------------------------------------

Using MPI_Iallgatherv:

for( i=0; i<8; i++ )
{
  // computation
    mytime( t_begin );
    computation;
    mytime( t_end );
    comp_time += (t_end - t_begin);

  // communication
    t_begin = t_end;
    MPI_Iallgatherv();
    mytime( t_end );
    comm_time += (t_end - t_begin);
}

// wait for non-blocking allgather to complete
mytime( t_begin );
for( i=0; i<8; i++ )
    MPI_Wait;
mytime( t_end );
wait_time = t_end - t_begin;

result:
    comp_time = 7,453,938 us
    comm_time = 365,511 us
    wait_time = 453,132 us

==============================

It seems that MPI_Iallgatherv() is more blocking than MPI_Allgather().

Does anyone have the same experience?


Thanks,
Zehan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20140401/555fe123/attachment.html>