[mvapich-discuss] Fwd: Completion of non-blocking collective MPI_Iallgather

[pramod kumbhar]

Dear All,

I am looking at the changelog for 2.2a/2.2b and wondering if this has been
addressed in the recent releases.

Thanks in advance!
Pramod

On Sun, Apr 5, 2015 at 8:14 AM, Akshay Venkatesh <akshay at cse.ohio-state.edu>
wrote:

> Hi Pramod,
>
> The recent release hasn't addressed this particular issue yet and is aimed
> for the 2.2 version instead.
>
> Thanks
> On Apr 4, 2015 5:00 AM, "pramod kumbhar" <pramod.s.kumbhar at gmail.com>
> wrote:
>
>> Hi Akshay,
>>
>> Is this fixed/improved in the latest release?
>>
>> Regards,
>> Pramod
>>
>> On Fri, Jan 30, 2015 at 12:27 AM, pramod kumbhar <
>> pramod.s.kumbhar at gmail.com> wrote:
>>
>>> Hi Akshay,
>>>
>>> Thanks for update. I tried this and it seems better now now with extra
>>> MPI_Test.
>>>
>>> -Pramod
>>>
>>>
>>> On Wed, Jan 28, 2015 at 9:22 PM, Akshay Venkatesh <
>>> akshay at cse.ohio-state.edu> wrote:
>>>
>>>> Hi Pramod,
>>>>
>>>> We investigated the issue you reported and were able to reproduce it on
>>>> our local machines. Unfortunately there isn't an immediate fix for this
>>>> owing an algorithm characteristic of Iallgather. We are working on an
>>>> alternative to allow for better overlap but for the present one way to make
>>>> possibly partial progress on the outstanding non-blocking collective is to
>>>> have a few rounds of MPI_Test() rather than a single call as you've done in
>>>> your code. Hopefully this allows you to benefit from some partial overlap.
>>>>
>>>> On Tue, Jan 27, 2015 at 9:45 AM, pramod kumbhar <
>>>> pramod.s.kumbhar at gmail.com> wrote:
>>>>
>>>>> Could someone provide any suggestion on this?
>>>>>
>>>>> Thanks.
>>>>>
>>>>> ---------- Forwarded message ----------
>>>>> From: Kumbhar Pramod Shivaji <pramod.kumbhar at epfl.ch>
>>>>> Date: Mon, Jan 26, 2015 at 6:02 PM
>>>>> Subject: [mvapich-discuss] Completion of non-blocking collective
>>>>> MPI_Iallgather
>>>>> To: "mvapich-discuss at cse.ohio-state.edu" <
>>>>> mvapich-discuss at cse.ohio-state.edu>
>>>>>
>>>>>
>>>>> Hello All,
>>>>>
>>>>> I am looking into mpi non-blocking collectives (specifically
>>>>> mpi_iallgather & mpi_iallgatherv)
>>>>> and wondering about the internal completion / progress of these
>>>>> routines. To explain, I have following
>>>>> pseudo example:
>>>>>
>>>>> MPI_Iallgather(…&req)        *  // this called by all ranks at the
>>>>> same time*
>>>>> ...
>>>>> while( i < 4) {
>>>>>
>>>>>           compute(…);   // compute for few seconds
>>>>>           MPI_Test(…&flag, &req);                               *
>>>>> // Can I expect this flag to be true? i.e. operation is completed here at
>>>>> some point?*
>>>>>           i++;
>>>>> }
>>>>> ….
>>>>> MPI_Wait(&req);   * // I see that MPI_Iallgather is getting finished
>>>>> when everyone calls MPI_Wait at this point, why?*
>>>>>
>>>>>
>>>>> I was running this test example on single node with 8 mpi ranks. I see
>>>>> that MPI_Test is always returning false. With
>>>>> compute() function of two seconds, I expect MPI_Iallgather to finish
>>>>> quickly. But instead mpi_iallgather is completed
>>>>> at MPI_Wait(). See attached sample trace of the program.
>>>>>
>>>>> Could anyone point out possible cause of this behaviour? It will be
>>>>> great help.
>>>>>
>>>>>
>>>>> Regards,
>>>>> Pramod
>>>>>
>>>>> MVAPICH 2 version: 2.0
>>>>> Configure command:  /configure --with-slurm
>>>>>  --with-slurm-include=slurm/default/include
>>>>> --with-slurm-lib=slurm/default/lib  --with-pm=none --with-pmi=slurm
>>>>> --enable-threads=multiple --enable-shared --enable-sharedlibs=gcc
>>>>> --enable-fc --enable-cxx --with-mpe --enable-rdma-cm --enable-fast
>>>>> --enable-smpcoll --with-hwloc --enable-xrc --with-device=ch3:mrail
>>>>> --with-rdma=gen2 --enable-cuda --with-cuda=/usr/local/cuda-5.0
>>>>> --with-cuda-include=/usr/local/cuda-5.0/include
>>>>> --with-cuda-lib=/usr/local/cuda-5.0/lib64 --enable-g=dbg --enable-debuginfo
>>>>> CC=gcc CXX=g++ FC=gfortran F77=gfortran
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> mvapich-discuss mailing list
>>>>> mvapich-discuss at cse.ohio-state.edu
>>>>> http://mailman.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> mvapich-discuss mailing list
>>>>> mvapich-discuss at cse.ohio-state.edu
>>>>> http://mailman.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> - Akshay
>>>>
>>>
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20160211/383375c2/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mpi_iallgather_trace.png
Type: image/png
Size: 79473 bytes
Desc: not available
URL: <http://mailman.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20160211/383375c2/attachment-0001.png>