[mvapich-discuss] Stuck in waitall

[Hari Subramoni]

Thanks Maksym. We will take a look at it and get back to you. Hopefully,
you should be able to use the work around and proceed for the time being.

Thx,
Hari

On Sep 22, 2016 11:27 AM, "Maksym Planeta" <mplaneta at os.inf.tu-dresden.de>
wrote:

> I was lucky, so there exists a reproducer for me:
>
> #include <mpi.h>
>
> int main(int argc, char ** argv)
> {
>   MPI_Init(&argc, &argv);
>
>   MPI_Comm comm;
>
>   MPI_Comm_dup(MPI_COMM_WORLD, &comm);
>
>
>   int rank;
>
>   MPI_Comm_rank(MPI_COMM_WORLD, &rank);
>
>   double send = rank;
>   double sum = 0;
>   MPI_Request req;
>   MPI_Ireduce(&send, &sum, 1, MPI_DOUBLE, MPI_SUM, 0, comm, &req);
>   MPI_Waitall(1, &req, MPI_STATUSES_IGNORE);
>   MPI_Finalize();
> }
>
> And I factored out the reason to the value of variable MV2_SPIN_COUNT. If
> it is 1 it reliably hangs, if it is more, then doesn't seem to hang.
>
> For time being i will use bigger values for SPIN_COUNT
>
> On 09/22/2016 03:54 PM, Hari Subramoni wrote:
>
>> Thanks for the report. Sorry to hear that you are facing issues. Do you
>> have a reproducer that we could use? Does the hang go away of you remove
>> any of the environment variables?
>>
>> Thanks,
>> Hari.
>>
>>
>> On Sep 22, 2016 7:11 AM, "Maksym Planeta" <mplaneta at os.inf.tu-dresden.de
>> <mailto:mplaneta at os.inf.tu-dresden.de>> wrote:
>>
>>     Hello,
>>
>>     I have MPI_Reduce which I want to replace with a non-blocking
>>     collective.
>>
>>     Each rank passes a double, and I expect to gather sum on rank 0. The
>>     reduce I call as following:
>>
>>            MPI_Reduce(&ts->time, &ts->sum, 1, MPI_DOUBLE, MPI_SUM,
>>                        0, scr_comm_node);
>>
>>     scr_comm_node is a communicator which unites ranks of the same node.
>>
>>     I replaced this call with combination of ireduce, test, and wait,
>>     but the code was stuck all the time.
>>
>>     I started to simplify the code and ended up with these MPI calls
>>     which basically follow each other:
>>
>>            MPI_Ireduce(&ts->time, &ts->sum, 1, MPI_DOUBLE, MPI_SUM,
>>                        0, scr_comm_node, &ts->request[ts->num_req++]);
>>            ...
>>            MPI_Waitall(ts->num_req, &ts->request[0], &ts->status[0]);
>>
>>            ... <Then follow collectives on other communicators>
>>
>>     And looking different stack traces I see that at some of the nodes
>>     there are ranks, which can't simply leave this MPI_Waitall.
>>
>>     Am I doing something wrong here, or does it look like a bug?
>>
>>     I set following environment variables:
>>
>>     export MV2_USE_BLOCKING=1
>>     # I set affinity on my own, because I have 2 processes per CPU
>>     export MV2_ENABLE_AFFINITY=0
>>     export MV2_RDMA_NUM_EXTRA_POLLS=1
>>     export MV2_CM_MAX_SPIN_COUNT=1
>>     export MV2_SPIN_COUNT=1
>>
>>
>>     mpiname output:
>>
>>     MVAPICH2 2.2 Thu Sep 08 22:00:00 EST 2016 ch3:mrail
>>
>>     Compilation
>>     CC: gcc    -g -O0
>>     CXX: g++   -g -O0
>>     F77: gfortran -L/lib -L/lib   -g -O0
>>     FC: gfortran   -g -O0
>>
>>     Configuration
>>     --enable-fortran=all --enable-cxx --enable-error-checking=all
>>     --enable-error-messages=none --enable-timing=none
>>     --enable-check-compiler-flags --enable-threads=multiple
>>     --enable-weak-symbols --disable-dependency-tracking
>>     --enable-fast-install --disable-rdma-cm --with-pm=mpirun:hydra
>>     --with-rdma=gen2 --with-device=ch3:mrail --enable-alloca
>>     --enable-hwloc --disable-fast --enable-g=dbg
>>     --enable-error-messages=all --enable-error-checking=all
>>     --prefix=/home/s9951545/apps.taurus/mvapich2/2.2-mpirun-dbg/
>>
>>
>>
>>     --
>>     Regards,
>>     Maksym Planeta
>>
>>
>>
>>     _______________________________________________
>>     mvapich-discuss mailing list
>>     mvapich-discuss at cse.ohio-state.edu
>>     <mailto:mvapich-discuss at cse.ohio-state.edu>
>>     http://mailman.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>>     <http://mailman.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss>
>>
>>
> --
> Regards,
> Maksym Planeta
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20160922/3383b278/attachment.html>