[mvapich-discuss] MVAPICH2 issue with CH3 gen2 channel
sreeram potluri
potluri at cse.ohio-state.edu
Thu May 5 13:45:59 EDT 2011
Dear Juan,
I had used the default "./configure" to build the MVAPICH2 library.
"--enable-threads=multiple"
is not required as you do not have multiple threads which do MPI calls. So I
have used the default option.
You can disable RDMA optimized one-sided implementations in MVAPICH2 by
setting MV2_USE_RDMA_ONE_SIDED=0 when running your job. This will help you
to work-around the failures with 2GB transfers. Below is an appropriate
pointer to the user-guide.
http://mvapich.cse.ohio-state.edu/support/user_guide_mvapich2-1.6.html#x1-20400010.84
Hope this helps.
Thank you
Sreeram Potluri
On Thu, May 5, 2011 at 12:51 PM, Juan Vercellone <juanjov at gmail.com> wrote:
> Dear Sreeram,
> Sorry to bother you again, but could you please provide the
> configuration options for the MVAPICH2 compile with which you ran my
> application?
>
> I went through a rough testing today, with the changes you suggested,
> and the only channel that worked for me was CH3-Nemesis-IB
> (--with-device=ch3:nemesis:ib). Every other one throws errors of
> different nature.
>
> Would you like me to provide all this output as well? I have no
> problem in doing so, but I must warn you it'd be quite long hehe.
>
> Thank you again!
> My kind regards,
>
> On Thu, May 5, 2011 at 9:15 AM, Juan Vercellone <juanjov at gmail.com> wrote:
> > Like a charm :)
> >
> > Thank you very much, Sreeram.
> > I really appreciate your help.
> >
> > My best regards,
> >
> > On Wed, May 4, 2011 at 2:42 PM, sreeram potluri
> > <potluri at cse.ohio-state.edu> wrote:
> >> Dear Juan
> >> I observed in (2), you are sending out a large ~2GB message.
> >> Currently, the one-sided implementation of MVAPICH2 is not dealing
> >> properly with very large messages. We will fix it soon and send you a
> >> patch. BTW, is there a real world use-case for this?
> >>
> >> There seems to be a bug in the application (3). The size of data in
> >> communication calls is greater than the window size. The window
> >> creation function takes size in terms of bytes while the communication
> >> operations take size in terms of datatype elements. So in the code
> >> snippet below from p2p.c:p2p_onesided the size in MPI_Win_create call
> >> should bufsize*sizeof(long) rather than bufsize.
> >>
> >> Another problem I see here is that you have used sizeof(MPI_LONG) to
> >> get the size of the MPI datatype. This is erroneous as MPI_LONG itself
> >> might just be a macro. One should rather use the MPI_Type_size
> >> function.
> >>
> >> With these two changes I was able to run the active one-sided cases
> >> without any problem.
> >>
> >> err |= MPI_Win_create(
> >> info->b->mat,
> >> bufsize,
> >> sizeof(MPI_LONG),
> >> MPI_INFO_NULL,
> >> MPI_COMM_WORLD,
> >> &win);
> >>
> >> - - -
> >>
> >> err |= MPI_Get(
> >> buf[bufindex]->mat, bufsize, MPI_LONG,
> >> serverpid, 0, bufsize, MPI_LONG, win);
> >>
> >> Thank you
> >> Sreeram Potluri
> >> On Tue, May 3, 2011 at 1:09 PM, Juan Vercellone <juanjov at gmail.com>
> wrote:
> >>>
> >>> Dear Sreeram,
> >>> Thank you very much for your quick response. Here is some of the
> >>> information requested:
> >>>
> >>> 1) Which information would be of relevance here? (i.e. Linux commands
> to
> >>> issue).
> >>> The hostfile, on the other hand, is set depending on the tests to be
> >>> ran. I use 3 hostfiles. Here I provide some system information
> >>> (network):
> >>>
> >>> - /etc/hosts: http://pastebin.com/Ti3b7NDw
> >>> - hostfile 1: http://pastebin.com/nUsajf45
> >>> - hostfile 2: http://pastebin.com/z2QRqwTa
> >>> - hostfile 3: http://pastebin.com/5HKHWX84
> >>> - /sbin/ifconfig -a output: http://pastebin.com/RsTmTyKd
> >>> - /proc/version: http://pastebin.com/jgcGNys2
> >>>
> >>> As you can see, when using the 3rd hostfile, there are 2 processes per
> >>> node (I mostly issue my MPI application with 4 processes).
> >>>
> >>> 2) The application effectively uses multiple threads, but none of them
> >>> makes MPI calls. They are just created to perform some calculations,
> >>> and are terminated before final calculation result is obtained.
> >>>
> >>> I could run the following application using MVAPICH2 1.5rc1, but not
> >>> with MVAPICH2 1.6: http://pastebin.com/FsGjdfyj
> >>> MVAPICH2 1.5rc1 output: http://pastebin.com/ryAtKYwu
> >>> MVAPICH2 1.6 output: http://pastebin.com/0w53cHLY
> >>>
> >>>
> >>> 3) The application I am running is a full MPI application built on my
> >>> own. To do a checkout of the repository:
> >>> svn checkout http://mat-prod.googlecode.com/svn/trunk/mat-prod-read-only
> >>>
> >>> Makefiles are included:
> >>> - To compile: make clean && make
> >>> - To run:
> >>> - make erun (run with hosts as hostfile)
> >>> - make irun (run with hostsib as hostfile)
> >>> - make orun (run with hostso as hostfile)
> >>>
> >>>
> >>> Thanks again for your prompt response!
> >>> Regards,
> >>>
> >>> On Tue, May 3, 2011 at 9:12 AM, sreeram potluri
> >>> <potluri at cse.ohio-state.edu> wrote:
> >>> > Dear Juan,
> >>> > Thanks for reporting your problem. We will need information about
> your
> >>> > system, build and application to debug this further
> >>> > 1) Configuration of the system (node and network) and organization of
> >>> > the host file (1 process/node?)
> >>> > 2) Regarding failures with v1.6, I see that you have built it
> >>> > with --enable-threads=multiple. Does your application use multiple
> >>> > threads
> >>> > that make MPI calls?
> >>> > Were you able run other tests (a hello world or the OSU benchmarks )
> >>> > with
> >>> > this build? If not there could be a more basic issue with it.
> >>> > 3) Regarding failures with 1.5rc1, is the test you are running an
> >>> > application benchmark or a full application? Will it be possible for
> you
> >>> > to
> >>> > send us your code? It will be easiest to debug if we can reproduce
> your
> >>> > issue locally.
> >>> > If not, excerpts from the code where it uses one-sided calls that are
> >>> > causing the issue can give some insights (window creation,
> communication
> >>> > and
> >>> > synchronization calls).
> >>> > Thank you
> >>> > Sreeram Potluri
> >>> > On Tue, May 3, 2011 at 7:10 AM, Juan Vercellone <juanjov at gmail.com>
> >>> > wrote:
> >>> >>
> >>> >> Hello, list.
> >>> >> I am having some trouble executing my MPI applications with MVAPICH2
> >>> >> v1.6 (latest stable release).
> >>> >>
> >>> >> This is what I get when launching my programs:
> >>> >> mpirun_rsh -np 4 -hostfile ./hostso mat_prod 3
> >>> >> MPI process (rank: 1) terminated unexpectedly on compute-0-4.local
> >>> >> MPI process (rank: 0) terminated unexpectedly on compute-0-3.local
> >>> >> child_handler: Error in init phase...wait for cleanup! (1/2mpispawn
> >>> >> connections)
> >>> >> child_handler: Error in init phase...wait for cleanup! (1/2mpispawn
> >>> >> connections)
> >>> >> Failed in initilization phase, cleaned up all the mpispawn!
> >>> >>
> >>> >> The application doesn't even start.
> >>> >>
> >>> >> The MVAPICH2 instance was compiled using the following
> configuration:
> >>> >> ./configure --enable-threads=multiple --disable-f90
> >>> >>
> >>> >> Using MVAPICH2 v1.5rc1 with the same configuration options, I can
> get
> >>> >> my application to work with some communication schemes, but get
> error
> >>> >> messages when attempting to use one-sided active synchronization
> >>> >> calls.
> >>> >> Here are these errors:
> >>> >>
> >>> >> (FOR ACTIVE SYNC WITH POST/WAIT/START/COMPLETE)
> >>> >> mpirun_rsh -np 4 -hostfile ./hostso mat_prod 4
> >>> >> send desc error
> >>> >> [0] Abort: [] Got completion with error 10, vendor code=88, dest
> rank=1
> >>> >> at line 580 in file ibv_channel_manager.c
> >>> >> send desc error
> >>> >> [2] Abort: [] Got completion with error 10, vendor code=88, dest
> rank=3
> >>> >> at line 580 in file ibv_channel_manager.c
> >>> >> [3] Abort: Got FATAL event 3
> >>> >> at line 935 in file ibv_channel_manager.c
> >>> >> MPI process (rank: 0) terminated unexpectedly on compute-0-3.local
> >>> >> Exit code -5 signaled from compute-0-3
> >>> >> [1] Abort: Got FATAL event 3
> >>> >> at line 935 in file ibv_channel_manager.c
> >>> >> MPI process (rank: 3) terminated unexpectedly on compute-0-4.local
> >>> >> make: *** [orun] Error 1
> >>> >>
> >>> >>
> >>> >> (FOR ACTIVE SYNC WITH FENCE)
> >>> >> mpirun_rsh -np 4 -hostfile ./hostso mat_prod 6
> >>> >> send desc error
> >>> >> [2] Abort: [] Got completion with error 10, vendor code=88, dest
> rank=3
> >>> >> at line 580 in file ibv_channel_manager.c
> >>> >> send desc error
> >>> >> [0] Abort: [] Got completion with error 10, vendor code=88, dest
> rank=1
> >>> >> at line 580 in file ibv_channel_manager.c
> >>> >> MPI process (rank: 2) terminated unexpectedly on compute-0-3.local
> >>> >> Exit code -5 signaled from compute-0-3
> >>> >> [3] Abort: Got FATAL event 3
> >>> >> at line 935 in file ibv_channel_manager.c
> >>> >> [1] Abort: Got FATAL event 3
> >>> >> at line 935 in file ibv_channel_manager.c
> >>> >> MPI process (rank: 3) terminated unexpectedly on compute-0-4.local
> >>> >> make: *** [orun] Error 1
> >>> >>
> >>> >> In order to proceed, I need you to please specify a list of all the
> >>> >> necessary information you would need me to provide in order to
> >>> >> continue solving this issue.
> >>> >>
> >>> >> Thank you very much.
> >>> >> Regards,
> >>> >>
> >>> >> P.S.: Everything is working fine with (and just with) Nemesis for
> >>> >> InfiniBand (--with-device=ch3:nemesis:ib).
> >>> >>
> >>> >> --
> >>> >> ---------- .-
> >>> >> VERCELLONE, Juan.
> >>> >> (also known as 1010ad1c97efb4734854b6ffd0899401)
> >>> >> _______________________________________________
> >>> >> mvapich-discuss mailing list
> >>> >> mvapich-discuss at cse.ohio-state.edu
> >>> >> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
> >>> >
> >>> >
> >>>
> >>>
> >>>
> >>> --
> >>> ---------- .-
> >>> VERCELLONE, Juan.
> >>> (also known as 1010ad1c97efb4734854b6ffd0899401)
> >>
> >>
> >
> >
> >
> > --
> > ---------- .-
> > VERCELLONE, Juan.
> > (also known as 1010ad1c97efb4734854b6ffd0899401)
> >
>
>
>
> --
> ---------- .-
> VERCELLONE, Juan.
> (also known as 1010ad1c97efb4734854b6ffd0899401)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20110505/9733bfbb/attachment-0001.html
More information about the mvapich-discuss
mailing list