[mvapich-discuss] MVAPICH2-1.0 fails to complete IMB 3.0 sendrecv test (Was: Error in final mpicc, mpiCC, ...)

Fri Oct 5 15:28:18 EDT 2007

I was finally able to free up some time and try to pinpoint what is  
going wrong and send some follow-up to this problem.

We are trying to build and run MVAPICH2 1.0 but the IMB 3.0 tests  
fail to complete (MPI1 tests only), hanging in the IMB_sendrecv tests  
when the message size goes about 4MB.

I have traced the problem to using the --enable-fast configure  
switch.  If I do not include that switch, things run fine.  Here is  
how I am configuring and building:

$ ./configure --prefix=${PREFIX} --libdir=${PREFIX}/lib64 \
   --enable-sharedlibs=gcc \
   --enable-romio --with-file-system=ufs+nfs+panfs \
   --enable-threads=multiple --enable-g=dbg \
   --enable-totalview --enable-debuginfo \
   --with-mpe --enable-f77 --enable-f90 --enable-cxx \
   --with-openib=${OPEN_IB_HOME} \
   --with-pbs=${PBS_HOME} --with-device=osu_ch3:mrail \
   --with-rdma=gen2 --with-pm=mpd

$ make
$ make install

That is the configuration that works on our IB cluster.  Again, if I  
add in the --enable-fast flag, then the IMB 3.0 sendrecv test hangs.

-david

On Sep 24, 2007, at 8:56 PM, Dhabaleswar Panda wrote:

> David - We tried mpi_sendrecv on message sizes > 4 MB on various
> system configurations and platforms with mvapich2-1.0. Things seem to
> be working fine. We are not able to reproduce the error you are
> mentioning. Could you let us know details of the situation where you
> are seeing the mpi_sendrecv failure.
>
> Thanks,
>
> DK
>
>> What's the best way to go about resolving this?  Note: we are unable
>> to run mvapich2-1.0 on our systems.  There is a problem with that
>> version which prevents mpi_sendrecv from working for message sizes
>> above 4MB.
>>
>> -david
>> --
>> David Gunter
>> HPC-4: HPC Environments: Parallel Tools Team
>> Los Alamos National Laboratory
>>
>>
>>
>> --Apple-Mail-6--585903557
>> Content-Transfer-Encoding: quoted-printable
>> Content-Type: text/html;
>> 	charset=ISO-8859-1
>>
>> <HTML><BODY style=3D"word-wrap: break-word; -khtml-nbsp-mode:  
>> space; =
>> -khtml-line-break: after-white-space; ">After building  
>> mvapich2-0.98p3 =
>> on one of our systems I noticed that the final lib install  
>> directory =
>> looked like this:<DIV><BR =
>> class=3D"khtml-block-placeholder"></DIV><DIV>/opt/mvapich2/ 
>> mvapich2-0.98p3=
>> /lib64</DIV><DIV><BR class=3D"khtml-block-placeholder"></ 
>> DIV><DIV>whereas =
>> inside the final "mpicc" and other compiler scripts was the =
>> following:</DIV><DIV><BR =
>> class=3D"khtml-block-placeholder"></DIV><DIV>libdir=3D$ 
>> {exec_prefix}/lib</=
>> DIV><DIV><BR class=3D"khtml-block-placeholder"></DIV><DIV>which  
>> maps out =
>> to=A0/opt/mvapich2/mvapich2-0.98p3/lib.</DIV><DIV><BR =
>> class=3D"khtml-block-placeholder"></DIV><DIV>Thus, any attempt to =
>> compile using the default mpi compile scripts results in libraries  
>> not =
>> being found.</DIV><DIV><BR =
>> class=3D"khtml-block-placeholder"></DIV><DIV>What's the best way  
>> to go =
>> about resolving this?=A0 Note: we are unable to run mvapich2-1.0  
>> on our =
>> systems.=A0 There is a problem with that version which prevents =
>> mpi_sendrecv from working for message sizes above 4MB.</ 
>> DIV><DIV><BR =
>> class=3D"khtml-block-placeholder"></DIV><DIV>-david</DIV><DIV><SPAN =
>> class=3D"Apple-style-span" style=3D"border-collapse: separate; =
>> border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family:  
>> Helvetica; =
>> font-size: 12px; font-style: normal; font-variant: normal; font- 
>> weight: =
>> normal; letter-spacing: normal; line-height: normal; text-align:  
>> auto; =
>> -khtml-text-decorations-in-effect: none; text-indent: 0px; =
>> -apple-text-size-adjust: auto; text-transform: none; orphans: 2; =
>> white-space: normal; widows: 2; word-spacing: 0px; "><SPAN =
>> class=3D"Apple-style-span" style=3D"border-collapse: separate; =
>> border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family:  
>> Helvetica; =
>> font-size: 12px; font-style: normal; font-variant: normal; font- 
>> weight: =
>> normal; letter-spacing: normal; line-height: normal; text-align:  
>> auto; =
>> -khtml-text-decorations-in-effect: none; text-indent: 0px; =
>> -apple-text-size-adjust: auto; text-transform: none; orphans: 2; =
>> white-space: normal; widows: 2; word-spacing: 0px; "><SPAN =
>> class=3D"Apple-style-span" style=3D"border-collapse: separate; =
>> border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family:  
>> Helvetica; =
>> font-size: 12px; font-style: normal; font-variant: normal; font- 
>> weight: =
>> normal; letter-spacing: normal; line-height: normal; text-align:  
>> auto; =
>> -khtml-text-decorations-in-effect: none; text-indent: 0px; =
>> -apple-text-size-adjust: auto; text-transform: none; orphans: 2; =
>> white-space: normal; widows: 2; word-spacing: 0px; "><SPAN =
>> class=3D"Apple-style-span" style=3D"border-collapse: separate; =
>> border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family:  
>> Helvetica; =
>> font-size: 12px; font-style: normal; font-variant: normal; font- 
>> weight: =
>> normal; letter-spacing: normal; line-height: normal; text-align:  
>> auto; =
>> -khtml-text-decorations-in-effect: none; text-indent: 0px; =
>> -apple-text-size-adjust: auto; text-transform: none; orphans: 2; =
>> white-space: normal; widows: 2; word-spacing: 0px; "><SPAN =
>> class=3D"Apple-style-span" style=3D"border-collapse: separate; =
>> border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family:  
>> Helvetica; =
>> font-size: 12px; font-style: normal; font-variant: normal; font- 
>> weight: =
>> normal; letter-spacing: normal; line-height: normal; text-align:  
>> auto; =
>> -khtml-text-decorations-in-effect: none; text-indent: 0px; =
>> -apple-text-size-adjust: auto; text-transform: none; orphans: 2; =
>> white-space: normal; widows: 2; word-spacing: 0px; =
>> "><DIV>--</DIV><DIV>David Gunter</DIV><DIV>HPC-4: HPC Environments: =
>> Parallel Tools Team</DIV><DIV>Los Alamos National =
>> Laboratory</DIV></SPAN></SPAN><BR =
>> class=3D"Apple-interchange-newline"></SPAN></SPAN></SPAN> =
>> </DIV><BR></BODY></HTML>=
>>
>> --Apple-Mail-6--585903557--
>>
>> --===============1825395516==
>> Content-Type: text/plain; charset="us-ascii"
>> MIME-Version: 1.0
>> Content-Transfer-Encoding: 7bit
>> Content-Disposition: inline
>>
>> _______________________________________________
>> mvapich-discuss mailing list
>> mvapich-discuss at cse.ohio-state.edu
>> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>>
>> --===============1825395516==--
>>
>