[mvapich-discuss] MVAPICH2-1.0 fails to complete IMB 3.0 sendrecv test (Was: Error in final mpicc, mpiCC, ...)

Sat Oct 6 17:11:09 EDT 2007

Hi David,

Thanks for your mail. The enable-fast option pick the appropriate
options for fast execution.  This turns off error checking and timing
collection. We typically do not enable this because it will not report
errors.  We tested the latest version with this option and we are not
able to reproduce the problem for messages up to 32 MB. In our testbed:
The CPUs support the EM64T technology and run in 64 bit mode. The nodes
support 8x PCI Express interfaces and are equipped with MT25208 HCAs
with PCI Express interfaces.  

Thanks and regards,

:- Abhinav

> I was finally able to free up some time and try to pinpoint what is  
> going wrong and send some follow-up to this problem.
> 
> We are trying to build and run MVAPICH2 1.0 but the IMB 3.0 tests  
> fail to complete (MPI1 tests only), hanging in the IMB_sendrecv tests  
> when the message size goes about 4MB.
> 
> I have traced the problem to using the --enable-fast configure  
> switch.  If I do not include that switch, things run fine.  Here is  
> how I am configuring and building:
> 
> $ ./configure --prefix=${PREFIX} --libdir=${PREFIX}/lib64 \
>   --enable-sharedlibs=gcc \
>   --enable-romio --with-file-system=ufs+nfs+panfs \
>   --enable-threads=multiple --enable-g=dbg \
>   --enable-totalview --enable-debuginfo \
>   --with-mpe --enable-f77 --enable-f90 --enable-cxx \
>   --with-openib=${OPEN_IB_HOME} \
>   --with-pbs=${PBS_HOME} --with-device=osu_ch3:mrail \
>   --with-rdma=gen2 --with-pm=mpd
> 
> $ make
> $ make install
> 
> That is the configuration that works on our IB cluster.  Again, if I  
> add in the --enable-fast flag, then the IMB 3.0 sendrecv test hangs.
> 
> -david
> 
> 
> On Sep 24, 2007, at 8:56 PM, Dhabaleswar Panda wrote:
> 
> >David - We tried mpi_sendrecv on message sizes > 4 MB on various
> >system configurations and platforms with mvapich2-1.0. Things seem to
> >be working fine. We are not able to reproduce the error you are
> >mentioning. Could you let us know details of the situation where you
> >are seeing the mpi_sendrecv failure.
> >
> >Thanks,
> >
> >DK
> >
> >>What's the best way to go about resolving this?  Note: we are unable
> >>to run mvapich2-1.0 on our systems.  There is a problem with that
> >>version which prevents mpi_sendrecv from working for message sizes
> >>above 4MB.
> >>
> >>-david
> >>--
> >>David Gunter
> >>HPC-4: HPC Environments: Parallel Tools Team
> >>Los Alamos National Laboratory
> >>
> >>
> >>
> >>--Apple-Mail-6--585903557
> >>Content-Transfer-Encoding: quoted-printable
> >>Content-Type: text/html;
> >>	charset=ISO-8859-1
> >>
> >><HTML><BODY style=3D"word-wrap: break-word; -khtml-nbsp-mode:  
> >>space; =
> >>-khtml-line-break: after-white-space; ">After building  
> >>mvapich2-0.98p3 =
> >>on one of our systems I noticed that the final lib install  
> >>directory =
> >>looked like this:<DIV><BR =
> >>class=3D"khtml-block-placeholder"></DIV><DIV>/opt/mvapich2/ 
> >>mvapich2-0.98p3=
> >>/lib64</DIV><DIV><BR class=3D"khtml-block-placeholder"></ 
> >>DIV><DIV>whereas =
> >>inside the final "mpicc" and other compiler scripts was the =
> >>following:</DIV><DIV><BR =
> >>class=3D"khtml-block-placeholder"></DIV><DIV>libdir=3D$ 
> >>{exec_prefix}/lib</=
> >>DIV><DIV><BR class=3D"khtml-block-placeholder"></DIV><DIV>which  
> >>maps out =
> >>to=A0/opt/mvapich2/mvapich2-0.98p3/lib.</DIV><DIV><BR =
> >>class=3D"khtml-block-placeholder"></DIV><DIV>Thus, any attempt to =
> >>compile using the default mpi compile scripts results in libraries  
> >>not =
> >>being found.</DIV><DIV><BR =
> >>class=3D"khtml-block-placeholder"></DIV><DIV>What's the best way  
> >>to go =
> >>about resolving this?=A0 Note: we are unable to run mvapich2-1.0  
> >>on our =
> >>systems.=A0 There is a problem with that version which prevents =
> >>mpi_sendrecv from working for message sizes above 4MB.</ 
> >>DIV><DIV><BR =
> >>class=3D"khtml-block-placeholder"></DIV><DIV>-david</DIV><DIV><SPAN =
> >>class=3D"Apple-style-span" style=3D"border-collapse: separate; =
> >>border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family:  
> >>Helvetica; =
> >>font-size: 12px; font-style: normal; font-variant: normal; font- 
> >>weight: =
> >>normal; letter-spacing: normal; line-height: normal; text-align:  
> >>auto; =
> >>-khtml-text-decorations-in-effect: none; text-indent: 0px; =
> >>-apple-text-size-adjust: auto; text-transform: none; orphans: 2; =
> >>white-space: normal; widows: 2; word-spacing: 0px; "><SPAN =
> >>class=3D"Apple-style-span" style=3D"border-collapse: separate; =
> >>border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family:  
> >>Helvetica; =
> >>font-size: 12px; font-style: normal; font-variant: normal; font- 
> >>weight: =
> >>normal; letter-spacing: normal; line-height: normal; text-align:  
> >>auto; =
> >>-khtml-text-decorations-in-effect: none; text-indent: 0px; =
> >>-apple-text-size-adjust: auto; text-transform: none; orphans: 2; =
> >>white-space: normal; widows: 2; word-spacing: 0px; "><SPAN =
> >>class=3D"Apple-style-span" style=3D"border-collapse: separate; =
> >>border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family:  
> >>Helvetica; =
> >>font-size: 12px; font-style: normal; font-variant: normal; font- 
> >>weight: =
> >>normal; letter-spacing: normal; line-height: normal; text-align:  
> >>auto; =
> >>-khtml-text-decorations-in-effect: none; text-indent: 0px; =
> >>-apple-text-size-adjust: auto; text-transform: none; orphans: 2; =
> >>white-space: normal; widows: 2; word-spacing: 0px; "><SPAN =
> >>class=3D"Apple-style-span" style=3D"border-collapse: separate; =
> >>border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family:  
> >>Helvetica; =
> >>font-size: 12px; font-style: normal; font-variant: normal; font- 
> >>weight: =
> >>normal; letter-spacing: normal; line-height: normal; text-align:  
> >>auto; =
> >>-khtml-text-decorations-in-effect: none; text-indent: 0px; =
> >>-apple-text-size-adjust: auto; text-transform: none; orphans: 2; =
> >>white-space: normal; widows: 2; word-spacing: 0px; "><SPAN =
> >>class=3D"Apple-style-span" style=3D"border-collapse: separate; =
> >>border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family:  
> >>Helvetica; =
> >>font-size: 12px; font-style: normal; font-variant: normal; font- 
> >>weight: =
> >>normal; letter-spacing: normal; line-height: normal; text-align:  
> >>auto; =
> >>-khtml-text-decorations-in-effect: none; text-indent: 0px; =
> >>-apple-text-size-adjust: auto; text-transform: none; orphans: 2; =
> >>white-space: normal; widows: 2; word-spacing: 0px; =
> >>"><DIV>--</DIV><DIV>David Gunter</DIV><DIV>HPC-4: HPC Environments: =
> >>Parallel Tools Team</DIV><DIV>Los Alamos National =
> >>Laboratory</DIV></SPAN></SPAN><BR =
> >>class=3D"Apple-interchange-newline"></SPAN></SPAN></SPAN> =
> >></DIV><BR></BODY></HTML>=
> >>
> >>--Apple-Mail-6--585903557--
> >>
> >>--===============1825395516==
> >>Content-Type: text/plain; charset="us-ascii"
> >>MIME-Version: 1.0
> >>Content-Transfer-Encoding: 7bit
> >>Content-Disposition: inline
> >>
> >>_______________________________________________
> >>mvapich-discuss mailing list
> >>mvapich-discuss at cse.ohio-state.edu
> >>http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
> >>
> >>--===============1825395516==--
> >>
> >
> 
> _______________________________________________
> mvapich-discuss mailing list
> mvapich-discuss at cse.ohio-state.edu
> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss