[mvapich-discuss] MVAPICH2-0.9.8 internal errors

Dhabaleswar Panda panda at cse.ohio-state.edu
Wed May 9 16:02:00 EDT 2007 

Hi David, 

Do you see this problem with MVAPICH2 0.9.8p2 version? We had several
fixes to the released version during the last months. This latest
version (0.9.8p2) is available from the mvapich page. This version is
also included and is being tested with OFED 1.2. We have also done
testing of this latest version with OFED 1.1.

Please let us know if you still see the error with this latest version
(p2) on your system and we will investigate this issue further. 

Thanks, 

DK


> I am curious to know if you were able to reproduce this problem,  
> whether it has been fixed or not.
> 
> Thanks,
> david
> --
> David Gunter
> HPC-4: HPC Environments: Parallel Tools Team
> Los Alamos National Laboratory
> 
> 
> On Mar 22, 2007, at 11:20 AM, David Gunter wrote:
> 
> > I am using the released 0.9.8 version.
> >
> > -david
> >
> > On Mar 22, 2007, at 9:37 AM, Dhabaleswar Panda wrote:
> >
> >> Hi David,
> >>
> >> Thanks for this information. One more question - are you using
> >> MVAPICH2 0.9.8 `released' version or the `branch' version (with some
> >> recent fixes). If you can let us know this information, it will help
> >> us.
> >>
> >> Thanks,
> >>
> >> DK
> >>
> >>
> >>> I have recompiled mvapich2 without using the --enable-debuginfo flag
> >>> and the problem has gone away.  However, I wish to have debuginfo
> >>> available to our TotalView users so hopefully this can be resolved.
> >>>
> >>> Here is the configuration that generates the error message I saw
> >>> previously:
> >>>
> >>> ./configure --prefix=/opt/mvapich2/mvapich2-0.9.8-gcc/ib --with-
> >>> openib=/usr/local/ofed --enable-romio --with-file-system=ufs+nfs
> >>> +panfs --enable-sharedlibs=gcc --enable-debuginfo --enable-fast --
> >>> with-mpe
> >>>
> >>> OFED is ofed-1.1
> >>>
> >>> CFLAGS=-D_X86_64_ -D_SMP_ -DUSE_HEADER_CACHING  -DONE_SIDED -
> >>> DMPID_USE_SEQUENCE_NUMBERS  -I/usr/local/ofed/include -O2   -
> >>> D_SHMEM_COLL_
> >>>
> >>> CC=/usr/bin/gcc
> >>> CXX=/usr/bin/g++
> >>> FC=/usr/bin/gfortran
> >>> F77=/usr/bin/gfortran
> >>> F90=/usr/bin/gfortran
> >>>
> >>> The test was run on 32 process, 16 process, 8 process and 4  
> >>> process -
> >>> all of which generated this error message.
> >>>
> >>> Thanks,
> >>> david
> >>>
> >>>
> >>>
> >>> --
> >>> David Gunter
> >>> HPC-4: HPC Environments: Parallel Tools Team
> >>> Los Alamos National Laboratory
> >>>
> >>>
> >>> On Mar 21, 2007, at 4:50 PM, wei huang wrote:
> >>>
> >>>> Hi,
> >>>>
> >>>> Thanks for letting us know this problem.
> >>>>
> >>>> We will try to reproduce the problem on our cluster. To help us on
> >>>> looking
> >>>> into this problem, would you please let us know the following:
> >>>>
> >>>> 1) The exact CFLAGS you are using when configuring mvapich2 (are
> >>>> you using
> >>>> default scripts provided by us?)
> >>>> 2) Any runtime environment variables you have setup up?
> >>>> 3) On how many nodes do you run the test?
> >>>>
> >>>> Thanks.
> >>>>
> >>>> Regards,
> >>>> Wei Huang
> >>>>
> >>>> 774 Dreese Lab, 2015 Neil Ave,
> >>>> Dept. of Computer Science and Engineering
> >>>> Ohio State University
> >>>> OH 43210
> >>>> Tel: (614)292-8501
> >>>>
> >>>>
> >>>> On Wed, 21 Mar 2007, David Gunter wrote:
> >>>>
> >>>>> I have built mvapich2 for an OFED-based IB Opteron cluster.  When
> >>>>> running the Intel MPI Benchmarks (IMB3) I keep seeing the  
> >>>>> following
> >>>>> errors messages in many spots, although the tests run to  
> >>>>> completion:
> >>>>>
> >>>>> Internal error: communicator is already on free list
> >>>>>
> >>>>> What is this referring to?
> >>>>>
> >>>>> Thanks.
> >>>>> --david
> >>>>>
> >>>>> --
> >>>>> David Gunter
> >>>>> HPC-4: HPC Environments: Parallel Tools Team
> >>>>> Los Alamos National Laboratory
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>>
> >>>
> >>> --Apple-Mail-1-849288998
> >>> Content-Transfer-Encoding: quoted-printable
> >>> Content-Type: text/html;
> >>> 	charset=ISO-8859-1
> >>>
> >>> <HTML><BODY style=3D"word-wrap: break-word; -khtml-nbsp-mode:  
> >>> space; =
> >>> -khtml-line-break: after-white-space; ">I have recompiled mvapich2 =
> >>> without using the --enable-debuginfo flag and the problem has gone =
> >>> away.=A0 However, I wish to have debuginfo available to our  
> >>> TotalView =
> >>> users so hopefully this can be resolved.<DIV><BR =
> >>> class=3D"khtml-block-placeholder"></DIV><DIV>Here is the  
> >>> configuration =
> >>> that generates the error message I saw previously:</DIV><DIV><BR =
> >>> class=3D"khtml-block-placeholder"></DIV><DIV>./configure =
> >>> --prefix=3D/opt/mvapich2/mvapich2-0.9.8-gcc/ib=A0--with-openib=3D/ 
> >>> usr/loca=
> >>> l/ofed --enable-romio --with-file-system=3Dufs+nfs+panfs =
> >>> --enable-sharedlibs=3Dgcc --enable-debuginfo --enable-fast =
> >>> --with-mpe</DIV><DIV><BR class=3D"khtml-block-placeholder"></ 
> >>> DIV><DIV>OFED=
> >>>  is ofed-1.1</DIV><DIV><BR =
> >>> class=3D"khtml-block-placeholder"></DIV><DIV>CFLAGS=3D-D_X86_64_ - 
> >>> D_SMP_ =
> >>> -DUSE_HEADER_CACHING=A0 -DONE_SIDED -DMPID_USE_SEQUENCE_NUMBERS=A0 =
> >>> -I/usr/local/ofed/include -O2=A0=A0 -D_SHMEM_COLL_</DIV><DIV><BR =
> >>> class=3D"khtml-block-placeholder"></DIV><DIV>CC=3D/usr/bin/gcc</ 
> >>> DIV><DIV>C=
> >>> XX=3D/usr/bin/g++</DIV><DIV>FC=3D/usr/bin/gfortran</ 
> >>> DIV><DIV>F77=3D/usr/bi=
> >>> n/gfortran</DIV><DIV>F90=3D/usr/bin/gfortran</DIV><DIV><BR =
> >>> class=3D"khtml-block-placeholder"></DIV><DIV>The test was run on  
> >>> 32 =
> >>> process, 16 process, 8 process and 4 process - all of which  
> >>> generated =
> >>> this error message.</DIV><DIV><BR =
> >>> class=3D"khtml-block-placeholder"></DIV><DIV>Thanks,</ 
> >>> DIV><DIV>david</DIV>=
> >>> <DIV><BR class=3D"khtml-block-placeholder"></DIV><DIV><BR =
> >>> class=3D"khtml-block-placeholder"></DIV><DIV><BR><DIV> <SPAN =
> >>> class=3D"Apple-style-span" style=3D"border-collapse: separate; =
> >>> border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family:  
> >>> Helvetica; =
> >>> font-size: 12px; font-style: normal; font-variant: normal; font- 
> >>> weight: =
> >>> normal; letter-spacing: normal; line-height: normal; text-align:  
> >>> auto; =
> >>> -khtml-text-decorations-in-effect: none; text-indent: 0px; =
> >>> -apple-text-size-adjust: auto; text-transform: none; orphans: 2; =
> >>> white-space: normal; widows: 2; word-spacing: 0px; "><SPAN =
> >>> class=3D"Apple-style-span" style=3D"border-collapse: separate; =
> >>> border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family:  
> >>> Helvetica; =
> >>> font-size: 12px; font-style: normal; font-variant: normal; font- 
> >>> weight: =
> >>> normal; letter-spacing: normal; line-height: normal; text-align:  
> >>> auto; =
> >>> -khtml-text-decorations-in-effect: none; text-indent: 0px; =
> >>> -apple-text-size-adjust: auto; text-transform: none; orphans: 2; =
> >>> white-space: normal; widows: 2; word-spacing: 0px; "><SPAN =
> >>> class=3D"Apple-style-span" style=3D"border-collapse: separate; =
> >>> border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family:  
> >>> Helvetica; =
> >>> font-size: 12px; font-style: normal; font-variant: normal; font- 
> >>> weight: =
> >>> normal; letter-spacing: normal; line-height: normal; text-align:  
> >>> auto; =
> >>> -khtml-text-decorations-in-effect: none; text-indent: 0px; =
> >>> -apple-text-size-adjust: auto; text-transform: none; orphans: 2; =
> >>> white-space: normal; widows: 2; word-spacing: 0px; "><SPAN =
> >>> class=3D"Apple-style-span" style=3D"border-collapse: separate; =
> >>> border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family:  
> >>> Helvetica; =
> >>> font-size: 12px; font-style: normal; font-variant: normal; font- 
> >>> weight: =
> >>> normal; letter-spacing: normal; line-height: normal; text-align:  
> >>> auto; =
> >>> -khtml-text-decorations-in-effect: none; text-indent: 0px; =
> >>> -apple-text-size-adjust: auto; text-transform: none; orphans: 2; =
> >>> white-space: normal; widows: 2; word-spacing: 0px; "><SPAN =
> >>> class=3D"Apple-style-span" style=3D"border-collapse: separate; =
> >>> border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family:  
> >>> Helvetica; =
> >>> font-size: 12px; font-style: normal; font-variant: normal; font- 
> >>> weight: =
> >>> normal; letter-spacing: normal; line-height: normal; text-align:  
> >>> auto; =
> >>> -khtml-text-decorations-in-effect: none; text-indent: 0px; =
> >>> -apple-text-size-adjust: auto; text-transform: none; orphans: 2; =
> >>> white-space: normal; widows: 2; word-spacing: 0px; =
> >>> "><DIV>--</DIV><DIV>David Gunter</DIV><DIV>HPC-4: HPC  
> >>> Environments: =
> >>> Parallel Tools Team</DIV><DIV>Los Alamos National =
> >>> Laboratory</DIV></SPAN></SPAN><BR =
> >>> class=3D"Apple-interchange-newline"></SPAN></SPAN></SPAN> =
> >>> </DIV><BR><DIV><DIV>On Mar 21, 2007, at 4:50 PM, wei huang =
> >>> wrote:</DIV><BR class=3D"Apple-interchange-newline"><BLOCKQUOTE =
> >>> type=3D"cite"><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >>> margin-bottom: 0px; margin-left: 0px; ">Hi,</DIV><DIV  
> >>> style=3D"margin-top:=
> >>>  0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >>> min-height: 14px; "><BR></DIV><DIV style=3D"margin-top: 0px; =
> >>> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Thanks  
> >>> for =
> >>> letting us know this problem.</DIV><DIV style=3D"margin-top: 0px; =
> >>> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min- 
> >>> height: =
> >>> 14px; "><BR></DIV><DIV style=3D"margin-top: 0px; margin-right:  
> >>> 0px; =
> >>> margin-bottom: 0px; margin-left: 0px; ">We will try to reproduce  
> >>> the =
> >>> problem on our cluster. To help us on looking</DIV><DIV =
> >>> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >>> margin-left: 0px; ">into this problem, would you please let us  
> >>> know the =
> >>> following:</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >>> margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></ 
> >>> DIV><DIV =
> >>> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >>> margin-left: 0px; ">1) The exact CFLAGS you are using when  
> >>> configuring =
> >>> mvapich2 (are you using</DIV><DIV style=3D"margin-top: 0px; =
> >>> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">default =
> >>> scripts provided by us?)</DIV><DIV style=3D"margin-top: 0px; =
> >>> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">2) Any =
> >>> runtime environment variables you have setup up?</DIV><DIV =
> >>> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >>> margin-left: 0px; ">3) On how many nodes do you run the test?</ 
> >>> DIV><DIV =
> >>> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >>> margin-left: 0px; min-height: 14px; "><BR></DIV><DIV  
> >>> style=3D"margin-top: =
> >>> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >>> ">Thanks.</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >>> margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></ 
> >>> DIV><DIV =
> >>> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >>> margin-left: 0px; ">Regards,</DIV><DIV style=3D"margin-top: 0px; =
> >>> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Wei =
> >>> Huang</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >>> margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></ 
> >>> DIV><DIV =
> >>> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >>> margin-left: 0px; ">774 Dreese Lab, 2015 Neil Ave,</DIV><DIV =
> >>> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >>> margin-left: 0px; ">Dept. of Computer Science and Engineering</ 
> >>> DIV><DIV =
> >>> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >>> margin-left: 0px; ">Ohio State University</DIV><DIV  
> >>> style=3D"margin-top: =
> >>> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">OH =
> >>> 43210</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >>> margin-bottom: 0px; margin-left: 0px; ">Tel: (614)292-8501</ 
> >>> DIV><DIV =
> >>> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >>> margin-left: 0px; min-height: 14px; "><BR></DIV><DIV  
> >>> style=3D"margin-top: =
> >>> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >>> min-height: 14px; "><BR></DIV><DIV style=3D"margin-top: 0px; =
> >>> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">On  
> >>> Wed, 21 =
> >>> Mar 2007, David Gunter wrote:</DIV><DIV style=3D"margin-top: 0px; =
> >>> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min- 
> >>> height: =
> >>> 14px; "><BR></DIV> <BLOCKQUOTE type=3D"cite"><DIV style=3D"margin- 
> >>> top: =
> >>> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">I  
> >>> have =
> >>> built mvapich2 for an OFED-based IB Opteron cluster.<SPAN =
> >>> class=3D"Apple-converted-space">=A0 </SPAN>When</DIV><DIV =
> >>> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >>> margin-left: 0px; ">running the Intel MPI Benchmarks (IMB3) I keep =
> >>> seeing the following</DIV><DIV style=3D"margin-top: 0px; margin- 
> >>> right: =
> >>> 0px; margin-bottom: 0px; margin-left: 0px; ">errors messages in  
> >>> many =
> >>> spots, although the tests run to completion:</DIV><DIV =
> >>> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >>> margin-left: 0px; min-height: 14px; "><BR></DIV><DIV  
> >>> style=3D"margin-top: =
> >>> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;  
> >>> ">Internal =
> >>> error: communicator is already on free list</DIV><DIV  
> >>> style=3D"margin-top:=
> >>>  0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >>> min-height: 14px; "><BR></DIV><DIV style=3D"margin-top: 0px; =
> >>> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">What  
> >>> is this =
> >>> referring to?</DIV><DIV style=3D"margin-top: 0px; margin-right:  
> >>> 0px; =
> >>> margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></ 
> >>> DIV><DIV =
> >>> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >>> margin-left: 0px; ">Thanks.</DIV><DIV style=3D"margin-top: 0px; =
> >>> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >>> ">--david</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >>> margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></ 
> >>> DIV><DIV =
> >>> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >>> margin-left: 0px; ">--</DIV><DIV style=3D"margin-top: 0px; margin- 
> >>> right: =
> >>> 0px; margin-bottom: 0px; margin-left: 0px; ">David Gunter</ 
> >>> DIV><DIV =
> >>> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >>> margin-left: 0px; ">HPC-4: HPC Environments: Parallel Tools =
> >>> Team</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >>> margin-bottom: 0px; margin-left: 0px; ">Los Alamos National =
> >>> Laboratory</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >>> margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></ 
> >>> DIV><DIV =
> >>> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >>> margin-left: 0px; min-height: 14px; "><BR></DIV><DIV  
> >>> style=3D"margin-top: =
> >>> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >>> min-height: 14px; "><BR></DIV> </BLOCKQUOTE><DIV style=3D"margin- 
> >>> top: =
> >>> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >>> min-height: 14px; "><BR></DIV> =
> >>> </BLOCKQUOTE></DIV><BR></DIV></BODY></HTML>=
> >>>
> >>> --Apple-Mail-1-849288998--
> >>>
> >>> --===============1750880957==
> >>> Content-Type: text/plain; charset="us-ascii"
> >>> MIME-Version: 1.0
> >>> Content-Transfer-Encoding: 7bit
> >>> Content-Disposition: inline
> >>>
> >>> _______________________________________________
> >>> mvapich-discuss mailing list
> >>> mvapich-discuss at cse.ohio-state.edu
> >>> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
> >>>
> >>> --===============1750880957==--
> >>>
> >>
> >
> > _______________________________________________
> > mvapich-discuss mailing list
> > mvapich-discuss at cse.ohio-state.edu
> > http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
> 
> 
> --Apple-Mail-2-709635751
> Content-Transfer-Encoding: quoted-printable
> Content-Type: text/html;
> 	charset=ISO-8859-1
> 
> <HTML><BODY style=3D"word-wrap: break-word; -khtml-nbsp-mode: space; =
> -khtml-line-break: after-white-space; ">I am curious to know if you were =
> able to reproduce this problem, whether it has been fixed or =
> not.<DIV><BR =
> class=3D"khtml-block-placeholder"></DIV><DIV>Thanks,</DIV><DIV>david</DIV>=
> <DIV><DIV><SPAN class=3D"Apple-style-span" style=3D"border-collapse: =
> separate; border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family: =
> Helvetica; font-size: 12px; font-style: normal; font-variant: normal; =
> font-weight: normal; letter-spacing: normal; line-height: normal; =
> text-align: auto; -khtml-text-decorations-in-effect: none; text-indent: =
> 0px; -apple-text-size-adjust: auto; text-transform: none; orphans: 2; =
> white-space: normal; widows: 2; word-spacing: 0px; "><SPAN =
> class=3D"Apple-style-span" style=3D"border-collapse: separate; =
> border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family: Helvetica; =
> font-size: 12px; font-style: normal; font-variant: normal; font-weight: =
> normal; letter-spacing: normal; line-height: normal; text-align: auto; =
> -khtml-text-decorations-in-effect: none; text-indent: 0px; =
> -apple-text-size-adjust: auto; text-transform: none; orphans: 2; =
> white-space: normal; widows: 2; word-spacing: 0px; "><SPAN =
> class=3D"Apple-style-span" style=3D"border-collapse: separate; =
> border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family: Helvetica; =
> font-size: 12px; font-style: normal; font-variant: normal; font-weight: =
> normal; letter-spacing: normal; line-height: normal; text-align: auto; =
> -khtml-text-decorations-in-effect: none; text-indent: 0px; =
> -apple-text-size-adjust: auto; text-transform: none; orphans: 2; =
> white-space: normal; widows: 2; word-spacing: 0px; "><SPAN =
> class=3D"Apple-style-span" style=3D"border-collapse: separate; =
> border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family: Helvetica; =
> font-size: 12px; font-style: normal; font-variant: normal; font-weight: =
> normal; letter-spacing: normal; line-height: normal; text-align: auto; =
> -khtml-text-decorations-in-effect: none; text-indent: 0px; =
> -apple-text-size-adjust: auto; text-transform: none; orphans: 2; =
> white-space: normal; widows: 2; word-spacing: 0px; "><SPAN =
> class=3D"Apple-style-span" style=3D"border-collapse: separate; =
> border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family: Helvetica; =
> font-size: 12px; font-style: normal; font-variant: normal; font-weight: =
> normal; letter-spacing: normal; line-height: normal; text-align: auto; =
> -khtml-text-decorations-in-effect: none; text-indent: 0px; =
> -apple-text-size-adjust: auto; text-transform: none; orphans: 2; =
> white-space: normal; widows: 2; word-spacing: 0px; =
> "><DIV>--</DIV><DIV>David Gunter</DIV><DIV>HPC-4: HPC Environments: =
> Parallel Tools Team</DIV><DIV>Los Alamos National =
> Laboratory</DIV></SPAN></SPAN><BR =
> class=3D"Apple-interchange-newline"></SPAN></SPAN></SPAN> =
> </DIV><BR><DIV><DIV>On Mar 22, 2007, at 11:20 AM, David Gunter =
> wrote:</DIV><BR class=3D"Apple-interchange-newline"><BLOCKQUOTE =
> type=3D"cite"><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">I am using the released 0.9.8 =
> version.</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">-david</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: =
> 14px; "><BR></DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">On Mar 22, 2007, at 9:37 AM, =
> Dhabaleswar Panda wrote:</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: =
> 14px; "><BR></DIV> <BLOCKQUOTE type=3D"cite"><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Hi =
> David,</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">Thanks for this information. One more question - are =
> you using</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">MVAPICH2 0.9.8 `released' =
> version or the `branch' version (with some</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">recent =
> fixes). If you can let us know this information, it will help</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">us.</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: =
> 14px; "><BR></DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">Thanks,</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; min-height: 14px; "><BR></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">DK</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; min-height: 14px; "><BR></DIV> <BLOCKQUOTE =
> type=3D"cite"><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">I have recompiled mvapich2 =
> without using the --enable-debuginfo flag</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">and the =
> problem has gone away.<SPAN class=3D"Apple-converted-space">=A0 =
> </SPAN>However, I wish to have debuginfo</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">available to our TotalView users so hopefully this can be =
> resolved.</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">Here is the configuration that generates the error =
> message I saw</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">previously:</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; min-height: 14px; "><BR></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">./configure --prefix=3D/opt/mvapich2/mvapich2-0.9.8-gcc/ib =
> --with-</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">openib=3D/usr/local/ofed =
> --enable-romio --with-file-system=3Dufs+nfs</DIV><DIV style=3D"margin-top:=
>  0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">+panfs =
> --enable-sharedlibs=3Dgcc --enable-debuginfo --enable-fast --</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">with-mpe</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: =
> 14px; "><BR></DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">OFED is ofed-1.1</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; min-height: 14px; "><BR></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">CFLAGS=3D-D_X86_64_ -D_SMP_ -DUSE_HEADER_CACHING<SPAN =
> class=3D"Apple-converted-space">=A0 </SPAN>-DONE_SIDED -</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">DMPID_USE_SEQUENCE_NUMBERS<SPAN =
> class=3D"Apple-converted-space">=A0 </SPAN>-I/usr/local/ofed/include -O2 =
> <SPAN class=3D"Apple-converted-space">=A0 </SPAN>-</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">D_SHMEM_COLL_</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: =
> 14px; "><BR></DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">CC=3D/usr/bin/gcc</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">CXX=3D/usr/bin/g++</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">FC=3D/usr/bin/gfortran</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">F77=3D/usr/bin/gfortran</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">F90=3D/usr/bin/gfortran</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: =
> 14px; "><BR></DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">The test was run on 32 process, =
> 16 process, 8 process and 4 process -</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">all of =
> which generated this error message.</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: =
> 14px; "><BR></DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">Thanks,</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">david</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: =
> 14px; "><BR></DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; min-height: 14px; "><BR></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">--</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">David Gunter</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">HPC-4: HPC Environments: Parallel Tools =
> Team</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">Los Alamos National =
> Laboratory</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; min-height: 14px; "><BR></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">On Mar =
> 21, 2007, at 4:50 PM, wei huang wrote:</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> min-height: 14px; "><BR></DIV> <BLOCKQUOTE type=3D"cite"><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">Hi,</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: =
> 14px; "><BR></DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">Thanks for letting us know this =
> problem.</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">We will try to reproduce the problem on our cluster. =
> To help us on</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">looking</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">into this problem, would you please let us know the =
> following:</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">1) The exact CFLAGS you are using when configuring =
> mvapich2 (are</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">you using</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">default scripts provided by us?)</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">2) Any runtime environment variables you have setup =
> up?</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">3) On how many nodes do you run =
> the test?</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">Thanks.</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: =
> 14px; "><BR></DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">Regards,</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">Wei Huang</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: =
> 14px; "><BR></DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">774 Dreese Lab, 2015 Neil =
> Ave,</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">Dept. of Computer Science and =
> Engineering</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">Ohio State University</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">OH 43210</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Tel: =
> (614)292-8501</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; min-height: 14px; "><BR></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">On Wed, =
> 21 Mar 2007, David Gunter wrote:</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: =
> 14px; "><BR></DIV> <BLOCKQUOTE type=3D"cite"><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">I have =
> built mvapich2 for an OFED-based IB Opteron cluster.<SPAN =
> class=3D"Apple-converted-space">=A0 </SPAN>When</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">running the Intel MPI Benchmarks (IMB3) I keep =
> seeing the following</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">errors messages in many =
> spots, although the tests run to completion:</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; min-height: 14px; "><BR></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Internal =
> error: communicator is already on free list</DIV><DIV style=3D"margin-top:=
>  0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> min-height: 14px; "><BR></DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">What is this =
> referring to?</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">Thanks.</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">--david</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">--</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">David Gunter</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">HPC-4: HPC Environments: Parallel Tools =
> Team</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">Los Alamos National =
> Laboratory</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; min-height: 14px; "><BR></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> min-height: 14px; "><BR></DIV> </BLOCKQUOTE><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> min-height: 14px; "><BR></DIV> </BLOCKQUOTE><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> min-height: 14px; "><BR></DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: =
> 14px; "><BR></DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; =
> ">--Apple-Mail-1-849288998</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">Content-Transfer-Encoding: quoted-printable</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">Content-Type: text/html;</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; "><SPAN class=3D"Apple-tab-span" =
> style=3D"white-space:pre">	</SPAN>charset=3DISO-8859-1</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; min-height: 14px; "><BR></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">&lt;HTML&gt;&lt;BODY style=3D3D"word-wrap: break-word; =
> -khtml-nbsp-mode: space; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">-khtml-line-break: after-white-space; "&gt;I have recompiled mvapich2 =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">without using the =
> --enable-debuginfo flag and the problem has gone =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">away.=3DA0 However, I wish to have debuginfo =
> available to our TotalView =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">users so =
> hopefully this can be resolved.&lt;DIV&gt;&lt;BR =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; =
> ">class=3D3D"khtml-block-placeholder"&gt;&lt;/DIV&gt;&lt;DIV&gt;Here is =
> the configuration =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">that generates the error =
> message I saw previously:&lt;/DIV&gt;&lt;DIV&gt;&lt;BR =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; =
> ">class=3D3D"khtml-block-placeholder"&gt;&lt;/DIV&gt;&lt;DIV&gt;./configur=
> e =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; =
> ">--prefix=3D3D/opt/mvapich2/mvapich2-0.9.8-gcc/ib=3DA0--with-openib=3D3D/=
> usr/loca=3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">l/ofed --enable-romio =
> --with-file-system=3D3Dufs+nfs+panfs =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">--enable-sharedlibs=3D3Dgcc --enable-debuginfo --enable-fast =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; =
> ">--with-mpe&lt;/DIV&gt;&lt;DIV&gt;&lt;BR =
> class=3D3D"khtml-block-placeholder"&gt;&lt;/DIV&gt;&lt;DIV&gt;OFED=3D</DIV=
> ><DIV style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; "><SPAN class=3D"Apple-converted-space">=A0</SPAN>is =
> ofed-1.1&lt;/DIV&gt;&lt;DIV&gt;&lt;BR =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">class=3D3D"khtml-block-placeholder"&gt;&lt;/DIV&gt;&lt;DIV&gt;CFLAGS=3D3=
> D-D_X86_64_ -D_SMP_ =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">-DUSE_HEADER_CACHING=3DA0 -DONE_SIDED -DMPID_USE_SEQUENCE_NUMBERS=3DA0 =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">-I/usr/local/ofed/include =
> -O2=3DA0=3DA0 -D_SHMEM_COLL_&lt;/DIV&gt;&lt;DIV&gt;&lt;BR =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; =
> ">class=3D3D"khtml-block-placeholder"&gt;&lt;/DIV&gt;&lt;DIV&gt;CC=3D3D/us=
> r/bin/gcc&lt;/DIV&gt;&lt;DIV&gt;C=3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">XX=3D3D/usr/bin/g++&lt;/DIV&gt;&lt;DIV&gt;FC=3D3D/usr/bin/gfortran&lt;/D=
> IV&gt;&lt;DIV&gt;F77=3D3D/usr/bi=3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">n/gfortran&lt;/DIV&gt;&lt;DIV&gt;F90=3D3D/usr/bin/gfortran&lt;/DIV&gt;&l=
> t;DIV&gt;&lt;BR =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; =
> ">class=3D3D"khtml-block-placeholder"&gt;&lt;/DIV&gt;&lt;DIV&gt;The test =
> was run on 32 =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">process, 16 process, 8 process =
> and 4 process - all of which generated =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">this =
> error message.&lt;/DIV&gt;&lt;DIV&gt;&lt;BR =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; =
> ">class=3D3D"khtml-block-placeholder"&gt;&lt;/DIV&gt;&lt;DIV&gt;Thanks,&lt=
> ;/DIV&gt;&lt;DIV&gt;david&lt;/DIV&gt;=3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">&lt;DIV&gt;&lt;BR =
> class=3D3D"khtml-block-placeholder"&gt;&lt;/DIV&gt;&lt;DIV&gt;&lt;BR =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; =
> ">class=3D3D"khtml-block-placeholder"&gt;&lt;/DIV&gt;&lt;DIV&gt;&lt;BR&gt;=
> &lt;DIV&gt; &lt;SPAN =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">class=3D3D"Apple-style-span" style=3D3D"border-collapse: separate; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">border-spacing: 0px 0px; color: =
> rgb(0, 0, 0); font-family: Helvetica; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">font-size: 12px; font-style: normal; font-variant: normal; =
> font-weight: =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">normal; letter-spacing: normal; =
> line-height: normal; text-align: auto; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">-khtml-text-decorations-in-effect: none; text-indent: 0px; =3D</DIV><DIV=
>  style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">-apple-text-size-adjust: auto; text-transform: none; =
> orphans: 2; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">white-space: normal; widows: 2; =
> word-spacing: 0px; "&gt;&lt;SPAN =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">class=3D3D"Apple-style-span" style=3D3D"border-collapse: separate; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">border-spacing: 0px 0px; color: =
> rgb(0, 0, 0); font-family: Helvetica; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">font-size: 12px; font-style: normal; font-variant: normal; =
> font-weight: =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">normal; letter-spacing: normal; =
> line-height: normal; text-align: auto; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">-khtml-text-decorations-in-effect: none; text-indent: 0px; =3D</DIV><DIV=
>  style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">-apple-text-size-adjust: auto; text-transform: none; =
> orphans: 2; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">white-space: normal; widows: 2; =
> word-spacing: 0px; "&gt;&lt;SPAN =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">class=3D3D"Apple-style-span" style=3D3D"border-collapse: separate; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">border-spacing: 0px 0px; color: =
> rgb(0, 0, 0); font-family: Helvetica; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">font-size: 12px; font-style: normal; font-variant: normal; =
> font-weight: =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">normal; letter-spacing: normal; =
> line-height: normal; text-align: auto; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">-khtml-text-decorations-in-effect: none; text-indent: 0px; =3D</DIV><DIV=
>  style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">-apple-text-size-adjust: auto; text-transform: none; =
> orphans: 2; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">white-space: normal; widows: 2; =
> word-spacing: 0px; "&gt;&lt;SPAN =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">class=3D3D"Apple-style-span" style=3D3D"border-collapse: separate; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">border-spacing: 0px 0px; color: =
> rgb(0, 0, 0); font-family: Helvetica; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">font-size: 12px; font-style: normal; font-variant: normal; =
> font-weight: =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">normal; letter-spacing: normal; =
> line-height: normal; text-align: auto; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">-khtml-text-decorations-in-effect: none; text-indent: 0px; =3D</DIV><DIV=
>  style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">-apple-text-size-adjust: auto; text-transform: none; =
> orphans: 2; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">white-space: normal; widows: 2; =
> word-spacing: 0px; "&gt;&lt;SPAN =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">class=3D3D"Apple-style-span" style=3D3D"border-collapse: separate; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">border-spacing: 0px 0px; color: =
> rgb(0, 0, 0); font-family: Helvetica; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">font-size: 12px; font-style: normal; font-variant: normal; =
> font-weight: =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">normal; letter-spacing: normal; =
> line-height: normal; text-align: auto; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">-khtml-text-decorations-in-effect: none; text-indent: 0px; =3D</DIV><DIV=
>  style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">-apple-text-size-adjust: auto; text-transform: none; =
> orphans: 2; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">white-space: normal; widows: 2; =
> word-spacing: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; =
> ">"&gt;&lt;DIV&gt;--&lt;/DIV&gt;&lt;DIV&gt;David =
> Gunter&lt;/DIV&gt;&lt;DIV&gt;HPC-4: HPC Environments: =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">Parallel Tools Team&lt;/DIV&gt;&lt;DIV&gt;Los Alamos =
> National =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; =
> ">Laboratory&lt;/DIV&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;BR =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; =
> ">class=3D3D"Apple-interchange-newline"&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/=
> SPAN&gt; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; =
> ">&lt;/DIV&gt;&lt;BR&gt;&lt;DIV&gt;&lt;DIV&gt;On Mar 21, 2007, at 4:50 =
> PM, wei huang =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">wrote:&lt;/DIV&gt;&lt;BR =
> class=3D3D"Apple-interchange-newline"&gt;&lt;BLOCKQUOTE =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">type=3D3D"cite"&gt;&lt;DIV style=3D3D"margin-top: =
> 0px; margin-right: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">margin-bottom: 0px; margin-left: 0px; "&gt;Hi,&lt;/DIV&gt;&lt;DIV =
> style=3D3D"margin-top:=3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><SPAN =
> class=3D"Apple-converted-space">=A0</SPAN>0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">min-height: 14px; "&gt;&lt;BR&gt;&lt;/DIV&gt;&lt;DIV =
> style=3D3D"margin-top: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; "&gt;Thanks for =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">letting us know this problem.&lt;/DIV&gt;&lt;DIV =
> style=3D3D"margin-top: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; min-height: =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">14px; "&gt;&lt;BR&gt;&lt;/DIV&gt;&lt;DIV =
> style=3D3D"margin-top: 0px; margin-right: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-bottom: 0px; margin-left: 0px; "&gt;We will =
> try to reproduce the =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">problem on =
> our cluster. To help us on looking&lt;/DIV&gt;&lt;DIV =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">style=3D3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-left: =
> 0px; "&gt;into this problem, would you please let us know the =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">following:&lt;/DIV&gt;&lt;DIV =
> style=3D3D"margin-top: 0px; margin-right: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-bottom: 0px; margin-left: 0px; min-height: =
> 14px; "&gt;&lt;BR&gt;&lt;/DIV&gt;&lt;DIV =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">style=3D3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-left: =
> 0px; "&gt;1) The exact CFLAGS you are using when configuring =3D</DIV><DIV=
>  style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">mvapich2 (are you using&lt;/DIV&gt;&lt;DIV =
> style=3D3D"margin-top: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; "&gt;default =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">scripts provided by us?)&lt;/DIV&gt;&lt;DIV =
> style=3D3D"margin-top: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; "&gt;2) Any =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">runtime environment variables you have setup =
> up?&lt;/DIV&gt;&lt;DIV =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">style=3D3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-left: 0px; "&gt;3) On how =
> many nodes do you run the test?&lt;/DIV&gt;&lt;DIV =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">style=3D3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-left: =
> 0px; min-height: 14px; "&gt;&lt;BR&gt;&lt;/DIV&gt;&lt;DIV =
> style=3D3D"margin-top: =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">"&gt;Thanks.&lt;/DIV&gt;&lt;DIV style=3D3D"margin-top:=
>  0px; margin-right: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">margin-bottom: 0px; margin-left: 0px; min-height: 14px; =
> "&gt;&lt;BR&gt;&lt;/DIV&gt;&lt;DIV =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">style=3D3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-left: 0px; =
> "&gt;Regards,&lt;/DIV&gt;&lt;DIV style=3D3D"margin-top: 0px; =3D</DIV><DIV=
>  style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-right: 0px; margin-bottom: 0px; margin-left: =
> 0px; "&gt;Wei =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">Huang&lt;/DIV&gt;&lt;DIV =
> style=3D3D"margin-top: 0px; margin-right: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-bottom: 0px; margin-left: 0px; min-height: =
> 14px; "&gt;&lt;BR&gt;&lt;/DIV&gt;&lt;DIV =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">style=3D3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-left: =
> 0px; "&gt;774 Dreese Lab, 2015 Neil Ave,&lt;/DIV&gt;&lt;DIV =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">style=3D3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-left: =
> 0px; "&gt;Dept. of Computer Science and Engineering&lt;/DIV&gt;&lt;DIV =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">style=3D3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">margin-left: 0px; "&gt;Ohio State University&lt;/DIV&gt;&lt;DIV =
> style=3D3D"margin-top: =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "&gt;OH =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">43210&lt;/DIV&gt;&lt;DIV =
> style=3D3D"margin-top: 0px; margin-right: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-bottom: 0px; margin-left: 0px; "&gt;Tel: =
> (614)292-8501&lt;/DIV&gt;&lt;DIV =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">style=3D3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-left: 0px; min-height: =
> 14px; "&gt;&lt;BR&gt;&lt;/DIV&gt;&lt;DIV style=3D3D"margin-top: =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">min-height: 14px; "&gt;&lt;BR&gt;&lt;/DIV&gt;&lt;DIV =
> style=3D3D"margin-top: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; "&gt;On Wed, 21 =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">Mar 2007, David Gunter wrote:&lt;/DIV&gt;&lt;DIV =
> style=3D3D"margin-top: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; min-height: =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">14px; "&gt;&lt;BR&gt;&lt;/DIV&gt; &lt;BLOCKQUOTE =
> type=3D3D"cite"&gt;&lt;DIV style=3D3D"margin-top: =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; "&gt;I have =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">built =
> mvapich2 for an OFED-based IB Opteron cluster.&lt;SPAN =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">class=3D3D"Apple-converted-space"&gt;=3DA0 =
> &lt;/SPAN&gt;When&lt;/DIV&gt;&lt;DIV =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">style=3D3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-left: 0px; "&gt;running =
> the Intel MPI Benchmarks (IMB3) I keep =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">seeing =
> the following&lt;/DIV&gt;&lt;DIV style=3D3D"margin-top: 0px; =
> margin-right: =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">0px; margin-bottom: 0px; =
> margin-left: 0px; "&gt;errors messages in many =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">spots, although the tests run to =
> completion:&lt;/DIV&gt;&lt;DIV =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">style=3D3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-left: 0px; min-height: =
> 14px; "&gt;&lt;BR&gt;&lt;/DIV&gt;&lt;DIV style=3D3D"margin-top: =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; "&gt;Internal =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">error: communicator is already on free =
> list&lt;/DIV&gt;&lt;DIV style=3D3D"margin-top:=3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; "><SPAN class=3D"Apple-converted-space">=A0</SPAN>0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">min-height: 14px; "&gt;&lt;BR&gt;&lt;/DIV&gt;&lt;DIV =
> style=3D3D"margin-top: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; "&gt;What is this =3D</DIV><DIV=
>  style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">referring to?&lt;/DIV&gt;&lt;DIV =
> style=3D3D"margin-top: 0px; margin-right: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-bottom: 0px; margin-left: 0px; min-height: =
> 14px; "&gt;&lt;BR&gt;&lt;/DIV&gt;&lt;DIV =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">style=3D3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-left: =
> 0px; "&gt;Thanks.&lt;/DIV&gt;&lt;DIV style=3D3D"margin-top: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">"&gt;--david&lt;/DIV&gt;&lt;DIV style=3D3D"margin-top: 0px; =
> margin-right: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">margin-bottom: 0px; =
> margin-left: 0px; min-height: 14px; "&gt;&lt;BR&gt;&lt;/DIV&gt;&lt;DIV =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">style=3D3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">margin-left: 0px; "&gt;--&lt;/DIV&gt;&lt;DIV style=3D3D"margin-top: =
> 0px; margin-right: =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">0px; margin-bottom: 0px; =
> margin-left: 0px; "&gt;David Gunter&lt;/DIV&gt;&lt;DIV =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">style=3D3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-left: =
> 0px; "&gt;HPC-4: HPC Environments: Parallel Tools =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">Team&lt;/DIV&gt;&lt;DIV style=3D3D"margin-top: 0px; =
> margin-right: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">margin-bottom: 0px; =
> margin-left: 0px; "&gt;Los Alamos National =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">Laboratory&lt;/DIV&gt;&lt;DIV style=3D3D"margin-top: =
> 0px; margin-right: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">margin-bottom: 0px; margin-left: 0px; min-height: 14px; =
> "&gt;&lt;BR&gt;&lt;/DIV&gt;&lt;DIV =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">style=3D3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-left: 0px; min-height: =
> 14px; "&gt;&lt;BR&gt;&lt;/DIV&gt;&lt;DIV style=3D3D"margin-top: =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">min-height: 14px; "&gt;&lt;BR&gt;&lt;/DIV&gt; =
> &lt;/BLOCKQUOTE&gt;&lt;DIV style=3D3D"margin-top: =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">min-height: 14px; =
> "&gt;&lt;BR&gt;&lt;/DIV&gt; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">&lt;/BLOCKQUOTE&gt;&lt;/DIV&gt;&lt;BR&gt;&lt;/DIV&gt;&lt;/BODY&gt;&lt;/H=
> TML&gt;=3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">--Apple-Mail-1-849288998--</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; min-height: 14px; "><BR></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">--=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D1750880957=3D=3D</DIV><DI=
> V style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">Content-Type: text/plain; =
> charset=3D"us-ascii"</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">MIME-Version: 1.0</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">Content-Transfer-Encoding: 7bit</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">Content-Disposition: inline</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; min-height: 14px; "><BR></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">_______________________________________________</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">mvapich-discuss mailing list</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; "><A =
> href=3D"mailto:mvapich-discuss at cse.ohio-state.edu">mvapich-discuss at cse.ohi=
> o-state.edu</A></DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; "><A =
> href=3D"http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss">h=
> ttp://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss</A></DIV><D=
> IV style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; min-height: 14px; "><BR></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">--=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D1750880957=3D=3D--</DIV><=
> DIV style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; min-height: 14px; "><BR></DIV> </BLOCKQUOTE><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; min-height: 14px; "><BR></DIV> </BLOCKQUOTE><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; min-height: 14px; "><BR></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">_______________________________________________</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">mvapich-discuss mailing list</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; "><A =
> href=3D"mailto:mvapich-discuss at cse.ohio-state.edu">mvapich-discuss at cse.ohi=
> o-state.edu</A></DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; "><A =
> href=3D"http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss">h=
> ttp://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss</A></DIV> =
> </BLOCKQUOTE></DIV><BR></DIV></BODY></HTML>=
> 
> --Apple-Mail-2-709635751--
> 
> --===============1220798408==
> Content-Type: text/plain; charset="us-ascii"
> MIME-Version: 1.0
> Content-Transfer-Encoding: 7bit
> Content-Disposition: inline
> 
> _______________________________________________
> mvapich-discuss mailing list
> mvapich-discuss at cse.ohio-state.edu
> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
> 
> --===============1220798408==--
>

More information about the mvapich-discuss mailing list