[mvapich-discuss] MVAPICH2-0.9.8 internal errors
Dhabaleswar Panda
panda at cse.ohio-state.edu
Mon May 14 09:46:02 EDT 2007
Hi David,
We have investigated this issue further.
As indicated in the following URL by TotalView, the complete support
of TotalView for MPICH2 stack is only available for 1.0.5p4.
http://forum.etnus.com/gforum.cgi?post=305;guest=305853
MVAPICH2-0.9.8p2 is based on MPICH2 1.0.p3. We had an initial support
for TotalView in this MVAPICH2 0.9.8 release. This is leading to the
problem you are experiencing.
Our upcoming release of MVAPICH2-1.0 is being designed with the latest
release of MPICH2. We have done some quick test with this version and
we do not see this problem at all with --enable-debuginfo. This
version will be released in a couple of weeks.
Thus, if you wait for a couple of weeks, you will be able to use MVAPICH2
with all TotalView features.
Thanks,
DK
> I had a chance to download mvapich2-0.9.8p2 this morning and retest
> and yes, the problem still exists. I am running the IMB 3.0 suite
> and I see the following type of behavior:
>
> #----------------------------------------------------------------
> # Benchmarking Allreduce
> # #processes = 2
> # ( 2 additional processes waiting in MPI_Barrier)
> #----------------------------------------------------------------
> #bytes #repetitions t_min[usec] t_max[usec] t_avg[usec]
> 0 1000 0.03 0.03 0.03
> 4 1000 1.79 1.79 1.79
> 8 1000 1.81 1.82 1.81
> 16 1000 1.84 1.84 1.84
> 32 1000 1.93 1.93 1.93
> 64 1000 1.97 1.97 1.97
> 128 1000 2.12 2.13 2.13
> 256 1000 2.39 2.39 2.39
> 512 1000 2.87 2.88 2.87
> 1024 1000 3.90 3.90 3.90
> 2048 1000 5.97 5.97 5.97
> 4096 1000 10.38 10.38 10.38
> 8192 1000 19.20 19.20 19.20
> 16384 1000 36.35 36.35 36.35
> 32768 1000 69.93 69.93 69.93
> 65536 640 142.94 142.94 142.94
> 131072 320 281.42 281.42 281.42
> 262144 160 607.62 607.66 607.64
> 524288 80 1662.42 1662.94 1662.68
> 1048576 40 3636.55 3636.75 3636.65
> 2097152 20 8966.41 8974.30 8970.36
> 4194304 10 17831.90 17837.21 17834.56
> Internal error: communicator is already on free list
> Internal error: communicator is already on free list
>
>
> Wherever a test uses all the processes in a communicator (4 in this
> run) then there are no internal error messages. The number of such
> messages printed is always equal to the number of processes that did
> not participate in a particular test, i.e. the number that always
> wait in MPI_Barrier.
>
> Again, if I reconfigure with --disable-debuginfo then I also don't
> see these error messages.
>
> -david
>
> --
> David Gunter
> HPC-4: Parallel Tools Team
> Los Alamos National Laboratory
>
>
> On May 9, 2007, at 2:02 PM, Dhabaleswar Panda wrote:
>
> > Hi David,
> >
> > Do you see this problem with MVAPICH2 0.9.8p2 version? We had several
> > fixes to the released version during the last months. This latest
> > version (0.9.8p2) is available from the mvapich page. This version is
> > also included and is being tested with OFED 1.2. We have also done
> > testing of this latest version with OFED 1.1.
> >
> > Please let us know if you still see the error with this latest version
> > (p2) on your system and we will investigate this issue further.
> >
> > Thanks,
> >
> > DK
> >
> >
> >> I am curious to know if you were able to reproduce this problem,
> >> whether it has been fixed or not.
> >>
> >> Thanks,
> >> david
> >> --
> >> David Gunter
> >> HPC-4: HPC Environments: Parallel Tools Team
> >> Los Alamos National Laboratory
> >>
> >>
> >> On Mar 22, 2007, at 11:20 AM, David Gunter wrote:
> >>
> >>> I am using the released 0.9.8 version.
> >>>
> >>> -david
> >>>
> >>> On Mar 22, 2007, at 9:37 AM, Dhabaleswar Panda wrote:
> >>>
> >>>> Hi David,
> >>>>
> >>>> Thanks for this information. One more question - are you using
> >>>> MVAPICH2 0.9.8 `released' version or the `branch' version (with
> >>>> some
> >>>> recent fixes). If you can let us know this information, it will
> >>>> help
> >>>> us.
> >>>>
> >>>> Thanks,
> >>>>
> >>>> DK
> >>>>
> >>>>
> >>>>> I have recompiled mvapich2 without using the --enable-debuginfo
> >>>>> flag
> >>>>> and the problem has gone away. However, I wish to have debuginfo
> >>>>> available to our TotalView users so hopefully this can be
> >>>>> resolved.
> >>>>>
> >>>>> Here is the configuration that generates the error message I saw
> >>>>> previously:
> >>>>>
> >>>>> ./configure --prefix=/opt/mvapich2/mvapich2-0.9.8-gcc/ib --with-
> >>>>> openib=/usr/local/ofed --enable-romio --with-file-system=ufs+nfs
> >>>>> +panfs --enable-sharedlibs=gcc --enable-debuginfo --enable-fast --
> >>>>> with-mpe
> >>>>>
> >>>>> OFED is ofed-1.1
> >>>>>
> >>>>> CFLAGS=-D_X86_64_ -D_SMP_ -DUSE_HEADER_CACHING -DONE_SIDED -
> >>>>> DMPID_USE_SEQUENCE_NUMBERS -I/usr/local/ofed/include -O2 -
> >>>>> D_SHMEM_COLL_
> >>>>>
> >>>>> CC=/usr/bin/gcc
> >>>>> CXX=/usr/bin/g++
> >>>>> FC=/usr/bin/gfortran
> >>>>> F77=/usr/bin/gfortran
> >>>>> F90=/usr/bin/gfortran
> >>>>>
> >>>>> The test was run on 32 process, 16 process, 8 process and 4
> >>>>> process -
> >>>>> all of which generated this error message.
> >>>>>
> >>>>> Thanks,
> >>>>> david
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> David Gunter
> >>>>> HPC-4: HPC Environments: Parallel Tools Team
> >>>>> Los Alamos National Laboratory
> >>>>>
> >>>>>
> >>>>> On Mar 21, 2007, at 4:50 PM, wei huang wrote:
> >>>>>
> >>>>>> Hi,
> >>>>>>
> >>>>>> Thanks for letting us know this problem.
> >>>>>>
> >>>>>> We will try to reproduce the problem on our cluster. To help
> >>>>>> us on
> >>>>>> looking
> >>>>>> into this problem, would you please let us know the following:
> >>>>>>
> >>>>>> 1) The exact CFLAGS you are using when configuring mvapich2 (are
> >>>>>> you using
> >>>>>> default scripts provided by us?)
> >>>>>> 2) Any runtime environment variables you have setup up?
> >>>>>> 3) On how many nodes do you run the test?
> >>>>>>
> >>>>>> Thanks.
> >>>>>>
> >>>>>> Regards,
> >>>>>> Wei Huang
> >>>>>>
> >>>>>> 774 Dreese Lab, 2015 Neil Ave,
> >>>>>> Dept. of Computer Science and Engineering
> >>>>>> Ohio State University
> >>>>>> OH 43210
> >>>>>> Tel: (614)292-8501
> >>>>>>
> >>>>>>
> >>>>>> On Wed, 21 Mar 2007, David Gunter wrote:
> >>>>>>
> >>>>>>> I have built mvapich2 for an OFED-based IB Opteron cluster.
> >>>>>>> When
> >>>>>>> running the Intel MPI Benchmarks (IMB3) I keep seeing the
> >>>>>>> following
> >>>>>>> errors messages in many spots, although the tests run to
> >>>>>>> completion:
> >>>>>>>
> >>>>>>> Internal error: communicator is already on free list
> >>>>>>>
> >>>>>>> What is this referring to?
> >>>>>>>
> >>>>>>> Thanks.
> >>>>>>> --david
> >>>>>>>
> >>>>>>> --
> >>>>>>> David Gunter
> >>>>>>> HPC-4: HPC Environments: Parallel Tools Team
> >>>>>>> Los Alamos National Laboratory
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>> --Apple-Mail-1-849288998
> >>>>> Content-Transfer-Encoding: quoted-printable
> >>>>> Content-Type: text/html;
> >>>>> charset=ISO-8859-1
> >>>>>
> >>>>> <HTML><BODY style=3D"word-wrap: break-word; -khtml-nbsp-mode:
> >>>>> space; =
> >>>>> -khtml-line-break: after-white-space; ">I have recompiled
> >>>>> mvapich2 =
> >>>>> without using the --enable-debuginfo flag and the problem has
> >>>>> gone =
> >>>>> away.=A0 However, I wish to have debuginfo available to our
> >>>>> TotalView =
> >>>>> users so hopefully this can be resolved.<DIV><BR =
> >>>>> class=3D"khtml-block-placeholder"></DIV><DIV>Here is the
> >>>>> configuration =
> >>>>> that generates the error message I saw previously:</DIV><DIV><BR =
> >>>>> class=3D"khtml-block-placeholder"></DIV><DIV>./configure =
> >>>>> --prefix=3D/opt/mvapich2/mvapich2-0.9.8-gcc/ib=A0--with-openib=3D/
> >>>>> usr/loca=
> >>>>> l/ofed --enable-romio --with-file-system=3Dufs+nfs+panfs =
> >>>>> --enable-sharedlibs=3Dgcc --enable-debuginfo --enable-fast =
> >>>>> --with-mpe</DIV><DIV><BR class=3D"khtml-block-placeholder"></
> >>>>> DIV><DIV>OFED=
> >>>>> is ofed-1.1</DIV><DIV><BR =
> >>>>> class=3D"khtml-block-placeholder"></DIV><DIV>CFLAGS=3D-D_X86_64_ -
> >>>>> D_SMP_ =
> >>>>> -DUSE_HEADER_CACHING=A0 -DONE_SIDED -
> >>>>> DMPID_USE_SEQUENCE_NUMBERS=A0 =
> >>>>> -I/usr/local/ofed/include -O2=A0=A0 -D_SHMEM_COLL_</DIV><DIV><BR =
> >>>>> class=3D"khtml-block-placeholder"></DIV><DIV>CC=3D/usr/bin/gcc</
> >>>>> DIV><DIV>C=
> >>>>> XX=3D/usr/bin/g++</DIV><DIV>FC=3D/usr/bin/gfortran</
> >>>>> DIV><DIV>F77=3D/usr/bi=
> >>>>> n/gfortran</DIV><DIV>F90=3D/usr/bin/gfortran</DIV><DIV><BR =
> >>>>> class=3D"khtml-block-placeholder"></DIV><DIV>The test was run on
> >>>>> 32 =
> >>>>> process, 16 process, 8 process and 4 process - all of which
> >>>>> generated =
> >>>>> this error message.</DIV><DIV><BR =
> >>>>> class=3D"khtml-block-placeholder"></DIV><DIV>Thanks,</
> >>>>> DIV><DIV>david</DIV>=
> >>>>> <DIV><BR class=3D"khtml-block-placeholder"></DIV><DIV><BR =
> >>>>> class=3D"khtml-block-placeholder"></DIV><DIV><BR><DIV> <SPAN =
> >>>>> class=3D"Apple-style-span" style=3D"border-collapse: separate; =
> >>>>> border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family:
> >>>>> Helvetica; =
> >>>>> font-size: 12px; font-style: normal; font-variant: normal; font-
> >>>>> weight: =
> >>>>> normal; letter-spacing: normal; line-height: normal; text-align:
> >>>>> auto; =
> >>>>> -khtml-text-decorations-in-effect: none; text-indent: 0px; =
> >>>>> -apple-text-size-adjust: auto; text-transform: none; orphans: 2; =
> >>>>> white-space: normal; widows: 2; word-spacing: 0px; "><SPAN =
> >>>>> class=3D"Apple-style-span" style=3D"border-collapse: separate; =
> >>>>> border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family:
> >>>>> Helvetica; =
> >>>>> font-size: 12px; font-style: normal; font-variant: normal; font-
> >>>>> weight: =
> >>>>> normal; letter-spacing: normal; line-height: normal; text-align:
> >>>>> auto; =
> >>>>> -khtml-text-decorations-in-effect: none; text-indent: 0px; =
> >>>>> -apple-text-size-adjust: auto; text-transform: none; orphans: 2; =
> >>>>> white-space: normal; widows: 2; word-spacing: 0px; "><SPAN =
> >>>>> class=3D"Apple-style-span" style=3D"border-collapse: separate; =
> >>>>> border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family:
> >>>>> Helvetica; =
> >>>>> font-size: 12px; font-style: normal; font-variant: normal; font-
> >>>>> weight: =
> >>>>> normal; letter-spacing: normal; line-height: normal; text-align:
> >>>>> auto; =
> >>>>> -khtml-text-decorations-in-effect: none; text-indent: 0px; =
> >>>>> -apple-text-size-adjust: auto; text-transform: none; orphans: 2; =
> >>>>> white-space: normal; widows: 2; word-spacing: 0px; "><SPAN =
> >>>>> class=3D"Apple-style-span" style=3D"border-collapse: separate; =
> >>>>> border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family:
> >>>>> Helvetica; =
> >>>>> font-size: 12px; font-style: normal; font-variant: normal; font-
> >>>>> weight: =
> >>>>> normal; letter-spacing: normal; line-height: normal; text-align:
> >>>>> auto; =
> >>>>> -khtml-text-decorations-in-effect: none; text-indent: 0px; =
> >>>>> -apple-text-size-adjust: auto; text-transform: none; orphans: 2; =
> >>>>> white-space: normal; widows: 2; word-spacing: 0px; "><SPAN =
> >>>>> class=3D"Apple-style-span" style=3D"border-collapse: separate; =
> >>>>> border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family:
> >>>>> Helvetica; =
> >>>>> font-size: 12px; font-style: normal; font-variant: normal; font-
> >>>>> weight: =
> >>>>> normal; letter-spacing: normal; line-height: normal; text-align:
> >>>>> auto; =
> >>>>> -khtml-text-decorations-in-effect: none; text-indent: 0px; =
> >>>>> -apple-text-size-adjust: auto; text-transform: none; orphans: 2; =
> >>>>> white-space: normal; widows: 2; word-spacing: 0px; =
> >>>>> "><DIV>--</DIV><DIV>David Gunter</DIV><DIV>HPC-4: HPC
> >>>>> Environments: =
> >>>>> Parallel Tools Team</DIV><DIV>Los Alamos National =
> >>>>> Laboratory</DIV></SPAN></SPAN><BR =
> >>>>> class=3D"Apple-interchange-newline"></SPAN></SPAN></SPAN> =
> >>>>> </DIV><BR><DIV><DIV>On Mar 21, 2007, at 4:50 PM, wei huang =
> >>>>> wrote:</DIV><BR class=3D"Apple-interchange-newline"><BLOCKQUOTE =
> >>>>> type=3D"cite"><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >>>>> margin-bottom: 0px; margin-left: 0px; ">Hi,</DIV><DIV
> >>>>> style=3D"margin-top:=
> >>>>> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >>>>> min-height: 14px; "><BR></DIV><DIV style=3D"margin-top: 0px; =
> >>>>> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Thanks
> >>>>> for =
> >>>>> letting us know this problem.</DIV><DIV style=3D"margin-top:
> >>>>> 0px; =
> >>>>> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-
> >>>>> height: =
> >>>>> 14px; "><BR></DIV><DIV style=3D"margin-top: 0px; margin-right:
> >>>>> 0px; =
> >>>>> margin-bottom: 0px; margin-left: 0px; ">We will try to reproduce
> >>>>> the =
> >>>>> problem on our cluster. To help us on looking</DIV><DIV =
> >>>>> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >>>>> margin-left: 0px; ">into this problem, would you please let us
> >>>>> know the =
> >>>>> following:</DIV><DIV style=3D"margin-top: 0px; margin-right:
> >>>>> 0px; =
> >>>>> margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></
> >>>>> DIV><DIV =
> >>>>> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >>>>> margin-left: 0px; ">1) The exact CFLAGS you are using when
> >>>>> configuring =
> >>>>> mvapich2 (are you using</DIV><DIV style=3D"margin-top: 0px; =
> >>>>> margin-right: 0px; margin-bottom: 0px; margin-left: 0px;
> >>>>> ">default =
> >>>>> scripts provided by us?)</DIV><DIV style=3D"margin-top: 0px; =
> >>>>> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">2)
> >>>>> Any =
> >>>>> runtime environment variables you have setup up?</DIV><DIV =
> >>>>> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >>>>> margin-left: 0px; ">3) On how many nodes do you run the test?</
> >>>>> DIV><DIV =
> >>>>> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >>>>> margin-left: 0px; min-height: 14px; "><BR></DIV><DIV
> >>>>> style=3D"margin-top: =
> >>>>> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >>>>> ">Thanks.</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >>>>> margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></
> >>>>> DIV><DIV =
> >>>>> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >>>>> margin-left: 0px; ">Regards,</DIV><DIV style=3D"margin-top: 0px; =
> >>>>> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Wei =
> >>>>> Huang</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >>>>> margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></
> >>>>> DIV><DIV =
> >>>>> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >>>>> margin-left: 0px; ">774 Dreese Lab, 2015 Neil Ave,</DIV><DIV =
> >>>>> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >>>>> margin-left: 0px; ">Dept. of Computer Science and Engineering</
> >>>>> DIV><DIV =
> >>>>> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >>>>> margin-left: 0px; ">Ohio State University</DIV><DIV
> >>>>> style=3D"margin-top: =
> >>>>> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;
> >>>>> ">OH =
> >>>>> 43210</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >>>>> margin-bottom: 0px; margin-left: 0px; ">Tel: (614)292-8501</
> >>>>> DIV><DIV =
> >>>>> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >>>>> margin-left: 0px; min-height: 14px; "><BR></DIV><DIV
> >>>>> style=3D"margin-top: =
> >>>>> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >>>>> min-height: 14px; "><BR></DIV><DIV style=3D"margin-top: 0px; =
> >>>>> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">On
> >>>>> Wed, 21 =
> >>>>> Mar 2007, David Gunter wrote:</DIV><DIV style=3D"margin-top:
> >>>>> 0px; =
> >>>>> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-
> >>>>> height: =
> >>>>> 14px; "><BR></DIV> <BLOCKQUOTE type=3D"cite"><DIV style=3D"margin-
> >>>>> top: =
> >>>>> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">I
> >>>>> have =
> >>>>> built mvapich2 for an OFED-based IB Opteron cluster.<SPAN =
> >>>>> class=3D"Apple-converted-space">=A0 </SPAN>When</DIV><DIV =
> >>>>> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >>>>> margin-left: 0px; ">running the Intel MPI Benchmarks (IMB3) I
> >>>>> keep =
> >>>>> seeing the following</DIV><DIV style=3D"margin-top: 0px; margin-
> >>>>> right: =
> >>>>> 0px; margin-bottom: 0px; margin-left: 0px; ">errors messages in
> >>>>> many =
> >>>>> spots, although the tests run to completion:</DIV><DIV =
> >>>>> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >>>>> margin-left: 0px; min-height: 14px; "><BR></DIV><DIV
> >>>>> style=3D"margin-top: =
> >>>>> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;
> >>>>> ">Internal =
> >>>>> error: communicator is already on free list</DIV><DIV
> >>>>> style=3D"margin-top:=
> >>>>> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >>>>> min-height: 14px; "><BR></DIV><DIV style=3D"margin-top: 0px; =
> >>>>> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">What
> >>>>> is this =
> >>>>> referring to?</DIV><DIV style=3D"margin-top: 0px; margin-right:
> >>>>> 0px; =
> >>>>> margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></
> >>>>> DIV><DIV =
> >>>>> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >>>>> margin-left: 0px; ">Thanks.</DIV><DIV style=3D"margin-top: 0px; =
> >>>>> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >>>>> ">--david</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >>>>> margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></
> >>>>> DIV><DIV =
> >>>>> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >>>>> margin-left: 0px; ">--</DIV><DIV style=3D"margin-top: 0px; margin-
> >>>>> right: =
> >>>>> 0px; margin-bottom: 0px; margin-left: 0px; ">David Gunter</
> >>>>> DIV><DIV =
> >>>>> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >>>>> margin-left: 0px; ">HPC-4: HPC Environments: Parallel Tools =
> >>>>> Team</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >>>>> margin-bottom: 0px; margin-left: 0px; ">Los Alamos National =
> >>>>> Laboratory</DIV><DIV style=3D"margin-top: 0px; margin-right:
> >>>>> 0px; =
> >>>>> margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></
> >>>>> DIV><DIV =
> >>>>> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >>>>> margin-left: 0px; min-height: 14px; "><BR></DIV><DIV
> >>>>> style=3D"margin-top: =
> >>>>> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >>>>> min-height: 14px; "><BR></DIV> </BLOCKQUOTE><DIV style=3D"margin-
> >>>>> top: =
> >>>>> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >>>>> min-height: 14px; "><BR></DIV> =
> >>>>> </BLOCKQUOTE></DIV><BR></DIV></BODY></HTML>=
> >>>>>
> >>>>> --Apple-Mail-1-849288998--
> >>>>>
> >>>>> --===============1750880957==
> >>>>> Content-Type: text/plain; charset="us-ascii"
> >>>>> MIME-Version: 1.0
> >>>>> Content-Transfer-Encoding: 7bit
> >>>>> Content-Disposition: inline
> >>>>>
> >>>>> _______________________________________________
> >>>>> mvapich-discuss mailing list
> >>>>> mvapich-discuss at cse.ohio-state.edu
> >>>>> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
> >>>>>
> >>>>> --===============1750880957==--
> >>>>>
> >>>>
> >>>
> >>> _______________________________________________
> >>> mvapich-discuss mailing list
> >>> mvapich-discuss at cse.ohio-state.edu
> >>> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
> >>
> >>
> >> --Apple-Mail-2-709635751
> >> Content-Transfer-Encoding: quoted-printable
> >> Content-Type: text/html;
> >> charset=ISO-8859-1
> >>
> >> <HTML><BODY style=3D"word-wrap: break-word; -khtml-nbsp-mode:
> >> space; =
> >> -khtml-line-break: after-white-space; ">I am curious to know if
> >> you were =
> >> able to reproduce this problem, whether it has been fixed or =
> >> not.<DIV><BR =
> >> class=3D"khtml-block-placeholder"></DIV><DIV>Thanks,</
> >> DIV><DIV>david</DIV>=
> >> <DIV><DIV><SPAN class=3D"Apple-style-span" style=3D"border-
> >> collapse: =
> >> separate; border-spacing: 0px 0px; color: rgb(0, 0, 0); font-
> >> family: =
> >> Helvetica; font-size: 12px; font-style: normal; font-variant:
> >> normal; =
> >> font-weight: normal; letter-spacing: normal; line-height: normal; =
> >> text-align: auto; -khtml-text-decorations-in-effect: none; text-
> >> indent: =
> >> 0px; -apple-text-size-adjust: auto; text-transform: none; orphans:
> >> 2; =
> >> white-space: normal; widows: 2; word-spacing: 0px; "><SPAN =
> >> class=3D"Apple-style-span" style=3D"border-collapse: separate; =
> >> border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family:
> >> Helvetica; =
> >> font-size: 12px; font-style: normal; font-variant: normal; font-
> >> weight: =
> >> normal; letter-spacing: normal; line-height: normal; text-align:
> >> auto; =
> >> -khtml-text-decorations-in-effect: none; text-indent: 0px; =
> >> -apple-text-size-adjust: auto; text-transform: none; orphans: 2; =
> >> white-space: normal; widows: 2; word-spacing: 0px; "><SPAN =
> >> class=3D"Apple-style-span" style=3D"border-collapse: separate; =
> >> border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family:
> >> Helvetica; =
> >> font-size: 12px; font-style: normal; font-variant: normal; font-
> >> weight: =
> >> normal; letter-spacing: normal; line-height: normal; text-align:
> >> auto; =
> >> -khtml-text-decorations-in-effect: none; text-indent: 0px; =
> >> -apple-text-size-adjust: auto; text-transform: none; orphans: 2; =
> >> white-space: normal; widows: 2; word-spacing: 0px; "><SPAN =
> >> class=3D"Apple-style-span" style=3D"border-collapse: separate; =
> >> border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family:
> >> Helvetica; =
> >> font-size: 12px; font-style: normal; font-variant: normal; font-
> >> weight: =
> >> normal; letter-spacing: normal; line-height: normal; text-align:
> >> auto; =
> >> -khtml-text-decorations-in-effect: none; text-indent: 0px; =
> >> -apple-text-size-adjust: auto; text-transform: none; orphans: 2; =
> >> white-space: normal; widows: 2; word-spacing: 0px; "><SPAN =
> >> class=3D"Apple-style-span" style=3D"border-collapse: separate; =
> >> border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family:
> >> Helvetica; =
> >> font-size: 12px; font-style: normal; font-variant: normal; font-
> >> weight: =
> >> normal; letter-spacing: normal; line-height: normal; text-align:
> >> auto; =
> >> -khtml-text-decorations-in-effect: none; text-indent: 0px; =
> >> -apple-text-size-adjust: auto; text-transform: none; orphans: 2; =
> >> white-space: normal; widows: 2; word-spacing: 0px; =
> >> "><DIV>--</DIV><DIV>David Gunter</DIV><DIV>HPC-4: HPC Environments: =
> >> Parallel Tools Team</DIV><DIV>Los Alamos National =
> >> Laboratory</DIV></SPAN></SPAN><BR =
> >> class=3D"Apple-interchange-newline"></SPAN></SPAN></SPAN> =
> >> </DIV><BR><DIV><DIV>On Mar 22, 2007, at 11:20 AM, David Gunter =
> >> wrote:</DIV><BR class=3D"Apple-interchange-newline"><BLOCKQUOTE =
> >> type=3D"cite"><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >> margin-bottom: 0px; margin-left: 0px; ">I am using the released
> >> 0.9.8 =
> >> version.</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >> margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></
> >> DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">-david</DIV><DIV style=3D"margin-top: 0px; =
> >> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-
> >> height: =
> >> 14px; "><BR></DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >> margin-bottom: 0px; margin-left: 0px; ">On Mar 22, 2007, at 9:37
> >> AM, =
> >> Dhabaleswar Panda wrote:</DIV><DIV style=3D"margin-top: 0px; =
> >> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-
> >> height: =
> >> 14px; "><BR></DIV> <BLOCKQUOTE type=3D"cite"><DIV style=3D"margin-
> >> top: =
> >> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Hi =
> >> David,</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >> margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></
> >> DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">Thanks for this information. One more question
> >> - are =
> >> you using</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >> margin-bottom: 0px; margin-left: 0px; ">MVAPICH2 0.9.8 `released' =
> >> version or the `branch' version (with some</DIV><DIV
> >> style=3D"margin-top: =
> >> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;
> >> ">recent =
> >> fixes). If you can let us know this information, it will help</
> >> DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">us.</DIV><DIV style=3D"margin-top: 0px; =
> >> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-
> >> height: =
> >> 14px; "><BR></DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >> margin-bottom: 0px; margin-left: 0px; ">Thanks,</DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; min-height: 14px; "><BR></DIV><DIV
> >> style=3D"margin-top: =
> >> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >> ">DK</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >> margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></
> >> DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; min-height: 14px; "><BR></DIV> <BLOCKQUOTE =
> >> type=3D"cite"><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >> margin-bottom: 0px; margin-left: 0px; ">I have recompiled mvapich2 =
> >> without using the --enable-debuginfo flag</DIV><DIV
> >> style=3D"margin-top: =
> >> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;
> >> ">and the =
> >> problem has gone away.<SPAN class=3D"Apple-converted-space">=A0 =
> >> </SPAN>However, I wish to have debuginfo</DIV><DIV style=3D"margin-
> >> top: =
> >> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >> ">available to our TotalView users so hopefully this can be =
> >> resolved.</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >> margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></
> >> DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">Here is the configuration that generates the
> >> error =
> >> message I saw</DIV><DIV style=3D"margin-top: 0px; margin-right:
> >> 0px; =
> >> margin-bottom: 0px; margin-left: 0px; ">previously:</DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; min-height: 14px; "><BR></DIV><DIV
> >> style=3D"margin-top: =
> >> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >> ">./configure --prefix=3D/opt/mvapich2/mvapich2-0.9.8-gcc/ib =
> >> --with-</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >> margin-bottom: 0px; margin-left: 0px; ">openib=3D/usr/local/ofed =
> >> --enable-romio --with-file-system=3Dufs+nfs</DIV><DIV
> >> style=3D"margin-top:=
> >> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">
> >> +panfs =
> >> --enable-sharedlibs=3Dgcc --enable-debuginfo --enable-fast --</
> >> DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">with-mpe</DIV><DIV style=3D"margin-top: 0px; =
> >> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-
> >> height: =
> >> 14px; "><BR></DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >> margin-bottom: 0px; margin-left: 0px; ">OFED is ofed-1.1</DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; min-height: 14px; "><BR></DIV><DIV
> >> style=3D"margin-top: =
> >> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >> ">CFLAGS=3D-D_X86_64_ -D_SMP_ -DUSE_HEADER_CACHING<SPAN =
> >> class=3D"Apple-converted-space">=A0 </SPAN>-DONE_SIDED -</DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">DMPID_USE_SEQUENCE_NUMBERS<SPAN =
> >> class=3D"Apple-converted-space">=A0 </SPAN>-I/usr/local/ofed/
> >> include -O2 =
> >> <SPAN class=3D"Apple-converted-space">=A0 </SPAN>-</DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">D_SHMEM_COLL_</DIV><DIV style=3D"margin-top:
> >> 0px; =
> >> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-
> >> height: =
> >> 14px; "><BR></DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >> margin-bottom: 0px; margin-left: 0px; ">CC=3D/usr/bin/gcc</DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">CXX=3D/usr/bin/g++</DIV><DIV style=3D"margin-
> >> top: =
> >> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >> ">FC=3D/usr/bin/gfortran</DIV><DIV style=3D"margin-top: 0px; =
> >> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >> ">F77=3D/usr/bin/gfortran</DIV><DIV style=3D"margin-top: 0px; =
> >> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >> ">F90=3D/usr/bin/gfortran</DIV><DIV style=3D"margin-top: 0px; =
> >> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-
> >> height: =
> >> 14px; "><BR></DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >> margin-bottom: 0px; margin-left: 0px; ">The test was run on 32
> >> process, =
> >> 16 process, 8 process and 4 process -</DIV><DIV style=3D"margin-
> >> top: =
> >> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;
> >> ">all of =
> >> which generated this error message.</DIV><DIV style=3D"margin-top:
> >> 0px; =
> >> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-
> >> height: =
> >> 14px; "><BR></DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >> margin-bottom: 0px; margin-left: 0px; ">Thanks,</DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">david</DIV><DIV style=3D"margin-top: 0px; =
> >> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-
> >> height: =
> >> 14px; "><BR></DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >> margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></
> >> DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; min-height: 14px; "><BR></DIV><DIV
> >> style=3D"margin-top: =
> >> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >> ">--</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >> margin-bottom: 0px; margin-left: 0px; ">David Gunter</DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">HPC-4: HPC Environments: Parallel Tools =
> >> Team</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >> margin-bottom: 0px; margin-left: 0px; ">Los Alamos National =
> >> Laboratory</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >> margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></
> >> DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; min-height: 14px; "><BR></DIV><DIV
> >> style=3D"margin-top: =
> >> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">On
> >> Mar =
> >> 21, 2007, at 4:50 PM, wei huang wrote:</DIV><DIV style=3D"margin-
> >> top: =
> >> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >> min-height: 14px; "><BR></DIV> <BLOCKQUOTE type=3D"cite"><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">Hi,</DIV><DIV style=3D"margin-top: 0px; =
> >> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-
> >> height: =
> >> 14px; "><BR></DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >> margin-bottom: 0px; margin-left: 0px; ">Thanks for letting us know
> >> this =
> >> problem.</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >> margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></
> >> DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">We will try to reproduce the problem on our
> >> cluster. =
> >> To help us on</DIV><DIV style=3D"margin-top: 0px; margin-right:
> >> 0px; =
> >> margin-bottom: 0px; margin-left: 0px; ">looking</DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">into this problem, would you please let us
> >> know the =
> >> following:</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >> margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></
> >> DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">1) The exact CFLAGS you are using when
> >> configuring =
> >> mvapich2 (are</DIV><DIV style=3D"margin-top: 0px; margin-right:
> >> 0px; =
> >> margin-bottom: 0px; margin-left: 0px; ">you using</DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">default scripts provided by us?)</DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">2) Any runtime environment variables you have
> >> setup =
> >> up?</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >> margin-bottom: 0px; margin-left: 0px; ">3) On how many nodes do
> >> you run =
> >> the test?</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >> margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></
> >> DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">Thanks.</DIV><DIV style=3D"margin-top: 0px; =
> >> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-
> >> height: =
> >> 14px; "><BR></DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >> margin-bottom: 0px; margin-left: 0px; ">Regards,</DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">Wei Huang</DIV><DIV style=3D"margin-top: 0px; =
> >> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-
> >> height: =
> >> 14px; "><BR></DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >> margin-bottom: 0px; margin-left: 0px; ">774 Dreese Lab, 2015 Neil =
> >> Ave,</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >> margin-bottom: 0px; margin-left: 0px; ">Dept. of Computer Science
> >> and =
> >> Engineering</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >> margin-bottom: 0px; margin-left: 0px; ">Ohio State University</
> >> DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">OH 43210</DIV><DIV style=3D"margin-top: 0px; =
> >> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Tel: =
> >> (614)292-8501</DIV><DIV style=3D"margin-top: 0px; margin-right:
> >> 0px; =
> >> margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></
> >> DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; min-height: 14px; "><BR></DIV><DIV
> >> style=3D"margin-top: =
> >> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">On
> >> Wed, =
> >> 21 Mar 2007, David Gunter wrote:</DIV><DIV style=3D"margin-top:
> >> 0px; =
> >> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-
> >> height: =
> >> 14px; "><BR></DIV> <BLOCKQUOTE type=3D"cite"><DIV style=3D"margin-
> >> top: =
> >> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">I
> >> have =
> >> built mvapich2 for an OFED-based IB Opteron cluster.<SPAN =
> >> class=3D"Apple-converted-space">=A0 </SPAN>When</DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">running the Intel MPI Benchmarks (IMB3) I keep =
> >> seeing the following</DIV><DIV style=3D"margin-top: 0px; margin-
> >> right: =
> >> 0px; margin-bottom: 0px; margin-left: 0px; ">errors messages in
> >> many =
> >> spots, although the tests run to completion:</DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; min-height: 14px; "><BR></DIV><DIV
> >> style=3D"margin-top: =
> >> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;
> >> ">Internal =
> >> error: communicator is already on free list</DIV><DIV
> >> style=3D"margin-top:=
> >> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >> min-height: 14px; "><BR></DIV><DIV style=3D"margin-top: 0px; =
> >> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">What is
> >> this =
> >> referring to?</DIV><DIV style=3D"margin-top: 0px; margin-right:
> >> 0px; =
> >> margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></
> >> DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">Thanks.</DIV><DIV style=3D"margin-top: 0px; =
> >> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >> ">--david</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >> margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></
> >> DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">--</DIV><DIV style=3D"margin-top: 0px; margin-
> >> right: =
> >> 0px; margin-bottom: 0px; margin-left: 0px; ">David Gunter</DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">HPC-4: HPC Environments: Parallel Tools =
> >> Team</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >> margin-bottom: 0px; margin-left: 0px; ">Los Alamos National =
> >> Laboratory</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >> margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></
> >> DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; min-height: 14px; "><BR></DIV><DIV
> >> style=3D"margin-top: =
> >> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >> min-height: 14px; "><BR></DIV> </BLOCKQUOTE><DIV style=3D"margin-
> >> top: =
> >> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >> min-height: 14px; "><BR></DIV> </BLOCKQUOTE><DIV style=3D"margin-
> >> top: =
> >> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >> min-height: 14px; "><BR></DIV><DIV style=3D"margin-top: 0px; =
> >> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-
> >> height: =
> >> 14px; "><BR></DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >> margin-bottom: 0px; margin-left: 0px; =
> >> ">--Apple-Mail-1-849288998</DIV><DIV style=3D"margin-top: 0px; =
> >> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >> ">Content-Transfer-Encoding: quoted-printable</DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">Content-Type: text/html;</DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; "><SPAN class=3D"Apple-tab-span" =
> >> style=3D"white-space:pre"> </SPAN>charset=3DISO-8859-1</DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; min-height: 14px; "><BR></DIV><DIV
> >> style=3D"margin-top: =
> >> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >> "><HTML><BODY style=3D3D"word-wrap: break-word; =
> >> -khtml-nbsp-mode: space; =3D</DIV><DIV style=3D"margin-top: 0px; =
> >> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >> ">-khtml-line-break: after-white-space; ">I have recompiled
> >> mvapich2 =
> >> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >> margin-bottom: 0px; margin-left: 0px; ">without using the =
> >> --enable-debuginfo flag and the problem has gone =3D</DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">away.=3DA0 However, I wish to have debuginfo =
> >> available to our TotalView =3D</DIV><DIV style=3D"margin-top: 0px; =
> >> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">users so =
> >> hopefully this can be resolved.<DIV><BR =3D</DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; =
> >> ">class=3D3D"khtml-block-placeholder"></
> >> DIV><DIV>Here is =
> >> the configuration =3D</DIV><DIV style=3D"margin-top: 0px; margin-
> >> right: =
> >> 0px; margin-bottom: 0px; margin-left: 0px; ">that generates the
> >> error =
> >> message I saw previously:</DIV><DIV><BR =3D</
> >> DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; =
> >> ">class=3D3D"khtml-block-placeholder"></DIV><DIV>./
> >> configur=
> >> e =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >> margin-bottom: 0px; margin-left: 0px; =
> >> ">--prefix=3D3D/opt/mvapich2/mvapich2-0.9.8-gcc/ib=3DA0--with-
> >> openib=3D3D/=
> >> usr/loca=3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >> margin-bottom: 0px; margin-left: 0px; ">l/ofed --enable-romio =
> >> --with-file-system=3D3Dufs+nfs+panfs =3D</DIV><DIV style=3D"margin-
> >> top: =
> >> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >> ">--enable-sharedlibs=3D3Dgcc --enable-debuginfo --enable-fast =
> >> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >> margin-bottom: 0px; margin-left: 0px; =
> >> ">--with-mpe</DIV><DIV><BR =
> >> class=3D3D"khtml-block-placeholder"></
> >> DIV><DIV>OFED=3D</DIV=
> >>> <DIV style=3D"margin-top: 0px; margin-right: 0px; margin-bottom:
> >>> 0px; =
> >> margin-left: 0px; "><SPAN class=3D"Apple-converted-space">=A0</
> >> SPAN>is =
> >> ofed-1.1</DIV><DIV><BR =3D</DIV><DIV
> >> style=3D"margin-top: =
> >> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >> ">class=3D3D"khtml-block-placeholder"></
> >> DIV><DIV>CFLAGS=3D3=
> >> D-D_X86_64_ -D_SMP_ =3D</DIV><DIV style=3D"margin-top: 0px; =
> >> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >> ">-DUSE_HEADER_CACHING=3DA0 -DONE_SIDED -
> >> DMPID_USE_SEQUENCE_NUMBERS=3DA0 =
> >> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >> margin-bottom: 0px; margin-left: 0px; ">-I/usr/local/ofed/include =
> >> -O2=3DA0=3DA0 -D_SHMEM_COLL_</DIV><DIV><BR =3D</
> >> DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; =
> >> ">class=3D3D"khtml-block-placeholder"></
> >> DIV><DIV>CC=3D3D/us=
> >> r/bin/gcc</DIV><DIV>C=3D</DIV><DIV style=3D"margin-
> >> top: 0px; =
> >> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >> ">XX=3D3D/usr/bin/g++</DIV><DIV>FC=3D3D/usr/bin/
> >> gfortran</D=
> >> IV><DIV>F77=3D3D/usr/bi=3D</DIV><DIV style=3D"margin-top:
> >> 0px; =
> >> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >> ">n/gfortran</DIV><DIV>F90=3D3D/usr/bin/gfortran</
> >> DIV>&l=
> >> t;DIV><BR =3D</DIV><DIV style=3D"margin-top: 0px; margin-
> >> right: =
> >> 0px; margin-bottom: 0px; margin-left: 0px; =
> >> ">class=3D3D"khtml-block-placeholder"></
> >> DIV><DIV>The test =
> >> was run on 32 =3D</DIV><DIV style=3D"margin-top: 0px; margin-
> >> right: 0px; =
> >> margin-bottom: 0px; margin-left: 0px; ">process, 16 process, 8
> >> process =
> >> and 4 process - all of which generated =3D</DIV><DIV
> >> style=3D"margin-top: =
> >> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;
> >> ">this =
> >> error message.</DIV><DIV><BR =3D</DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; =
> >> ">class=3D3D"khtml-block-placeholder"></
> >> DIV><DIV>Thanks,<=
> >> ;/DIV><DIV>david</DIV>=3D</DIV><DIV style=3D"margin-
> >> top: =
> >> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >> "><DIV><BR =
> >> class=3D3D"khtml-block-placeholder"></
> >> DIV><DIV><BR =
> >> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >> margin-bottom: 0px; margin-left: 0px; =
> >> ">class=3D3D"khtml-block-placeholder"></
> >> DIV><DIV><BR>=
> >> <DIV> <SPAN =3D</DIV><DIV style=3D"margin-top: 0px; =
> >> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >> ">class=3D3D"Apple-style-span" style=3D3D"border-collapse:
> >> separate; =
> >> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >> margin-bottom: 0px; margin-left: 0px; ">border-spacing: 0px 0px;
> >> color: =
> >> rgb(0, 0, 0); font-family: Helvetica; =3D</DIV><DIV
> >> style=3D"margin-top: =
> >> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >> ">font-size: 12px; font-style: normal; font-variant: normal; =
> >> font-weight: =3D</DIV><DIV style=3D"margin-top: 0px; margin-right:
> >> 0px; =
> >> margin-bottom: 0px; margin-left: 0px; ">normal; letter-spacing:
> >> normal; =
> >> line-height: normal; text-align: auto; =3D</DIV><DIV
> >> style=3D"margin-top: =
> >> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >> ">-khtml-text-decorations-in-effect: none; text-indent: 0px; =3D</
> >> DIV><DIV=
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">-apple-text-size-adjust: auto; text-transform:
> >> none; =
> >> orphans: 2; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right:
> >> 0px; =
> >> margin-bottom: 0px; margin-left: 0px; ">white-space: normal;
> >> widows: 2; =
> >> word-spacing: 0px; "><SPAN =3D</DIV><DIV style=3D"margin-
> >> top: 0px; =
> >> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >> ">class=3D3D"Apple-style-span" style=3D3D"border-collapse:
> >> separate; =
> >> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >> margin-bottom: 0px; margin-left: 0px; ">border-spacing: 0px 0px;
> >> color: =
> >> rgb(0, 0, 0); font-family: Helvetica; =3D</DIV><DIV
> >> style=3D"margin-top: =
> >> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >> ">font-size: 12px; font-style: normal; font-variant: normal; =
> >> font-weight: =3D</DIV><DIV style=3D"margin-top: 0px; margin-right:
> >> 0px; =
> >> margin-bottom: 0px; margin-left: 0px; ">normal; letter-spacing:
> >> normal; =
> >> line-height: normal; text-align: auto; =3D</DIV><DIV
> >> style=3D"margin-top: =
> >> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >> ">-khtml-text-decorations-in-effect: none; text-indent: 0px; =3D</
> >> DIV><DIV=
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">-apple-text-size-adjust: auto; text-transform:
> >> none; =
> >> orphans: 2; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right:
> >> 0px; =
> >> margin-bottom: 0px; margin-left: 0px; ">white-space: normal;
> >> widows: 2; =
> >> word-spacing: 0px; "><SPAN =3D</DIV><DIV style=3D"margin-
> >> top: 0px; =
> >> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >> ">class=3D3D"Apple-style-span" style=3D3D"border-collapse:
> >> separate; =
> >> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >> margin-bottom: 0px; margin-left: 0px; ">border-spacing: 0px 0px;
> >> color: =
> >> rgb(0, 0, 0); font-family: Helvetica; =3D</DIV><DIV
> >> style=3D"margin-top: =
> >> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >> ">font-size: 12px; font-style: normal; font-variant: normal; =
> >> font-weight: =3D</DIV><DIV style=3D"margin-top: 0px; margin-right:
> >> 0px; =
> >> margin-bottom: 0px; margin-left: 0px; ">normal; letter-spacing:
> >> normal; =
> >> line-height: normal; text-align: auto; =3D</DIV><DIV
> >> style=3D"margin-top: =
> >> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >> ">-khtml-text-decorations-in-effect: none; text-indent: 0px; =3D</
> >> DIV><DIV=
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">-apple-text-size-adjust: auto; text-transform:
> >> none; =
> >> orphans: 2; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right:
> >> 0px; =
> >> margin-bottom: 0px; margin-left: 0px; ">white-space: normal;
> >> widows: 2; =
> >> word-spacing: 0px; "><SPAN =3D</DIV><DIV style=3D"margin-
> >> top: 0px; =
> >> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >> ">class=3D3D"Apple-style-span" style=3D3D"border-collapse:
> >> separate; =
> >> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >> margin-bottom: 0px; margin-left: 0px; ">border-spacing: 0px 0px;
> >> color: =
> >> rgb(0, 0, 0); font-family: Helvetica; =3D</DIV><DIV
> >> style=3D"margin-top: =
> >> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >> ">font-size: 12px; font-style: normal; font-variant: normal; =
> >> font-weight: =3D</DIV><DIV style=3D"margin-top: 0px; margin-right:
> >> 0px; =
> >> margin-bottom: 0px; margin-left: 0px; ">normal; letter-spacing:
> >> normal; =
> >> line-height: normal; text-align: auto; =3D</DIV><DIV
> >> style=3D"margin-top: =
> >> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >> ">-khtml-text-decorations-in-effect: none; text-indent: 0px; =3D</
> >> DIV><DIV=
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">-apple-text-size-adjust: auto; text-transform:
> >> none; =
> >> orphans: 2; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right:
> >> 0px; =
> >> margin-bottom: 0px; margin-left: 0px; ">white-space: normal;
> >> widows: 2; =
> >> word-spacing: 0px; "><SPAN =3D</DIV><DIV style=3D"margin-
> >> top: 0px; =
> >> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >> ">class=3D3D"Apple-style-span" style=3D3D"border-collapse:
> >> separate; =
> >> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >> margin-bottom: 0px; margin-left: 0px; ">border-spacing: 0px 0px;
> >> color: =
> >> rgb(0, 0, 0); font-family: Helvetica; =3D</DIV><DIV
> >> style=3D"margin-top: =
> >> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >> ">font-size: 12px; font-style: normal; font-variant: normal; =
> >> font-weight: =3D</DIV><DIV style=3D"margin-top: 0px; margin-right:
> >> 0px; =
> >> margin-bottom: 0px; margin-left: 0px; ">normal; letter-spacing:
> >> normal; =
> >> line-height: normal; text-align: auto; =3D</DIV><DIV
> >> style=3D"margin-top: =
> >> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >> ">-khtml-text-decorations-in-effect: none; text-indent: 0px; =3D</
> >> DIV><DIV=
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">-apple-text-size-adjust: auto; text-transform:
> >> none; =
> >> orphans: 2; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right:
> >> 0px; =
> >> margin-bottom: 0px; margin-left: 0px; ">white-space: normal;
> >> widows: 2; =
> >> word-spacing: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-
> >> right: =
> >> 0px; margin-bottom: 0px; margin-left: 0px; =
> >> ">"><DIV>--</DIV><DIV>David =
> >> Gunter</DIV><DIV>HPC-4: HPC Environments: =3D</DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">Parallel Tools Team</DIV><DIV>Los
> >> Alamos =
> >> National =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >> margin-bottom: 0px; margin-left: 0px; =
> >> ">Laboratory</DIV></SPAN></SPAN><BR =3D</
> >> DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; =
> >> ">class=3D3D"Apple-interchange-newline"></SPAN></
> >> SPAN></=
> >> SPAN> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >> margin-bottom: 0px; margin-left: 0px; =
> >> "></DIV><BR><DIV><DIV>On Mar 21, 2007, at
> >> 4:50 =
> >> PM, wei huang =3D</DIV><DIV style=3D"margin-top: 0px; margin-
> >> right: 0px; =
> >> margin-bottom: 0px; margin-left: 0px; ">wrote:</DIV><BR =
> >> class=3D3D"Apple-interchange-newline"><BLOCKQUOTE =3D</
> >> DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">type=3D3D"cite"><DIV style=3D3D"margin-
> >> top: =
> >> 0px; margin-right: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> >> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >> ">margin-bottom: 0px; margin-left: 0px; ">Hi,</DIV><DIV =
> >> style=3D3D"margin-top:=3D</DIV><DIV style=3D"margin-top: 0px; =
> >> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><SPAN =
> >> class=3D"Apple-converted-space">=A0</SPAN>0px; margin-right: 0px; =
> >> margin-bottom: 0px; margin-left: 0px; =3D</DIV><DIV
> >> style=3D"margin-top: =
> >> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >> ">min-height: 14px; "><BR></DIV><DIV =
> >> style=3D3D"margin-top: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> >> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-
> >> right: =
> >> 0px; margin-bottom: 0px; margin-left: 0px; ">Thanks for =3D</
> >> DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">letting us know this problem.</
> >> DIV><DIV =
> >> style=3D3D"margin-top: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> >> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-
> >> right: =
> >> 0px; margin-bottom: 0px; margin-left: 0px; min-height: =3D</
> >> DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">14px; "><BR></DIV><DIV =
> >> style=3D3D"margin-top: 0px; margin-right: 0px; =3D</DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">margin-bottom: 0px; margin-left: 0px; ">We
> >> will =
> >> try to reproduce the =3D</DIV><DIV style=3D"margin-top: 0px; =
> >> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">problem
> >> on =
> >> our cluster. To help us on looking</DIV><DIV =3D</DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">style=3D3D"margin-top: 0px; margin-right: 0px; =
> >> margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> >> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-
> >> left: =
> >> 0px; ">into this problem, would you please let us know the =
> >> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >> margin-bottom: 0px; margin-left: 0px; ">following:</
> >> DIV><DIV =
> >> style=3D3D"margin-top: 0px; margin-right: 0px; =3D</DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">margin-bottom: 0px; margin-left: 0px; min-
> >> height: =
> >> 14px; "><BR></DIV><DIV =3D</DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">style=3D3D"margin-top: 0px; margin-right: 0px; =
> >> margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> >> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-
> >> left: =
> >> 0px; ">1) The exact CFLAGS you are using when configuring =3D</
> >> DIV><DIV=
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">mvapich2 (are you using</DIV><DIV =
> >> style=3D3D"margin-top: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> >> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-
> >> right: =
> >> 0px; margin-bottom: 0px; margin-left: 0px; ">default =3D</
> >> DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">scripts provided by us?)</DIV><DIV =
> >> style=3D3D"margin-top: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> >> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-
> >> right: =
> >> 0px; margin-bottom: 0px; margin-left: 0px; ">2) Any =3D</
> >> DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">runtime environment variables you have setup =
> >> up?</DIV><DIV =3D</DIV><DIV style=3D"margin-top: 0px; =
> >> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >> ">style=3D3D"margin-top: 0px; margin-right: 0px; margin-bottom:
> >> 0px; =
> >> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >> margin-bottom: 0px; margin-left: 0px; ">margin-left: 0px; ">3)
> >> On how =
> >> many nodes do you run the test?</DIV><DIV =3D</DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">style=3D3D"margin-top: 0px; margin-right: 0px; =
> >> margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> >> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-
> >> left: =
> >> 0px; min-height: 14px; "><BR></DIV><DIV =
> >> style=3D3D"margin-top: =3D</DIV><DIV style=3D"margin-top: 0px; =
> >> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">0px; =
> >> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =3D</
> >> DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">">Thanks.</DIV><DIV
> >> style=3D3D"margin-top:=
> >> 0px; margin-right: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> >> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >> ">margin-bottom: 0px; margin-left: 0px; min-height: 14px; =
> >> "><BR></DIV><DIV =3D</DIV><DIV style=3D"margin-
> >> top: =
> >> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >> ">style=3D3D"margin-top: 0px; margin-right: 0px; margin-bottom:
> >> 0px; =
> >> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >> margin-bottom: 0px; margin-left: 0px; ">margin-left: 0px; =
> >> ">Regards,</DIV><DIV style=3D3D"margin-top: 0px; =3D</
> >> DIV><DIV=
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">margin-right: 0px; margin-bottom: 0px; margin-
> >> left: =
> >> 0px; ">Wei =3D</DIV><DIV style=3D"margin-top: 0px; margin-
> >> right: 0px; =
> >> margin-bottom: 0px; margin-left: 0px; ">Huang</DIV><DIV =
> >> style=3D3D"margin-top: 0px; margin-right: 0px; =3D</DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">margin-bottom: 0px; margin-left: 0px; min-
> >> height: =
> >> 14px; "><BR></DIV><DIV =3D</DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">style=3D3D"margin-top: 0px; margin-right: 0px; =
> >> margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> >> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-
> >> left: =
> >> 0px; ">774 Dreese Lab, 2015 Neil Ave,</DIV><DIV =3D</
> >> DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">style=3D3D"margin-top: 0px; margin-right: 0px; =
> >> margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> >> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-
> >> left: =
> >> 0px; ">Dept. of Computer Science and Engineering</
> >> DIV><DIV =
> >> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >> margin-bottom: 0px; margin-left: 0px; ">style=3D3D"margin-top: 0px; =
> >> margin-right: 0px; margin-bottom: 0px; =3D</DIV><DIV
> >> style=3D"margin-top: =
> >> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >> ">margin-left: 0px; ">Ohio State University</DIV><DIV =
> >> style=3D3D"margin-top: =3D</DIV><DIV style=3D"margin-top: 0px; =
> >> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">0px; =
> >> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">OH =
> >> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >> margin-bottom: 0px; margin-left: 0px; ">43210</DIV><DIV =
> >> style=3D3D"margin-top: 0px; margin-right: 0px; =3D</DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">margin-bottom: 0px; margin-left: 0px; ">Tel: =
> >> (614)292-8501</DIV><DIV =3D</DIV><DIV style=3D"margin-
> >> top: 0px; =
> >> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >> ">style=3D3D"margin-top: 0px; margin-right: 0px; margin-bottom:
> >> 0px; =
> >> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >> margin-bottom: 0px; margin-left: 0px; ">margin-left: 0px; min-
> >> height: =
> >> 14px; "><BR></DIV><DIV style=3D3D"margin-top: =
> >> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >> margin-bottom: 0px; margin-left: 0px; ">0px; margin-right: 0px; =
> >> margin-bottom: 0px; margin-left: 0px; =3D</DIV><DIV
> >> style=3D"margin-top: =
> >> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >> ">min-height: 14px; "><BR></DIV><DIV =
> >> style=3D3D"margin-top: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> >> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-
> >> right: =
> >> 0px; margin-bottom: 0px; margin-left: 0px; ">On Wed, 21 =3D</
> >> DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">Mar 2007, David Gunter wrote:</
> >> DIV><DIV =
> >> style=3D3D"margin-top: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> >> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-
> >> right: =
> >> 0px; margin-bottom: 0px; margin-left: 0px; min-height: =3D</
> >> DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">14px; "><BR></DIV>
> >> <BLOCKQUOTE =
> >> type=3D3D"cite"><DIV style=3D3D"margin-top: =3D</DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">I have =3D</DIV><DIV style=3D"margin-top:
> >> 0px; =
> >> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">built =
> >> mvapich2 for an OFED-based IB Opteron cluster.<SPAN =3D</
> >> DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">class=3D3D"Apple-converted-space">=3DA0 =
> >> </SPAN>When</DIV><DIV =3D</DIV><DIV style=3D"margin-
> >> top: =
> >> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >> ">style=3D3D"margin-top: 0px; margin-right: 0px; margin-bottom:
> >> 0px; =
> >> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >> margin-bottom: 0px; margin-left: 0px; ">margin-left: 0px;
> >> ">running =
> >> the Intel MPI Benchmarks (IMB3) I keep =3D</DIV><DIV
> >> style=3D"margin-top: =
> >> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;
> >> ">seeing =
> >> the following</DIV><DIV style=3D3D"margin-top: 0px; =
> >> margin-right: =3D</DIV><DIV style=3D"margin-top: 0px; margin-
> >> right: 0px; =
> >> margin-bottom: 0px; margin-left: 0px; ">0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">errors messages in many =3D</DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">spots, although the tests run to =
> >> completion:</DIV><DIV =3D</DIV><DIV style=3D"margin-top:
> >> 0px; =
> >> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >> ">style=3D3D"margin-top: 0px; margin-right: 0px; margin-bottom:
> >> 0px; =
> >> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >> margin-bottom: 0px; margin-left: 0px; ">margin-left: 0px; min-
> >> height: =
> >> 14px; "><BR></DIV><DIV style=3D3D"margin-top: =
> >> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >> margin-bottom: 0px; margin-left: 0px; ">0px; margin-right: 0px; =
> >> margin-bottom: 0px; margin-left: 0px; ">Internal =3D</DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">error: communicator is already on free =
> >> list</DIV><DIV style=3D3D"margin-top:=3D</DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; "><SPAN class=3D"Apple-converted-space">=A0</
> >> SPAN>0px; =
> >> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =3D</
> >> DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">min-height: 14px; "><BR></
> >> DIV><DIV =
> >> style=3D3D"margin-top: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> >> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-
> >> right: =
> >> 0px; margin-bottom: 0px; margin-left: 0px; ">What is this =3D</
> >> DIV><DIV=
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">referring to?</DIV><DIV =
> >> style=3D3D"margin-top: 0px; margin-right: 0px; =3D</DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">margin-bottom: 0px; margin-left: 0px; min-
> >> height: =
> >> 14px; "><BR></DIV><DIV =3D</DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">style=3D3D"margin-top: 0px; margin-right: 0px; =
> >> margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> >> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-
> >> left: =
> >> 0px; ">Thanks.</DIV><DIV style=3D3D"margin-top: 0px; =
> >> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >> margin-bottom: 0px; margin-left: 0px; ">margin-right: 0px; =
> >> margin-bottom: 0px; margin-left: 0px; =3D</DIV><DIV
> >> style=3D"margin-top: =
> >> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >> ">">--david</DIV><DIV style=3D3D"margin-top: 0px; =
> >> margin-right: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-
> >> right: =
> >> 0px; margin-bottom: 0px; margin-left: 0px; ">margin-bottom: 0px; =
> >> margin-left: 0px; min-height: 14px; "><BR></
> >> DIV><DIV =
> >> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >> margin-bottom: 0px; margin-left: 0px; ">style=3D3D"margin-top: 0px; =
> >> margin-right: 0px; margin-bottom: 0px; =3D</DIV><DIV
> >> style=3D"margin-top: =
> >> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >> ">margin-left: 0px; ">--</DIV><DIV style=3D3D"margin-
> >> top: =
> >> 0px; margin-right: =3D</DIV><DIV style=3D"margin-top: 0px; margin-
> >> right: =
> >> 0px; margin-bottom: 0px; margin-left: 0px; ">0px; margin-bottom:
> >> 0px; =
> >> margin-left: 0px; ">David Gunter</DIV><DIV =3D</
> >> DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">style=3D3D"margin-top: 0px; margin-right: 0px; =
> >> margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> >> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-
> >> left: =
> >> 0px; ">HPC-4: HPC Environments: Parallel Tools =3D</DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">Team</DIV><DIV style=3D3D"margin-top:
> >> 0px; =
> >> margin-right: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-
> >> right: =
> >> 0px; margin-bottom: 0px; margin-left: 0px; ">margin-bottom: 0px; =
> >> margin-left: 0px; ">Los Alamos National =3D</DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">Laboratory</DIV><DIV
> >> style=3D3D"margin-top: =
> >> 0px; margin-right: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> >> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >> ">margin-bottom: 0px; margin-left: 0px; min-height: 14px; =
> >> "><BR></DIV><DIV =3D</DIV><DIV style=3D"margin-
> >> top: =
> >> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >> ">style=3D3D"margin-top: 0px; margin-right: 0px; margin-bottom:
> >> 0px; =
> >> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >> margin-bottom: 0px; margin-left: 0px; ">margin-left: 0px; min-
> >> height: =
> >> 14px; "><BR></DIV><DIV style=3D3D"margin-top: =
> >> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >> margin-bottom: 0px; margin-left: 0px; ">0px; margin-right: 0px; =
> >> margin-bottom: 0px; margin-left: 0px; =3D</DIV><DIV
> >> style=3D"margin-top: =
> >> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >> ">min-height: 14px; "><BR></DIV> =
> >> </BLOCKQUOTE><DIV style=3D3D"margin-top: =3D</DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-
> >> right: =
> >> 0px; margin-bottom: 0px; margin-left: 0px; ">min-height: 14px; =
> >> "><BR></DIV> =3D</DIV><DIV style=3D"margin-top: 0px; =
> >> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >> "></BLOCKQUOTE></DIV><BR></DIV></
> >> BODY></H=
> >> TML>=3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> >> margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></
> >> DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">--Apple-Mail-1-849288998--</DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; min-height: 14px; "><BR></DIV><DIV
> >> style=3D"margin-top: =
> >> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >> ">--=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D1750880957=3D=3D</
> >> DIV><DI=
> >> V style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">Content-Type: text/plain; =
> >> charset=3D"us-ascii"</DIV><DIV style=3D"margin-top: 0px; margin-
> >> right: =
> >> 0px; margin-bottom: 0px; margin-left: 0px; ">MIME-Version: 1.0</
> >> DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">Content-Transfer-Encoding: 7bit</DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">Content-Disposition: inline</DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; min-height: 14px; "><BR></DIV><DIV
> >> style=3D"margin-top: =
> >> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >> ">_______________________________________________</DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">mvapich-discuss mailing list</DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; "><A =
> >> href=3D"mailto:mvapich-discuss at cse.ohio-state.edu">mvapich-
> >> discuss at cse.ohi=
> >> o-state.edu</A></DIV><DIV style=3D"margin-top: 0px; margin-right:
> >> 0px; =
> >> margin-bottom: 0px; margin-left: 0px; "><A =
> >> href=3D"http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-
> >> discuss">h=
> >> ttp://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss</
> >> A></DIV><D=
> >> IV style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; min-height: 14px; "><BR></DIV><DIV
> >> style=3D"margin-top: =
> >> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >> ">--=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D1750880957=3D=3D--
> >> </DIV><=
> >> DIV style=3D"margin-top: 0px; margin-right: 0px; margin-bottom:
> >> 0px; =
> >> margin-left: 0px; min-height: 14px; "><BR></DIV> </BLOCKQUOTE><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; min-height: 14px; "><BR></DIV> </BLOCKQUOTE><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; min-height: 14px; "><BR></DIV><DIV
> >> style=3D"margin-top: =
> >> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> >> ">_______________________________________________</DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; ">mvapich-discuss mailing list</DIV><DIV =
> >> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> >> margin-left: 0px; "><A =
> >> href=3D"mailto:mvapich-discuss at cse.ohio-state.edu">mvapich-
> >> discuss at cse.ohi=
> >> o-state.edu</A></DIV><DIV style=3D"margin-top: 0px; margin-right:
> >> 0px; =
> >> margin-bottom: 0px; margin-left: 0px; "><A =
> >> href=3D"http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-
> >> discuss">h=
> >> ttp://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss</
> >> A></DIV> =
> >> </BLOCKQUOTE></DIV><BR></DIV></BODY></HTML>=
> >>
> >> --Apple-Mail-2-709635751--
> >>
> >> --===============1220798408==
> >> Content-Type: text/plain; charset="us-ascii"
> >> MIME-Version: 1.0
> >> Content-Transfer-Encoding: 7bit
> >> Content-Disposition: inline
> >>
> >> _______________________________________________
> >> mvapich-discuss mailing list
> >> mvapich-discuss at cse.ohio-state.edu
> >> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
> >>
> >> --===============1220798408==--
> >>
> >
>
>
> --Apple-Mail-4-797936092
> Content-Transfer-Encoding: quoted-printable
> Content-Type: text/html;
> charset=ISO-8859-1
>
> <HTML><BODY style=3D"word-wrap: break-word; -khtml-nbsp-mode: space; =
> -khtml-line-break: after-white-space; ">I had a chance to download =
> mvapich2-0.9.8p2 this morning and retest and yes, the problem still =
> exists.=A0 I am running the IMB 3.0 suite and I see the following type =
> of behavior:<DIV><BR =
> class=3D"khtml-block-placeholder"></DIV><DIV>#----------------------------=
> ------------------------------------</DIV><DIV># Benchmarking Allreduce =
> </DIV><DIV># #processes =3D 2 </DIV><DIV># ( 2 additional processes =
> waiting in =
> MPI_Barrier)</DIV><DIV>#--------------------------------------------------=
> --------------</DIV><DIV>=A0 =A0 =A0=A0 #bytes #repetitions=A0 =
> t_min[usec]=A0 t_max[usec]=A0 t_avg[usec]</DIV><DIV>=A0 =A0 =A0 =A0 =A0 =
> =A0 0=A0 =A0 =A0 =A0=A0 1000=A0 =A0 =A0 =A0=A0 0.03=A0 =A0 =A0 =A0=A0 =
> 0.03=A0 =A0 =A0 =A0=A0 0.03</DIV><DIV>=A0 =A0 =A0 =A0 =A0 =A0 4=A0 =A0 =A0=
> =A0=A0 1000=A0 =A0 =A0 =A0=A0 1.79=A0 =A0 =A0 =A0=A0 1.79=A0 =A0 =A0 =A0=A0=
> 1.79</DIV><DIV>=A0 =A0 =A0 =A0 =A0 =A0 8=A0 =A0 =A0 =A0=A0 1000=A0 =A0 =
> =A0 =A0=A0 1.81=A0 =A0 =A0 =A0=A0 1.82=A0 =A0 =A0 =A0=A0 =
> 1.81</DIV><DIV>=A0 =A0 =A0 =A0 =A0=A0 16=A0 =A0 =A0 =A0=A0 1000=A0 =A0 =A0=
> =A0=A0 1.84=A0 =A0 =A0 =A0=A0 1.84=A0 =A0 =A0 =A0=A0 1.84</DIV><DIV>=A0 =
> =A0 =A0 =A0 =A0=A0 32=A0 =A0 =A0 =A0=A0 1000=A0 =A0 =A0 =A0=A0 1.93=A0 =A0=
> =A0 =A0=A0 1.93=A0 =A0 =A0 =A0=A0 1.93</DIV><DIV>=A0 =A0 =A0 =A0 =A0=A0 =
> 64=A0 =A0 =A0 =A0=A0 1000=A0 =A0 =A0 =A0=A0 1.97=A0 =A0 =A0 =A0=A0 1.97=A0=
> =A0 =A0 =A0=A0 1.97</DIV><DIV>=A0 =A0 =A0 =A0 =A0 128=A0 =A0 =A0 =A0=A0 =
> 1000=A0 =A0 =A0 =A0=A0 2.12=A0 =A0 =A0 =A0=A0 2.13=A0 =A0 =A0 =A0=A0 =
> 2.13</DIV><DIV>=A0 =A0 =A0 =A0 =A0 256=A0 =A0 =A0 =A0=A0 1000=A0 =A0 =A0 =
> =A0=A0 2.39=A0 =A0 =A0 =A0=A0 2.39=A0 =A0 =A0 =A0=A0 2.39</DIV><DIV>=A0 =
> =A0 =A0 =A0 =A0 512=A0 =A0 =A0 =A0=A0 1000=A0 =A0 =A0 =A0=A0 2.87=A0 =A0 =
> =A0 =A0=A0 2.88=A0 =A0 =A0 =A0=A0 2.87</DIV><DIV>=A0 =A0 =A0 =A0=A0 =
> 1024=A0 =A0 =A0 =A0=A0 1000=A0 =A0 =A0 =A0=A0 3.90=A0 =A0 =A0 =A0=A0 =
> 3.90=A0 =A0 =A0 =A0=A0 3.90</DIV><DIV>=A0 =A0 =A0 =A0=A0 2048=A0 =A0 =A0 =
> =A0=A0 1000=A0 =A0 =A0 =A0=A0 5.97=A0 =A0 =A0 =A0=A0 5.97=A0 =A0 =A0 =A0=A0=
> 5.97</DIV><DIV>=A0 =A0 =A0 =A0=A0 4096=A0 =A0 =A0 =A0=A0 1000=A0 =A0 =A0 =
> =A0 10.38=A0 =A0 =A0 =A0 10.38=A0 =A0 =A0 =A0 10.38</DIV><DIV>=A0 =A0 =A0 =
> =A0=A0 8192=A0 =A0 =A0 =A0=A0 1000=A0 =A0 =A0 =A0 19.20=A0 =A0 =A0 =A0 =
> 19.20=A0 =A0 =A0 =A0 19.20</DIV><DIV>=A0 =A0 =A0 =A0 16384=A0 =A0 =A0 =A0=A0=
> 1000=A0 =A0 =A0 =A0 36.35=A0 =A0 =A0 =A0 36.35=A0 =A0 =A0 =A0 =
> 36.35</DIV><DIV>=A0 =A0 =A0 =A0 32768=A0 =A0 =A0 =A0=A0 1000=A0 =A0 =A0 =
> =A0 69.93=A0 =A0 =A0 =A0 69.93=A0 =A0 =A0 =A0 69.93</DIV><DIV>=A0 =A0 =A0 =
> =A0 65536=A0 =A0 =A0 =A0 =A0 640=A0 =A0 =A0=A0 142.94=A0 =A0 =A0=A0 =
> 142.94=A0 =A0 =A0=A0 142.94</DIV><DIV>=A0 =A0 =A0=A0 131072=A0 =A0 =A0 =A0=
> =A0 320=A0 =A0 =A0=A0 281.42=A0 =A0 =A0=A0 281.42=A0 =A0 =A0=A0 =
> 281.42</DIV><DIV>=A0 =A0 =A0=A0 262144=A0 =A0 =A0 =A0 =A0 160=A0 =A0 =A0=A0=
> 607.62=A0 =A0 =A0=A0 607.66=A0 =A0 =A0=A0 607.64</DIV><DIV>=A0 =A0 =A0=A0=
> 524288=A0 =A0 =A0 =A0 =A0=A0 80=A0 =A0 =A0 1662.42=A0 =A0 =A0 1662.94=A0 =
> =A0 =A0 1662.68</DIV><DIV>=A0 =A0 =A0 1048576=A0 =A0 =A0 =A0 =A0=A0 40=A0 =
> =A0 =A0 3636.55=A0 =A0 =A0 3636.75=A0 =A0 =A0 3636.65</DIV><DIV>=A0 =A0 =
> =A0 2097152=A0 =A0 =A0 =A0 =A0=A0 20=A0 =A0 =A0 8966.41=A0 =A0 =A0 =
> 8974.30=A0 =A0 =A0 8970.36</DIV><DIV>=A0 =A0 =A0 4194304=A0 =A0 =A0 =A0 =
> =A0=A0 10=A0 =A0=A0 17831.90=A0 =A0=A0 17837.21=A0 =A0=A0 =
> 17834.56</DIV><DIV>Internal error: communicator is already on free =
> list</DIV><DIV>Internal error: communicator is already on free =
> list</DIV><DIV><BR class=3D"khtml-block-placeholder"></DIV><DIV><BR =
> class=3D"khtml-block-placeholder"></DIV><DIV>Wherever a test uses all =
> the processes in a communicator (4 in this run) then there are no =
> internal error messages.=A0 The number of such messages printed is =
> always equal to the number of processes that did not participate in a =
> particular test, i.e. the number that always wait in =
> MPI_Barrier.</DIV><DIV><BR =
> class=3D"khtml-block-placeholder"></DIV><DIV>Again, if I reconfigure =
> with --disable-debuginfo then I also don't see these error =
> messages.</DIV><DIV><BR =
> class=3D"khtml-block-placeholder"></DIV><DIV>-david</DIV><DIV><DIV><BR><DI=
> V> <SPAN class=3D"Apple-style-span" style=3D"border-collapse: separate; =
> border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family: Helvetica; =
> font-size: 12px; font-style: normal; font-variant: normal; font-weight: =
> normal; letter-spacing: normal; line-height: normal; text-align: auto; =
> -khtml-text-decorations-in-effect: none; text-indent: 0px; =
> -apple-text-size-adjust: auto; text-transform: none; orphans: 2; =
> white-space: normal; widows: 2; word-spacing: 0px; "><SPAN =
> class=3D"Apple-style-span" style=3D"border-collapse: separate; =
> border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family: Helvetica; =
> font-size: 12px; font-style: normal; font-variant: normal; font-weight: =
> normal; letter-spacing: normal; line-height: normal; text-align: auto; =
> -khtml-text-decorations-in-effect: none; text-indent: 0px; =
> -apple-text-size-adjust: auto; text-transform: none; orphans: 2; =
> white-space: normal; widows: 2; word-spacing: 0px; =
> "><DIV>--</DIV><DIV>David Gunter</DIV><DIV>HPC-4: Parallel Tools =
> Team</DIV><DIV>Los Alamos National Laboratory</DIV><BR =
> class=3D"Apple-interchange-newline"></SPAN></SPAN> =
> </DIV><BR><DIV><DIV>On May 9, 2007, at 2:02 PM, Dhabaleswar Panda =
> wrote:</DIV><BR class=3D"Apple-interchange-newline"><BLOCKQUOTE =
> type=3D"cite"><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">Hi David,<SPAN =
> class=3D"Apple-converted-space">=A0</SPAN></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> min-height: 14px; "><BR></DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Do you see =
> this problem with MVAPICH2 0.9.8p2 version? We had several</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">fixes to the released version during the last =
> months. This latest</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">version (0.9.8p2) is =
> available from the mvapich page. This version is</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">also included and is being tested with OFED 1.2. We =
> have also done</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">testing of this latest version =
> with OFED 1.1.</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">Please let us know if you still see the error with =
> this latest version</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">(p2) on your system and we =
> will investigate this issue further.<SPAN =
> class=3D"Apple-converted-space">=A0</SPAN></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> min-height: 14px; "><BR></DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Thanks,<SPAN =
> class=3D"Apple-converted-space">=A0</SPAN></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> min-height: 14px; "><BR></DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">DK</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; min-height: 14px; "><BR></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> min-height: 14px; "><BR></DIV> <BLOCKQUOTE type=3D"cite"><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">I am curious to know if you were able to reproduce =
> this problem, <SPAN class=3D"Apple-converted-space">=A0</SPAN></DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">whether it has been fixed or not.</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; min-height: 14px; "><BR></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">Thanks,</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">david</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">--</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">David Gunter</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">HPC-4: HPC Environments: Parallel Tools =
> Team</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">Los Alamos National =
> Laboratory</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; min-height: 14px; "><BR></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">On Mar =
> 22, 2007, at 11:20 AM, David Gunter wrote:</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> min-height: 14px; "><BR></DIV> <BLOCKQUOTE type=3D"cite"><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">I am using the released 0.9.8 version.</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; min-height: 14px; "><BR></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">-david</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">On Mar 22, 2007, at 9:37 AM, Dhabaleswar Panda =
> wrote:</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></DIV> =
> <BLOCKQUOTE type=3D"cite"><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">Hi David,</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; min-height: 14px; "><BR></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Thanks =
> for this information. One more question - are you using</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">MVAPICH2 0.9.8 `released' version or the `branch' =
> version (with some</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">recent fixes). If you can =
> let us know this information, it will help</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">us.</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">Thanks,</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: =
> 14px; "><BR></DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">DK</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> min-height: 14px; "><BR></DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: =
> 14px; "><BR></DIV> <BLOCKQUOTE type=3D"cite"><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">I have =
> recompiled mvapich2 without using the --enable-debuginfo flag</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">and the problem has gone away.<SPAN =
> class=3D"Apple-converted-space">=A0 </SPAN>However, I wish to have =
> debuginfo</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">available to our TotalView users =
> so hopefully this can be resolved.</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: =
> 14px; "><BR></DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">Here is the configuration that =
> generates the error message I saw</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">previously:</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">./configure =
> --prefix=3D/opt/mvapich2/mvapich2-0.9.8-gcc/ib --with-</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">openib=3D/usr/local/ofed --enable-romio =
> --with-file-system=3Dufs+nfs</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">+panfs =
> --enable-sharedlibs=3Dgcc --enable-debuginfo --enable-fast --</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">with-mpe</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: =
> 14px; "><BR></DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">OFED is ofed-1.1</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; min-height: 14px; "><BR></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">CFLAGS=3D-D_X86_64_ -D_SMP_ -DUSE_HEADER_CACHING<SPAN =
> class=3D"Apple-converted-space">=A0 </SPAN>-DONE_SIDED -</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">DMPID_USE_SEQUENCE_NUMBERS<SPAN =
> class=3D"Apple-converted-space">=A0 </SPAN>-I/usr/local/ofed/include -O2 =
> <SPAN class=3D"Apple-converted-space">=A0 </SPAN>-</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">D_SHMEM_COLL_</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: =
> 14px; "><BR></DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">CC=3D/usr/bin/gcc</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">CXX=3D/usr/bin/g++</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">FC=3D/usr/bin/gfortran</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">F77=3D/usr/bin/gfortran</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">F90=3D/usr/bin/gfortran</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: =
> 14px; "><BR></DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">The test was run on 32 process, =
> 16 process, 8 process and 4 <SPAN =
> class=3D"Apple-converted-space">=A0</SPAN></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">process =
> -</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: =
> 0px; margin-left: 0px; ">all of which generated this error =
> message.</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">Thanks,</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">david</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; min-height: 14px; "><BR></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> min-height: 14px; "><BR></DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">--</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">David Gunter</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">HPC-4: HPC =
> Environments: Parallel Tools Team</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Los Alamos =
> National Laboratory</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; =
> "><BR></DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">On Mar 21, 2007, at 4:50 PM, wei huang =
> wrote:</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></DIV> =
> <BLOCKQUOTE type=3D"cite"><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">Hi,</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; min-height: 14px; "><BR></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Thanks =
> for letting us know this problem.</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: =
> 14px; "><BR></DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">We will try to reproduce the =
> problem on our cluster. To help us on</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">looking</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">into this problem, would you =
> please let us know the following:</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: =
> 14px; "><BR></DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">1) The exact CFLAGS you are =
> using when configuring mvapich2 (are</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">you =
> using</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">default scripts provided by =
> us?)</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">2) Any runtime environment =
> variables you have setup up?</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">3) On how =
> many nodes do you run the test?</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: =
> 14px; "><BR></DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">Thanks.</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; min-height: 14px; "><BR></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">Regards,</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">Wei Huang</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; min-height: 14px; "><BR></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">774 =
> Dreese Lab, 2015 Neil Ave,</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Dept. of =
> Computer Science and Engineering</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Ohio State =
> University</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">OH 43210</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">Tel: (614)292-8501</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> min-height: 14px; "><BR></DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: =
> 14px; "><BR></DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">On Wed, 21 Mar 2007, David =
> Gunter wrote:</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></DIV> =
> <BLOCKQUOTE type=3D"cite"><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">I have built mvapich2 for =
> an OFED-based IB Opteron cluster.<SPAN class=3D"Apple-converted-space">=A0=
> </SPAN>When</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">running the Intel MPI Benchmarks =
> (IMB3) I keep seeing the <SPAN =
> class=3D"Apple-converted-space">=A0</SPAN></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">following</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">errors messages in many spots, =
> although the tests run to <SPAN =
> class=3D"Apple-converted-space">=A0</SPAN></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">completion:</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">Internal error: communicator is already on free =
> list</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">What is this referring to?</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; min-height: 14px; "><BR></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">Thanks.</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">--david</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; min-height: 14px; "><BR></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">--</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">David Gunter</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">HPC-4: HPC Environments: Parallel Tools =
> Team</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">Los Alamos National =
> Laboratory</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; min-height: 14px; "><BR></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> min-height: 14px; "><BR></DIV> </BLOCKQUOTE><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> min-height: 14px; "><BR></DIV> </BLOCKQUOTE><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> min-height: 14px; "><BR></DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: =
> 14px; "><BR></DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; =
> ">--Apple-Mail-1-849288998</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">Content-Transfer-Encoding: quoted-printable</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">Content-Type: text/html;</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; "><SPAN class=3D"Apple-tab-span" =
> style=3D"white-space:pre"> </SPAN>charset=3DISO-8859-1</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; min-height: 14px; "><BR></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> "><HTML><BODY style=3D3D"word-wrap: break-word; =
> -khtml-nbsp-mode: <SPAN class=3D"Apple-converted-space">=A0</SPAN></DIV><D=
> IV style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">space; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">-khtml-line-break: after-white-space; ">I have recompiled mvapich2 =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">without using the =
> --enable-debuginfo flag and the problem has gone =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">away.=3DA0 However, I wish to have debuginfo =
> available to our <SPAN class=3D"Apple-converted-space">=A0</SPAN></DIV><DI=
> V style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">TotalView =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">users so =
> hopefully this can be resolved.<DIV><BR =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; =
> ">class=3D3D"khtml-block-placeholder"></DIV><DIV>Here is =
> the <SPAN class=3D"Apple-converted-space">=A0</SPAN></DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">configuration =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">that =
> generates the error message I saw =
> previously:</DIV><DIV><BR =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; =
> ">class=3D3D"khtml-block-placeholder"></DIV><DIV>./configur=
> e =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; =
> ">--prefix=3D3D/opt/mvapich2/mvapich2-0.9.8-gcc/ib=3DA0--with-openib=3D3D/=
> <SPAN class=3D"Apple-converted-space">=A0</SPAN></DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">usr/loca=3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">l/ofed =
> --enable-romio --with-file-system=3D3Dufs+nfs+panfs =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">--enable-sharedlibs=3D3Dgcc --enable-debuginfo =
> --enable-fast =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; =
> ">--with-mpe</DIV><DIV><BR =
> class=3D3D"khtml-block-placeholder"></<SPAN =
> class=3D"Apple-converted-space">=A0</SPAN></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">DIV><DIV>OFED=3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><SPAN =
> class=3D"Apple-converted-space">=A0</SPAN>is =
> ofed-1.1</DIV><DIV><BR =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">class=3D3D"khtml-block-placeholder"></DIV><DIV>CFLAGS=3D3=
> D-D_X86_64_ -<SPAN class=3D"Apple-converted-space">=A0</SPAN></DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">D_SMP_ =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">-DUSE_HEADER_CACHING=3DA0 -DONE_SIDED -DMPID_USE_SEQUENCE_NUMBERS=3DA0 =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">-I/usr/local/ofed/include =
> -O2=3DA0=3DA0 -D_SHMEM_COLL_</DIV><DIV><BR =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; =
> ">class=3D3D"khtml-block-placeholder"></DIV><DIV>CC=3D3D/us=
> r/bin/gcc</<SPAN class=3D"Apple-converted-space">=A0</SPAN></DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">DIV><DIV>C=3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">XX=3D3D/usr/bin/g++</DIV><DIV>FC=3D3D/usr/bin/gfortran</<=
> SPAN class=3D"Apple-converted-space">=A0</SPAN></DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">DIV><DIV>F77=3D3D/usr/bi=3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; =
> ">n/gfortran</DIV><DIV>F90=3D3D/usr/bin/gfortran</DIV>&l=
> t;DIV><BR =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; =
> ">class=3D3D"khtml-block-placeholder"></DIV><DIV>The test =
> was run on <SPAN class=3D"Apple-converted-space">=A0</SPAN></DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">32 =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">process, 16 =
> process, 8 process and 4 process - all of which <SPAN =
> class=3D"Apple-converted-space">=A0</SPAN></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">generated =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">this error =
> message.</DIV><DIV><BR =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">class=3D3D"khtml-block-placeholder"></DIV><DIV>Thanks,<=
> ;/<SPAN class=3D"Apple-converted-space">=A0</SPAN></DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">DIV><DIV>david</DIV>=3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; "><DIV><BR =
> class=3D3D"khtml-block-placeholder"></DIV><DIV><BR =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; =
> ">class=3D3D"khtml-block-placeholder"></DIV><DIV><BR>=
> <DIV> <SPAN =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">class=3D3D"Apple-style-span" style=3D3D"border-collapse: separate; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">border-spacing: 0px 0px; color: =
> rgb(0, 0, 0); font-family: <SPAN =
> class=3D"Apple-converted-space">=A0</SPAN></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">Helvetica; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">font-size: 12px; font-style: =
> normal; font-variant: normal; font-<SPAN =
> class=3D"Apple-converted-space">=A0</SPAN></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">weight: =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">normal; letter-spacing: normal; =
> line-height: normal; text-align: <SPAN =
> class=3D"Apple-converted-space">=A0</SPAN></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">auto; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; =
> ">-khtml-text-decorations-in-effect: none; text-indent: 0px; =3D</DIV><DIV=
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">-apple-text-size-adjust: auto; text-transform: none; =
> orphans: 2; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">white-space: normal; widows: 2; =
> word-spacing: 0px; "><SPAN =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">class=3D3D"Apple-style-span" style=3D3D"border-collapse: separate; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">border-spacing: 0px 0px; color: =
> rgb(0, 0, 0); font-family: <SPAN =
> class=3D"Apple-converted-space">=A0</SPAN></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">Helvetica; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">font-size: 12px; font-style: =
> normal; font-variant: normal; font-<SPAN =
> class=3D"Apple-converted-space">=A0</SPAN></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">weight: =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">normal; letter-spacing: normal; =
> line-height: normal; text-align: <SPAN =
> class=3D"Apple-converted-space">=A0</SPAN></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">auto; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; =
> ">-khtml-text-decorations-in-effect: none; text-indent: 0px; =3D</DIV><DIV=
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">-apple-text-size-adjust: auto; text-transform: none; =
> orphans: 2; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">white-space: normal; widows: 2; =
> word-spacing: 0px; "><SPAN =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">class=3D3D"Apple-style-span" style=3D3D"border-collapse: separate; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">border-spacing: 0px 0px; color: =
> rgb(0, 0, 0); font-family: <SPAN =
> class=3D"Apple-converted-space">=A0</SPAN></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">Helvetica; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">font-size: 12px; font-style: =
> normal; font-variant: normal; font-<SPAN =
> class=3D"Apple-converted-space">=A0</SPAN></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">weight: =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">normal; letter-spacing: normal; =
> line-height: normal; text-align: <SPAN =
> class=3D"Apple-converted-space">=A0</SPAN></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">auto; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; =
> ">-khtml-text-decorations-in-effect: none; text-indent: 0px; =3D</DIV><DIV=
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">-apple-text-size-adjust: auto; text-transform: none; =
> orphans: 2; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">white-space: normal; widows: 2; =
> word-spacing: 0px; "><SPAN =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">class=3D3D"Apple-style-span" style=3D3D"border-collapse: separate; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">border-spacing: 0px 0px; color: =
> rgb(0, 0, 0); font-family: <SPAN =
> class=3D"Apple-converted-space">=A0</SPAN></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">Helvetica; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">font-size: 12px; font-style: =
> normal; font-variant: normal; font-<SPAN =
> class=3D"Apple-converted-space">=A0</SPAN></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">weight: =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">normal; letter-spacing: normal; =
> line-height: normal; text-align: <SPAN =
> class=3D"Apple-converted-space">=A0</SPAN></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">auto; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; =
> ">-khtml-text-decorations-in-effect: none; text-indent: 0px; =3D</DIV><DIV=
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">-apple-text-size-adjust: auto; text-transform: none; =
> orphans: 2; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">white-space: normal; widows: 2; =
> word-spacing: 0px; "><SPAN =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">class=3D3D"Apple-style-span" style=3D3D"border-collapse: separate; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">border-spacing: 0px 0px; color: =
> rgb(0, 0, 0); font-family: <SPAN =
> class=3D"Apple-converted-space">=A0</SPAN></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">Helvetica; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">font-size: 12px; font-style: =
> normal; font-variant: normal; font-<SPAN =
> class=3D"Apple-converted-space">=A0</SPAN></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">weight: =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">normal; letter-spacing: normal; =
> line-height: normal; text-align: <SPAN =
> class=3D"Apple-converted-space">=A0</SPAN></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">auto; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; =
> ">-khtml-text-decorations-in-effect: none; text-indent: 0px; =3D</DIV><DIV=
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">-apple-text-size-adjust: auto; text-transform: none; =
> orphans: 2; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">white-space: normal; widows: 2; =
> word-spacing: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; =
> ">"><DIV>--</DIV><DIV>David =
> Gunter</DIV><DIV>HPC-4: HPC <SPAN =
> class=3D"Apple-converted-space">=A0</SPAN></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">Environments: =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">Parallel Tools =
> Team</DIV><DIV>Los Alamos National =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; =
> ">Laboratory</DIV></SPAN></SPAN><BR =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; =
> ">class=3D3D"Apple-interchange-newline"></SPAN></SPAN></=
> SPAN> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; =
> "></DIV><BR><DIV><DIV>On Mar 21, 2007, at 4:50 =
> PM, wei huang =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">wrote:</DIV><BR =
> class=3D3D"Apple-interchange-newline"><BLOCKQUOTE =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">type=3D3D"cite"><DIV style=3D3D"margin-top: =
> 0px; margin-right: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">margin-bottom: 0px; margin-left: 0px; ">Hi,</DIV><DIV =
> <SPAN class=3D"Apple-converted-space">=A0</SPAN></DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">style=3D3D"margin-top:=3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; "><SPAN class=3D"Apple-converted-space">=A0</SPAN>0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">min-height: 14px; "><BR></DIV><DIV =
> style=3D3D"margin-top: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">Thanks <SPAN =
> class=3D"Apple-converted-space">=A0</SPAN></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">for =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">letting us know this =
> problem.</DIV><DIV style=3D3D"margin-top: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-right: 0px; margin-bottom: 0px; margin-left: =
> 0px; min-<SPAN class=3D"Apple-converted-space">=A0</SPAN></DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">height: =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">14px; =
> "><BR></DIV><DIV style=3D3D"margin-top: 0px; =
> margin-right: <SPAN class=3D"Apple-converted-space">=A0</SPAN></DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">margin-bottom: 0px; margin-left: 0px; ">We will try to reproduce =
> <SPAN class=3D"Apple-converted-space">=A0</SPAN></DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">the =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">problem on =
> our cluster. To help us on looking</DIV><DIV =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">style=3D3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-left: =
> 0px; ">into this problem, would you please let us <SPAN =
> class=3D"Apple-converted-space">=A0</SPAN></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">know the =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">following:</DIV><DIV =
> style=3D3D"margin-top: 0px; margin-right: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-bottom: 0px; margin-left: 0px; min-height: =
> 14px; "><BR></<SPAN =
> class=3D"Apple-converted-space">=A0</SPAN></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">DIV><DIV =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">style=3D3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">margin-left: 0px; ">1) The exact CFLAGS you are using when <SPAN =
> class=3D"Apple-converted-space">=A0</SPAN></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">configuring =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">mvapich2 (are you =
> using</DIV><DIV style=3D3D"margin-top: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-right: 0px; margin-bottom: 0px; margin-left: =
> 0px; ">default =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">scripts provided by =
> us?)</DIV><DIV style=3D3D"margin-top: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-right: 0px; margin-bottom: 0px; margin-left: =
> 0px; ">2) Any =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">runtime environment =
> variables you have setup up?</DIV><DIV =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">style=3D3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-left: =
> 0px; ">3) On how many nodes do you run the test?</<SPAN =
> class=3D"Apple-converted-space">=A0</SPAN></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">DIV><DIV =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">style=3D3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">margin-left: 0px; min-height: 14px; "><BR></DIV><DIV =
> <SPAN class=3D"Apple-converted-space">=A0</SPAN></DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">style=3D3D"margin-top: =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; =
> ">">Thanks.</DIV><DIV style=3D3D"margin-top: 0px; =
> margin-right: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">margin-bottom: 0px; =
> margin-left: 0px; min-height: 14px; "><BR></<SPAN =
> class=3D"Apple-converted-space">=A0</SPAN></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">DIV><DIV =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">style=3D3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">margin-left: 0px; ">Regards,</DIV><DIV =
> style=3D3D"margin-top: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">Wei =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">Huang</DIV><DIV style=3D3D"margin-top: 0px; =
> margin-right: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">margin-bottom: 0px; =
> margin-left: 0px; min-height: 14px; "><BR></<SPAN =
> class=3D"Apple-converted-space">=A0</SPAN></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">DIV><DIV =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">style=3D3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">margin-left: 0px; ">774 Dreese Lab, 2015 Neil =
> Ave,</DIV><DIV =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">style=3D3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-left: 0px; ">Dept. of =
> Computer Science and Engineering</<SPAN =
> class=3D"Apple-converted-space">=A0</SPAN></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">DIV><DIV =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">style=3D3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">margin-left: 0px; ">Ohio State University</DIV><DIV <SPAN =
> class=3D"Apple-converted-space">=A0</SPAN></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">style=3D3D"margin-top: =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">OH =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">43210</DIV><DIV =
> style=3D3D"margin-top: 0px; margin-right: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-bottom: 0px; margin-left: 0px; ">Tel: =
> (614)292-8501</<SPAN class=3D"Apple-converted-space">=A0</SPAN></DIV><D=
> IV style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">DIV><DIV =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">style=3D3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-left: 0px; min-height: =
> 14px; "><BR></DIV><DIV <SPAN =
> class=3D"Apple-converted-space">=A0</SPAN></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">style=3D3D"margin-top: =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">min-height: 14px; "><BR></DIV><DIV =
> style=3D3D"margin-top: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">On <SPAN =
> class=3D"Apple-converted-space">=A0</SPAN></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Wed, 21 =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">Mar 2007, David Gunter =
> wrote:</DIV><DIV style=3D3D"margin-top: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-right: 0px; margin-bottom: 0px; margin-left: =
> 0px; min-<SPAN class=3D"Apple-converted-space">=A0</SPAN></DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">height: =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">14px; =
> "><BR></DIV> <BLOCKQUOTE type=3D3D"cite"><DIV =
> style=3D3D"margin-<SPAN class=3D"Apple-converted-space">=A0</SPAN></DIV><D=
> IV style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">top: =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">I <SPAN =
> class=3D"Apple-converted-space">=A0</SPAN></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">have =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">built mvapich2 for an OFED-based =
> IB Opteron cluster.<SPAN =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">class=3D3D"Apple-converted-space">=3DA0 =
> </SPAN>When</DIV><DIV =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">style=3D3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-left: 0px; ">running =
> the Intel MPI Benchmarks (IMB3) I keep =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">seeing =
> the following</DIV><DIV style=3D3D"margin-top: 0px; =
> margin-<SPAN class=3D"Apple-converted-space">=A0</SPAN></DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">right: =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">0px; =
> margin-bottom: 0px; margin-left: 0px; ">errors messages in <SPAN =
> class=3D"Apple-converted-space">=A0</SPAN></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">many =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">spots, although the tests run to =
> completion:</DIV><DIV =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">style=3D3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-left: 0px; min-height: =
> 14px; "><BR></DIV><DIV <SPAN =
> class=3D"Apple-converted-space">=A0</SPAN></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">style=3D3D"margin-top: =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; <SPAN =
> class=3D"Apple-converted-space">=A0</SPAN></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">">Internal =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">error: communicator is =
> already on free list</DIV><DIV <SPAN =
> class=3D"Apple-converted-space">=A0</SPAN></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">style=3D3D"margin-top:=3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><SPAN =
> class=3D"Apple-converted-space">=A0</SPAN>0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">min-height: 14px; "><BR></DIV><DIV =
> style=3D3D"margin-top: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">What <SPAN =
> class=3D"Apple-converted-space">=A0</SPAN></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">is this =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">referring to?</DIV><DIV =
> style=3D3D"margin-top: 0px; margin-right: <SPAN =
> class=3D"Apple-converted-space">=A0</SPAN></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-bottom: 0px; margin-left: =
> 0px; min-height: 14px; "><BR></<SPAN =
> class=3D"Apple-converted-space">=A0</SPAN></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">DIV><DIV =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">style=3D3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">margin-left: 0px; ">Thanks.</DIV><DIV style=3D3D"margin-top:=
> 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">">--david</DIV><DIV style=3D3D"margin-top: 0px; =
> margin-right: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">margin-bottom: 0px; =
> margin-left: 0px; min-height: 14px; "><BR></<SPAN =
> class=3D"Apple-converted-space">=A0</SPAN></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">DIV><DIV =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">style=3D3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">margin-left: 0px; ">--</DIV><DIV style=3D3D"margin-top: =
> 0px; margin-<SPAN class=3D"Apple-converted-space">=A0</SPAN></DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">right: =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">0px; =
> margin-bottom: 0px; margin-left: 0px; ">David Gunter</<SPAN =
> class=3D"Apple-converted-space">=A0</SPAN></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">DIV><DIV =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">style=3D3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">margin-left: 0px; ">HPC-4: HPC Environments: Parallel Tools =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">Team</DIV><DIV =
> style=3D3D"margin-top: 0px; margin-right: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-bottom: 0px; margin-left: 0px; ">Los =
> Alamos National =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; =
> ">Laboratory</DIV><DIV style=3D3D"margin-top: 0px; =
> margin-right: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">margin-bottom: 0px; =
> margin-left: 0px; min-height: 14px; "><BR></<SPAN =
> class=3D"Apple-converted-space">=A0</SPAN></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">DIV><DIV =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">style=3D3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">margin-left: 0px; min-height: 14px; "><BR></DIV><DIV =
> <SPAN class=3D"Apple-converted-space">=A0</SPAN></DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">style=3D3D"margin-top: =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">min-height: 14px; =
> "><BR></DIV> </BLOCKQUOTE><DIV =
> style=3D3D"margin-<SPAN class=3D"Apple-converted-space">=A0</SPAN></DIV><D=
> IV style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">top: =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">min-height: 14px; "><BR></DIV> =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; =
> "></BLOCKQUOTE></DIV><BR></DIV></BODY></H=
> TML>=3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">--Apple-Mail-1-849288998--</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; min-height: 14px; "><BR></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">--=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D1750880957=3D=3D</DIV><DI=
> V style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">Content-Type: text/plain; =
> charset=3D"us-ascii"</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">MIME-Version: 1.0</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">Content-Transfer-Encoding: 7bit</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">Content-Disposition: inline</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; min-height: 14px; "><BR></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">_______________________________________________</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">mvapich-discuss mailing list</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; "><A =
> href=3D"mailto:mvapich-discuss at cse.ohio-state.edu">mvapich-discuss at cse.ohi=
> o-state.edu</A></DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; "><A =
> href=3D"http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss">h=
> ttp://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss</A></DIV><D=
> IV style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; min-height: 14px; "><BR></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">--=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D1750880957=3D=3D--</DIV><=
> DIV style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; min-height: 14px; "><BR></DIV> </BLOCKQUOTE><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; min-height: 14px; "><BR></DIV> </BLOCKQUOTE><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; min-height: 14px; "><BR></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">_______________________________________________</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">mvapich-discuss mailing list</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; "><A =
> href=3D"mailto:mvapich-discuss at cse.ohio-state.edu">mvapich-discuss at cse.ohi=
> o-state.edu</A></DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; "><A =
> href=3D"http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss">h=
> ttp://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss</A></DIV> =
> </BLOCKQUOTE><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; min-height: 14px; "><BR></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">--Apple-Mail-2-709635751</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">Content-Transfer-Encoding: quoted-printable</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">Content-Type: text/html;</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; "><SPAN class=3D"Apple-tab-span" =
> style=3D"white-space:pre"> </SPAN>charset=3DISO-8859-1</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; min-height: 14px; "><BR></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> "><HTML><BODY style=3D3D"word-wrap: break-word; =
> -khtml-nbsp-mode: space; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">-khtml-line-break: after-white-space; ">I am curious to know if you =
> were =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">able to reproduce this problem, =
> whether it has been fixed or =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">not.<DIV><BR =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">class=3D3D"khtml-block-placeholder"></DIV><DIV>Thanks,<=
> ;/DIV><DIV>david</DIV>=3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> "><DIV><DIV><SPAN class=3D3D"Apple-style-span" =
> style=3D3D"border-collapse: =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">separate; =
> border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family: =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">Helvetica; font-size: 12px; font-style: normal; =
> font-variant: normal; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">font-weight: =
> normal; letter-spacing: normal; line-height: normal; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">text-align: auto; -khtml-text-decorations-in-effect: =
> none; text-indent: =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">0px; =
> -apple-text-size-adjust: auto; text-transform: none; orphans: 2; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">white-space: normal; widows: 2; =
> word-spacing: 0px; "><SPAN =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">class=3D3D"Apple-style-span" style=3D3D"border-collapse: separate; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">border-spacing: 0px 0px; color: =
> rgb(0, 0, 0); font-family: Helvetica; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">font-size: 12px; font-style: normal; font-variant: normal; =
> font-weight: =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">normal; letter-spacing: normal; =
> line-height: normal; text-align: auto; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">-khtml-text-decorations-in-effect: none; text-indent: 0px; =3D</DIV><DIV=
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">-apple-text-size-adjust: auto; text-transform: none; =
> orphans: 2; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">white-space: normal; widows: 2; =
> word-spacing: 0px; "><SPAN =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">class=3D3D"Apple-style-span" style=3D3D"border-collapse: separate; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">border-spacing: 0px 0px; color: =
> rgb(0, 0, 0); font-family: Helvetica; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">font-size: 12px; font-style: normal; font-variant: normal; =
> font-weight: =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">normal; letter-spacing: normal; =
> line-height: normal; text-align: auto; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">-khtml-text-decorations-in-effect: none; text-indent: 0px; =3D</DIV><DIV=
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">-apple-text-size-adjust: auto; text-transform: none; =
> orphans: 2; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">white-space: normal; widows: 2; =
> word-spacing: 0px; "><SPAN =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">class=3D3D"Apple-style-span" style=3D3D"border-collapse: separate; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">border-spacing: 0px 0px; color: =
> rgb(0, 0, 0); font-family: Helvetica; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">font-size: 12px; font-style: normal; font-variant: normal; =
> font-weight: =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">normal; letter-spacing: normal; =
> line-height: normal; text-align: auto; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">-khtml-text-decorations-in-effect: none; text-indent: 0px; =3D</DIV><DIV=
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">-apple-text-size-adjust: auto; text-transform: none; =
> orphans: 2; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">white-space: normal; widows: 2; =
> word-spacing: 0px; "><SPAN =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">class=3D3D"Apple-style-span" style=3D3D"border-collapse: separate; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">border-spacing: 0px 0px; color: =
> rgb(0, 0, 0); font-family: Helvetica; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">font-size: 12px; font-style: normal; font-variant: normal; =
> font-weight: =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">normal; letter-spacing: normal; =
> line-height: normal; text-align: auto; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">-khtml-text-decorations-in-effect: none; text-indent: 0px; =3D</DIV><DIV=
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">-apple-text-size-adjust: auto; text-transform: none; =
> orphans: 2; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">white-space: normal; widows: 2; =
> word-spacing: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; =
> ">"><DIV>--</DIV><DIV>David =
> Gunter</DIV><DIV>HPC-4: HPC Environments: =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">Parallel Tools Team</DIV><DIV>Los Alamos =
> National =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; =
> ">Laboratory</DIV></SPAN></SPAN><BR =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; =
> ">class=3D3D"Apple-interchange-newline"></SPAN></SPAN></=
> SPAN> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; =
> "></DIV><BR><DIV><DIV>On Mar 22, 2007, at 11:20 =
> AM, David Gunter =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">wrote:</DIV><BR =
> class=3D3D"Apple-interchange-newline"><BLOCKQUOTE =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">type=3D3D"cite"><DIV style=3D3D"margin-top: =
> 0px; margin-right: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">margin-bottom: 0px; margin-left: 0px; ">I am using the released =
> 0.9.8 =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">version.</DIV><DIV =
> style=3D3D"margin-top: 0px; margin-right: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-bottom: 0px; margin-left: 0px; min-height: =
> 14px; "><BR></DIV><DIV =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">style=3D3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-left: =
> 0px; ">-david</DIV><DIV style=3D3D"margin-top: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; min-height: =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">14px; "><BR></DIV><DIV =
> style=3D3D"margin-top: 0px; margin-right: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-bottom: 0px; margin-left: 0px; ">On Mar =
> 22, 2007, at 9:37 AM, =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Dhabaleswar =
> Panda wrote:</DIV><DIV style=3D3D"margin-top: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-right: 0px; margin-bottom: 0px; margin-left: =
> 0px; min-height: =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">14px; =
> "><BR></DIV> <BLOCKQUOTE type=3D3D"cite"><DIV =
> style=3D3D"margin-top: =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Hi =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">David,</DIV><DIV =
> style=3D3D"margin-top: 0px; margin-right: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-bottom: 0px; margin-left: 0px; min-height: =
> 14px; "><BR></DIV><DIV =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">style=3D3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-left: =
> 0px; ">Thanks for this information. One more question - are =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">you using</DIV><DIV =
> style=3D3D"margin-top: 0px; margin-right: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-bottom: 0px; margin-left: 0px; ">MVAPICH2 =
> 0.9.8 `released' =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">version or the `branch' =
> version (with some</DIV><DIV style=3D3D"margin-top: =3D</DIV><DIV=
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">recent =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">fixes). If =
> you can let us know this information, it will help</DIV><DIV =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">style=3D3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">margin-left: 0px; ">us.</DIV><DIV style=3D3D"margin-top: =
> 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; min-height: =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">14px; "><BR></DIV><DIV =
> style=3D3D"margin-top: 0px; margin-right: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-bottom: 0px; margin-left: 0px; =
> ">Thanks,</DIV><DIV =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">style=3D3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-left: 0px; min-height: =
> 14px; "><BR></DIV><DIV style=3D3D"margin-top: =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">">DK</DIV><DIV style=3D3D"margin-top: 0px; margin-right: =
> 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-bottom: 0px; margin-left: =
> 0px; min-height: 14px; "><BR></DIV><DIV =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">style=3D3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-left: =
> 0px; min-height: 14px; "><BR></DIV> <BLOCKQUOTE =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">type=3D3D"cite"><DIV =
> style=3D3D"margin-top: 0px; margin-right: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-bottom: 0px; margin-left: 0px; ">I have =
> recompiled mvapich2 =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">without using =
> the --enable-debuginfo flag</DIV><DIV style=3D3D"margin-top: =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">and the =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">problem has gone away.<SPAN =
> class=3D3D"Apple-converted-space">=3DA0 =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; "></SPAN>However, I wish to have =
> debuginfo</DIV><DIV style=3D3D"margin-top: =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">">available to our =
> TotalView users so hopefully this can be =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">resolved.</DIV><DIV style=3D3D"margin-top: =
> 0px; margin-right: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">margin-bottom: 0px; margin-left: 0px; min-height: 14px; =
> "><BR></DIV><DIV =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">style=3D3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-left: 0px; ">Here is =
> the configuration that generates the error =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">message I saw</DIV><DIV =
> style=3D3D"margin-top: 0px; margin-right: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-bottom: 0px; margin-left: 0px; =
> ">previously:</DIV><DIV =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">style=3D3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-left: 0px; min-height: =
> 14px; "><BR></DIV><DIV style=3D3D"margin-top: =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">">./configure --prefix=3D3D/opt/mvapich2/mvapich2-0.9.8-gcc/ib =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">--with-</DIV><DIV =
> style=3D3D"margin-top: 0px; margin-right: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-bottom: 0px; margin-left: 0px; =
> ">openib=3D3D/usr/local/ofed =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">--enable-romio --with-file-system=3D3Dufs+nfs</DIV><DIV =
> style=3D3D"margin-top:=3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><SPAN =
> class=3D"Apple-converted-space">=A0</SPAN>0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">+panfs =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">--enable-sharedlibs=3D3Dgcc --enable-debuginfo =
> --enable-fast --</DIV><DIV =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">style=3D3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-left: 0px; =
> ">with-mpe</DIV><DIV style=3D3D"margin-top: 0px; =3D</DIV><DIV=
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-right: 0px; margin-bottom: 0px; margin-left: =
> 0px; min-height: =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">14px; =
> "><BR></DIV><DIV style=3D3D"margin-top: 0px; =
> margin-right: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">margin-bottom: 0px; =
> margin-left: 0px; ">OFED is ofed-1.1</DIV><DIV =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">style=3D3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-left: =
> 0px; min-height: 14px; "><BR></DIV><DIV =
> style=3D3D"margin-top: =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">">CFLAGS=3D3D-D_X86_64_ -D_SMP_ =
> -DUSE_HEADER_CACHING<SPAN =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">class=3D3D"Apple-converted-space">=3DA0 </SPAN>-DONE_SIDED =
> -</DIV><DIV =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">style=3D3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-left: 0px; =
> ">DMPID_USE_SEQUENCE_NUMBERS<SPAN =3D</DIV><DIV style=3D"margin-top:=
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">class=3D3D"Apple-converted-space">=3DA0 =
> </SPAN>-I/usr/local/ofed/include -O2 =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; "><SPAN class=3D3D"Apple-converted-space">=3DA0 =
> </SPAN>-</DIV><DIV =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">style=3D3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-left: 0px; =
> ">D_SHMEM_COLL_</DIV><DIV style=3D3D"margin-top: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; min-height: =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">14px; "><BR></DIV><DIV =
> style=3D3D"margin-top: 0px; margin-right: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-bottom: 0px; margin-left: 0px; =
> ">CC=3D3D/usr/bin/gcc</DIV><DIV =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">style=3D3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-left: =
> 0px; ">CXX=3D3D/usr/bin/g++</DIV><DIV style=3D3D"margin-top: =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">">FC=3D3D/usr/bin/gfortran</DIV><DIV style=3D3D"margin-top: =
> 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">">F77=3D3D/usr/bin/gfortran</DIV><DIV style=3D3D"margin-top:=
> 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">">F90=3D3D/usr/bin/gfortran</DIV><DIV style=3D3D"margin-top:=
> 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; min-height: =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">14px; "><BR></DIV><DIV =
> style=3D3D"margin-top: 0px; margin-right: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-bottom: 0px; margin-left: 0px; ">The test =
> was run on 32 process, =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">16 process, 8 =
> process and 4 process -</DIV><DIV style=3D3D"margin-top: =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">all of =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">which generated this error =
> message.</DIV><DIV style=3D3D"margin-top: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-right: 0px; margin-bottom: 0px; margin-left: =
> 0px; min-height: =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">14px; =
> "><BR></DIV><DIV style=3D3D"margin-top: 0px; =
> margin-right: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">margin-bottom: 0px; =
> margin-left: 0px; ">Thanks,</DIV><DIV =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">style=3D3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-left: =
> 0px; ">david</DIV><DIV style=3D3D"margin-top: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; min-height: =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">14px; "><BR></DIV><DIV =
> style=3D3D"margin-top: 0px; margin-right: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-bottom: 0px; margin-left: 0px; min-height: =
> 14px; "><BR></DIV><DIV =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">style=3D3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-left: =
> 0px; min-height: 14px; "><BR></DIV><DIV =
> style=3D3D"margin-top: =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">">--</DIV><DIV style=3D3D"margin-top: =
> 0px; margin-right: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">margin-bottom: 0px; margin-left: 0px; ">David =
> Gunter</DIV><DIV =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">style=3D3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-left: 0px; ">HPC-4: =
> HPC Environments: Parallel Tools =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">Team</DIV><DIV style=3D3D"margin-top: 0px; margin-right: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-bottom: 0px; margin-left: =
> 0px; ">Los Alamos National =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">Laboratory</DIV><DIV style=3D3D"margin-top: 0px; =
> margin-right: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">margin-bottom: 0px; =
> margin-left: 0px; min-height: 14px; "><BR></DIV><DIV =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">style=3D3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">margin-left: 0px; min-height: 14px; "><BR></DIV><DIV =
> style=3D3D"margin-top: =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">On Mar =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">21, 2007, at 4:50 PM, wei huang =
> wrote:</DIV><DIV style=3D3D"margin-top: =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">min-height: 14px; =
> "><BR></DIV> <BLOCKQUOTE type=3D3D"cite"><DIV =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">style=3D3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">margin-left: 0px; ">Hi,</DIV><DIV style=3D3D"margin-top: =
> 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; min-height: =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">14px; "><BR></DIV><DIV =
> style=3D3D"margin-top: 0px; margin-right: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-bottom: 0px; margin-left: 0px; ">Thanks =
> for letting us know this =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">problem.</DIV><DIV style=3D3D"margin-top: 0px; margin-right: =
> 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-bottom: 0px; margin-left: =
> 0px; min-height: 14px; "><BR></DIV><DIV =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">style=3D3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-left: =
> 0px; ">We will try to reproduce the problem on our cluster. =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">To help us on</DIV><DIV =
> style=3D3D"margin-top: 0px; margin-right: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-bottom: 0px; margin-left: 0px; =
> ">looking</DIV><DIV =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">style=3D3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-left: 0px; ">into this =
> problem, would you please let us know the =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">following:</DIV><DIV style=3D3D"margin-top: =
> 0px; margin-right: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">margin-bottom: 0px; margin-left: 0px; min-height: 14px; =
> "><BR></DIV><DIV =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">style=3D3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-left: 0px; ">1) The =
> exact CFLAGS you are using when configuring =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">mvapich2 (are</DIV><DIV =
> style=3D3D"margin-top: 0px; margin-right: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-bottom: 0px; margin-left: 0px; ">you =
> using</DIV><DIV =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">style=3D3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-left: 0px; ">default =
> scripts provided by us?)</DIV><DIV =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">style=3D3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-left: =
> 0px; ">2) Any runtime environment variables you have setup =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">up?</DIV><DIV =
> style=3D3D"margin-top: 0px; margin-right: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-bottom: 0px; margin-left: 0px; ">3) On how =
> many nodes do you run =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">the =
> test?</DIV><DIV style=3D3D"margin-top: 0px; margin-right: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-bottom: 0px; margin-left: =
> 0px; min-height: 14px; "><BR></DIV><DIV =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">style=3D3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-left: =
> 0px; ">Thanks.</DIV><DIV style=3D3D"margin-top: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; min-height: =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">14px; "><BR></DIV><DIV =
> style=3D3D"margin-top: 0px; margin-right: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-bottom: 0px; margin-left: 0px; =
> ">Regards,</DIV><DIV =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">style=3D3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-left: 0px; ">Wei =
> Huang</DIV><DIV style=3D3D"margin-top: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-right: 0px; margin-bottom: 0px; margin-left: =
> 0px; min-height: =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">14px; =
> "><BR></DIV><DIV style=3D3D"margin-top: 0px; =
> margin-right: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">margin-bottom: 0px; =
> margin-left: 0px; ">774 Dreese Lab, 2015 Neil =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">Ave,</DIV><DIV style=3D3D"margin-top: 0px; =
> margin-right: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">margin-bottom: 0px; =
> margin-left: 0px; ">Dept. of Computer Science and =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">Engineering</DIV><DIV style=3D3D"margin-top: =
> 0px; margin-right: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">margin-bottom: 0px; margin-left: 0px; ">Ohio State =
> University</DIV><DIV =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">style=3D3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-left: 0px; ">OH =
> 43210</DIV><DIV style=3D3D"margin-top: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-right: 0px; margin-bottom: 0px; margin-left: =
> 0px; ">Tel: =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; =
> ">(614)292-8501</DIV><DIV style=3D3D"margin-top: 0px; =
> margin-right: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">margin-bottom: 0px; =
> margin-left: 0px; min-height: 14px; "><BR></DIV><DIV =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">style=3D3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">margin-left: 0px; min-height: 14px; "><BR></DIV><DIV =
> style=3D3D"margin-top: =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">On Wed, =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">21 Mar 2007, David Gunter =
> wrote:</DIV><DIV style=3D3D"margin-top: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-right: 0px; margin-bottom: 0px; margin-left: =
> 0px; min-height: =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">14px; =
> "><BR></DIV> <BLOCKQUOTE type=3D3D"cite"><DIV =
> style=3D3D"margin-top: =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">I have =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">built mvapich2 for an OFED-based =
> IB Opteron cluster.<SPAN =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">class=3D3D"Apple-converted-space">=3DA0 =
> </SPAN>When</DIV><DIV =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">style=3D3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-left: 0px; ">running =
> the Intel MPI Benchmarks (IMB3) I keep =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">seeing =
> the following</DIV><DIV style=3D3D"margin-top: 0px; =
> margin-right: =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">0px; margin-bottom: 0px; =
> margin-left: 0px; ">errors messages in many =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">spots, although the tests run to =
> completion:</DIV><DIV =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">style=3D3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-left: 0px; min-height: =
> 14px; "><BR></DIV><DIV style=3D3D"margin-top: =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">Internal =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">error: communicator is already on free =
> list</DIV><DIV style=3D3D"margin-top:=3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; "><SPAN class=3D"Apple-converted-space">=A0</SPAN>0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">min-height: 14px; "><BR></DIV><DIV =
> style=3D3D"margin-top: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">What is this =3D</DIV><DIV=
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">referring to?</DIV><DIV =
> style=3D3D"margin-top: 0px; margin-right: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-bottom: 0px; margin-left: 0px; min-height: =
> 14px; "><BR></DIV><DIV =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">style=3D3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-left: =
> 0px; ">Thanks.</DIV><DIV style=3D3D"margin-top: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">">--david</DIV><DIV style=3D3D"margin-top: 0px; =
> margin-right: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">margin-bottom: 0px; =
> margin-left: 0px; min-height: 14px; "><BR></DIV><DIV =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">style=3D3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">margin-left: 0px; ">--</DIV><DIV style=3D3D"margin-top: =
> 0px; margin-right: =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">0px; margin-bottom: 0px; =
> margin-left: 0px; ">David Gunter</DIV><DIV =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">style=3D3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-left: =
> 0px; ">HPC-4: HPC Environments: Parallel Tools =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">Team</DIV><DIV style=3D3D"margin-top: 0px; =
> margin-right: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">margin-bottom: 0px; =
> margin-left: 0px; ">Los Alamos National =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">Laboratory</DIV><DIV style=3D3D"margin-top: =
> 0px; margin-right: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">margin-bottom: 0px; margin-left: 0px; min-height: 14px; =
> "><BR></DIV><DIV =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">style=3D3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-left: 0px; min-height: =
> 14px; "><BR></DIV><DIV style=3D3D"margin-top: =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">min-height: 14px; "><BR></DIV> =
> </BLOCKQUOTE><DIV style=3D3D"margin-top: =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">min-height: 14px; =
> "><BR></DIV> </BLOCKQUOTE><DIV =
> style=3D3D"margin-top: =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">min-height: 14px; "><BR></DIV><DIV =
> style=3D3D"margin-top: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; min-height: =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">14px; "><BR></DIV><DIV =
> style=3D3D"margin-top: 0px; margin-right: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-bottom: 0px; margin-left: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">">--Apple-Mail-1-849288998</DIV><DIV =
> style=3D3D"margin-top: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">">Content-Transfer-Encoding: =
> quoted-printable</DIV><DIV =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">style=3D3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-left: 0px; =
> ">Content-Type: text/html;</DIV><DIV =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">style=3D3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-left: =
> 0px; "><SPAN class=3D3D"Apple-tab-span" =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">style=3D3D"white-space:pre"><SPAN =
> class=3D"Apple-tab-span" style=3D"white-space:pre"> =
> </SPAN></SPAN>charset=3D3DISO-8859-1</DIV><DIV =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">style=3D3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">margin-left: 0px; min-height: 14px; "><BR></DIV><DIV =
> style=3D3D"margin-top: =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">">&lt;HTML&gt;&lt;BODY =
> style=3D3D3D"word-wrap: break-word; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">-khtml-nbsp-mode: space; =3D3D</DIV><DIV style=3D3D"margin-top:=
> 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">">-khtml-line-break: after-white-space; "&gt;I have recompiled =
> mvapich2 =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">=3D3D</DIV><DIV =
> style=3D3D"margin-top: 0px; margin-right: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-bottom: 0px; margin-left: 0px; ">without =
> using the =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">--enable-debuginfo flag and the =
> problem has gone =3D3D</DIV><DIV =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">style=3D3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-left: =
> 0px; ">away.=3D3DA0 However, I wish to have debuginfo =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">available to our TotalView =3D3D</DIV><DIV =
> style=3D3D"margin-top: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">users so =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">hopefully this can be =
> resolved.&lt;DIV&gt;&lt;BR =3D3D</DIV><DIV =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">style=3D3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">margin-left: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">">class=3D3D3D"khtml-block-placeholder"&gt;&lt;/DIV&gt;&a=
> mp;lt;DIV&gt;Here is =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">the =
> configuration =3D3D</DIV><DIV style=3D3D"margin-top: 0px; =
> margin-right: =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">0px; margin-bottom: 0px; =
> margin-left: 0px; ">that generates the error =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">message I saw =
> previously:&lt;/DIV&gt;&lt;DIV&gt;&lt;BR =
> =3D3D</DIV><DIV =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">style=3D3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-left: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; =
> ">">class=3D3D3D"khtml-block-placeholder"&gt;&lt;/DIV&gt;&a=
> mp;lt;DIV&gt;./configur=3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">e =
> =3D3D</DIV><DIV style=3D3D"margin-top: 0px; margin-right: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-bottom: 0px; margin-left: =
> 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; =
> ">">--prefix=3D3D3D/opt/mvapich2/mvapich2-0.9.8-gcc/ib=3D3DA0--with-ope=
> nib=3D3D3D/=3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">usr/loca=3D3D</DIV><DIV =
> style=3D3D"margin-top: 0px; margin-right: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-bottom: 0px; margin-left: 0px; ">l/ofed =
> --enable-romio =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; =
> ">--with-file-system=3D3D3Dufs+nfs+panfs =3D3D</DIV><DIV =
> style=3D3D"margin-top: =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">">--enable-sharedlibs=3D3D3Dgcc =
> --enable-debuginfo --enable-fast =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">=3D3D</DIV><DIV style=3D3D"margin-top: 0px; margin-right: =
> 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-bottom: 0px; margin-left: =
> 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; =
> ">">--with-mpe&lt;/DIV&gt;&lt;DIV&gt;&lt;BR =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; =
> ">class=3D3D3D"khtml-block-placeholder"&gt;&lt;/DIV&gt;&lt=
> ;DIV&gt;OFED=3D3D</DIV=3D</DIV> <BLOCKQUOTE type=3D"cite"><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; "><DIV style=3D3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; =3D</DIV> </BLOCKQUOTE><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">margin-left: 0px; "><SPAN =
> class=3D3D"Apple-converted-space">=3DA0</SPAN>is =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; =
> ">ofed-1.1&lt;/DIV&gt;&lt;DIV&gt;&lt;BR =
> =3D3D</DIV><DIV style=3D3D"margin-top: =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; =
> ">">class=3D3D3D"khtml-block-placeholder"&gt;&lt;/DIV&gt;&a=
> mp;lt;DIV&gt;CFLAGS=3D3D3=3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">D-D_X86_64_ =
> -D_SMP_ =3D3D</DIV><DIV style=3D3D"margin-top: 0px; =3D</DIV><DIV=
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-right: 0px; margin-bottom: 0px; margin-left: =
> 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">">-DUSE_HEADER_CACHING=3D3DA0 =
> -DONE_SIDED -DMPID_USE_SEQUENCE_NUMBERS=3D3DA0 =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">=3D3D</DIV><DIV style=3D3D"margin-top: 0px; =
> margin-right: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">margin-bottom: 0px; =
> margin-left: 0px; ">-I/usr/local/ofed/include =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">-O2=3D3DA0=3D3DA0 =
> -D_SHMEM_COLL_&lt;/DIV&gt;&lt;DIV&gt;&lt;BR =
> =3D3D</DIV><DIV =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">style=3D3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-left: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; =
> ">">class=3D3D3D"khtml-block-placeholder"&gt;&lt;/DIV&gt;&a=
> mp;lt;DIV&gt;CC=3D3D3D/us=3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">r/bin/gcc&lt;/DIV&gt;&lt;DIV&gt;C=3D3D</DIV><DI=
> V style=3D3D"margin-top: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; =
> ">">XX=3D3D3D/usr/bin/g++&lt;/DIV&gt;&lt;DIV&gt;FC=3D3D=
> 3D/usr/bin/gfortran&lt;/D=3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">IV&gt;&lt;DIV&gt;F77=3D3D3D/usr/bi=3D3D</DIV><DIV =
> style=3D3D"margin-top: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; =
> ">">n/gfortran&lt;/DIV&gt;&lt;DIV&gt;F90=3D3D3D/usr/bin=
> /gfortran&lt;/DIV&gt;&l=3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">t;DIV&gt;&lt;BR =3D3D</DIV><DIV style=3D3D"margin-top: =
> 0px; margin-right: =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">0px; margin-bottom: 0px; =
> margin-left: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; =
> ">">class=3D3D3D"khtml-block-placeholder"&gt;&lt;/DIV&gt;&a=
> mp;lt;DIV&gt;The test =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">was run on 32 =
> =3D3D</DIV><DIV style=3D3D"margin-top: 0px; margin-right: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-bottom: 0px; margin-left: =
> 0px; ">process, 16 process, 8 process =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">and 4 process - all of which generated =
> =3D3D</DIV><DIV style=3D3D"margin-top: =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">this =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">error =
> message.&lt;/DIV&gt;&lt;DIV&gt;&lt;BR =
> =3D3D</DIV><DIV =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">style=3D3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-left: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; =
> ">">class=3D3D3D"khtml-block-placeholder"&gt;&lt;/DIV&gt;&a=
> mp;lt;DIV&gt;Thanks,&lt=3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">;/DIV&gt;&lt;DIV&gt;david&lt;/DIV&gt;=3D3D</DIV&g=
> t;<DIV style=3D3D"margin-top: =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">">&lt;DIV&gt;&lt;BR =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; =
> ">class=3D3D3D"khtml-block-placeholder"&gt;&lt;/DIV&gt;&lt=
> ;DIV&gt;&lt;BR =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">=3D3D</DIV><DIV style=3D3D"margin-top: 0px; margin-right: =
> 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-bottom: 0px; margin-left: =
> 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; =
> ">">class=3D3D3D"khtml-block-placeholder"&gt;&lt;/DIV&gt;&a=
> mp;lt;DIV&gt;&lt;BR&gt;=3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">&lt;DIV&gt; &lt;SPAN =3D3D</DIV><DIV =
> style=3D3D"margin-top: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">">class=3D3D3D"Apple-style-span" =
> style=3D3D3D"border-collapse: separate; =3D</DIV><DIV style=3D"margin-top:=
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">=3D3D</DIV><DIV style=3D3D"margin-top: 0px; margin-right: =
> 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-bottom: 0px; margin-left: =
> 0px; ">border-spacing: 0px 0px; color: =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">rgb(0, 0, 0); font-family: Helvetica; =
> =3D3D</DIV><DIV style=3D3D"margin-top: =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">">font-size: 12px; =
> font-style: normal; font-variant: normal; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">font-weight: =3D3D</DIV><DIV =
> style=3D3D"margin-top: 0px; margin-right: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-bottom: 0px; margin-left: 0px; ">normal; =
> letter-spacing: normal; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">line-height: =
> normal; text-align: auto; =3D3D</DIV><DIV style=3D3D"margin-top: =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">">-khtml-text-decorations-in-effect: none; text-indent: 0px; =
> =3D3D</DIV><DIV=3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><SPAN =
> class=3D"Apple-converted-space">=A0</SPAN>style=3D3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">margin-left: 0px; ">-apple-text-size-adjust: auto; text-transform: =
> none; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">orphans: 2; =
> =3D3D</DIV><DIV style=3D3D"margin-top: 0px; margin-right: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-bottom: 0px; margin-left: =
> 0px; ">white-space: normal; widows: 2; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">word-spacing: 0px; "&gt;&lt;SPAN =
> =3D3D</DIV><DIV style=3D3D"margin-top: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-right: 0px; margin-bottom: 0px; margin-left: =
> 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; =
> ">">class=3D3D3D"Apple-style-span" style=3D3D3D"border-collapse: =
> separate; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">=3D3D</DIV><DIV =
> style=3D3D"margin-top: 0px; margin-right: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-bottom: 0px; margin-left: 0px; =
> ">border-spacing: 0px 0px; color: =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">rgb(0, =
> 0, 0); font-family: Helvetica; =3D3D</DIV><DIV =
> style=3D3D"margin-top: =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">">font-size: 12px; font-style: normal; =
> font-variant: normal; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">font-weight: =
> =3D3D</DIV><DIV style=3D3D"margin-top: 0px; margin-right: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-bottom: 0px; margin-left: =
> 0px; ">normal; letter-spacing: normal; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">line-height: normal; text-align: auto; =
> =3D3D</DIV><DIV style=3D3D"margin-top: =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; =
> ">">-khtml-text-decorations-in-effect: none; text-indent: 0px; =
> =3D3D</DIV><DIV=3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><SPAN =
> class=3D"Apple-converted-space">=A0</SPAN>style=3D3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">margin-left: 0px; ">-apple-text-size-adjust: auto; text-transform: =
> none; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">orphans: 2; =
> =3D3D</DIV><DIV style=3D3D"margin-top: 0px; margin-right: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-bottom: 0px; margin-left: =
> 0px; ">white-space: normal; widows: 2; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">word-spacing: 0px; "&gt;&lt;SPAN =
> =3D3D</DIV><DIV style=3D3D"margin-top: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-right: 0px; margin-bottom: 0px; margin-left: =
> 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; =
> ">">class=3D3D3D"Apple-style-span" style=3D3D3D"border-collapse: =
> separate; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">=3D3D</DIV><DIV =
> style=3D3D"margin-top: 0px; margin-right: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-bottom: 0px; margin-left: 0px; =
> ">border-spacing: 0px 0px; color: =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">rgb(0, =
> 0, 0); font-family: Helvetica; =3D3D</DIV><DIV =
> style=3D3D"margin-top: =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">">font-size: 12px; font-style: normal; =
> font-variant: normal; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">font-weight: =
> =3D3D</DIV><DIV style=3D3D"margin-top: 0px; margin-right: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-bottom: 0px; margin-left: =
> 0px; ">normal; letter-spacing: normal; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">line-height: normal; text-align: auto; =
> =3D3D</DIV><DIV style=3D3D"margin-top: =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; =
> ">">-khtml-text-decorations-in-effect: none; text-indent: 0px; =
> =3D3D</DIV><DIV=3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><SPAN =
> class=3D"Apple-converted-space">=A0</SPAN>style=3D3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">margin-left: 0px; ">-apple-text-size-adjust: auto; text-transform: =
> none; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">orphans: 2; =
> =3D3D</DIV><DIV style=3D3D"margin-top: 0px; margin-right: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-bottom: 0px; margin-left: =
> 0px; ">white-space: normal; widows: 2; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">word-spacing: 0px; "&gt;&lt;SPAN =
> =3D3D</DIV><DIV style=3D3D"margin-top: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-right: 0px; margin-bottom: 0px; margin-left: =
> 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; =
> ">">class=3D3D3D"Apple-style-span" style=3D3D3D"border-collapse: =
> separate; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">=3D3D</DIV><DIV =
> style=3D3D"margin-top: 0px; margin-right: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-bottom: 0px; margin-left: 0px; =
> ">border-spacing: 0px 0px; color: =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">rgb(0, =
> 0, 0); font-family: Helvetica; =3D3D</DIV><DIV =
> style=3D3D"margin-top: =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">">font-size: 12px; font-style: normal; =
> font-variant: normal; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">font-weight: =
> =3D3D</DIV><DIV style=3D3D"margin-top: 0px; margin-right: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-bottom: 0px; margin-left: =
> 0px; ">normal; letter-spacing: normal; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">line-height: normal; text-align: auto; =
> =3D3D</DIV><DIV style=3D3D"margin-top: =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; =
> ">">-khtml-text-decorations-in-effect: none; text-indent: 0px; =
> =3D3D</DIV><DIV=3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><SPAN =
> class=3D"Apple-converted-space">=A0</SPAN>style=3D3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">margin-left: 0px; ">-apple-text-size-adjust: auto; text-transform: =
> none; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">orphans: 2; =
> =3D3D</DIV><DIV style=3D3D"margin-top: 0px; margin-right: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-bottom: 0px; margin-left: =
> 0px; ">white-space: normal; widows: 2; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">word-spacing: 0px; "&gt;&lt;SPAN =
> =3D3D</DIV><DIV style=3D3D"margin-top: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-right: 0px; margin-bottom: 0px; margin-left: =
> 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; =
> ">">class=3D3D3D"Apple-style-span" style=3D3D3D"border-collapse: =
> separate; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">=3D3D</DIV><DIV =
> style=3D3D"margin-top: 0px; margin-right: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-bottom: 0px; margin-left: 0px; =
> ">border-spacing: 0px 0px; color: =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">rgb(0, =
> 0, 0); font-family: Helvetica; =3D3D</DIV><DIV =
> style=3D3D"margin-top: =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">">font-size: 12px; font-style: normal; =
> font-variant: normal; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">font-weight: =
> =3D3D</DIV><DIV style=3D3D"margin-top: 0px; margin-right: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-bottom: 0px; margin-left: =
> 0px; ">normal; letter-spacing: normal; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">line-height: normal; text-align: auto; =
> =3D3D</DIV><DIV style=3D3D"margin-top: =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; =
> ">">-khtml-text-decorations-in-effect: none; text-indent: 0px; =
> =3D3D</DIV><DIV=3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><SPAN =
> class=3D"Apple-converted-space">=A0</SPAN>style=3D3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">margin-left: 0px; ">-apple-text-size-adjust: auto; text-transform: =
> none; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">orphans: 2; =
> =3D3D</DIV><DIV style=3D3D"margin-top: 0px; margin-right: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-bottom: 0px; margin-left: =
> 0px; ">white-space: normal; widows: 2; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">word-spacing: 0px; =3D3D</DIV><DIV =
> style=3D3D"margin-top: 0px; margin-right: =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">0px; margin-bottom: 0px; margin-left: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; =
> ">">"&gt;&lt;DIV&gt;--&lt;/DIV&gt;&lt;DIV&g=
> t;David =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; =
> ">Gunter&lt;/DIV&gt;&lt;DIV&gt;HPC-4: HPC Environments: =
> =3D3D</DIV><DIV =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">style=3D3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-left: 0px; ">Parallel =
> Tools Team&lt;/DIV&gt;&lt;DIV&gt;Los Alamos =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">National =3D3D</DIV><DIV =
> style=3D3D"margin-top: 0px; margin-right: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-bottom: 0px; margin-left: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; =
> ">">Laboratory&lt;/DIV&gt;&lt;/SPAN&gt;&lt;/SPAN&am=
> p;gt;&lt;BR =3D3D</DIV><DIV =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">style=3D3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-left: =
> 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; =
> ">">class=3D3D3D"Apple-interchange-newline"&gt;&lt;/SPAN&gt=
> ;&lt;/SPAN&gt;&lt;/=3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">SPAN&gt; =
> =3D3D</DIV><DIV style=3D3D"margin-top: 0px; margin-right: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-bottom: 0px; margin-left: =
> 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; =
> ">">&lt;/DIV&gt;&lt;BR&gt;&lt;DIV&gt;&lt;DI=
> V&gt;On Mar 21, 2007, at 4:50 =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">PM, wei =
> huang =3D3D</DIV><DIV style=3D3D"margin-top: 0px; margin-right: =
> 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-bottom: 0px; margin-left: =
> 0px; ">wrote:&lt;/DIV&gt;&lt;BR =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; =
> ">class=3D3D3D"Apple-interchange-newline"&gt;&lt;BLOCKQUOTE =
> =3D3D</DIV><DIV =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">style=3D3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-left: 0px; =
> ">type=3D3D3D"cite"&gt;&lt;DIV style=3D3D3D"margin-top: =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">0px; margin-right: 0px; =
> =3D3D</DIV><DIV style=3D3D"margin-top: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-right: 0px; margin-bottom: 0px; margin-left: =
> 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">">margin-bottom: 0px; =
> margin-left: 0px; "&gt;Hi,&lt;/DIV&gt;&lt;DIV =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; =
> ">style=3D3D3D"margin-top:=3D3D</DIV><DIV style=3D3D"margin-top: =
> 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; "><SPAN =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; =
> ">class=3D3D"Apple-converted-space">=3DA0</SPAN>0px; =
> margin-right: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">margin-bottom: 0px; =
> margin-left: 0px; =3D3D</DIV><DIV style=3D3D"margin-top: =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">">min-height: 14px; =
> "&gt;&lt;BR&gt;&lt;/DIV&gt;&lt;DIV =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">style=3D3D3D"margin-top: 0px; =3D3D</DIV><DIV=
> style=3D3D"margin-top: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">margin-right: =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">0px; margin-bottom: 0px; =
> margin-left: 0px; "&gt;Thanks for =3D3D</DIV><DIV =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">style=3D3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">margin-left: 0px; ">letting us know this =
> problem.&lt;/DIV&gt;&lt;DIV =3D</DIV><DIV style=3D"margin-top:=
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">style=3D3D3D"margin-top: 0px; =3D3D</DIV><DIV =
> style=3D3D"margin-top: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">margin-right: =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">0px; margin-bottom: 0px; =
> margin-left: 0px; min-height: =3D3D</DIV><DIV =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">style=3D3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-left: =
> 0px; ">14px; =
> "&gt;&lt;BR&gt;&lt;/DIV&gt;&lt;DIV =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">style=3D3D3D"margin-top: 0px; margin-right: 0px; =
> =3D3D</DIV><DIV =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">style=3D3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-left: 0px; =
> ">margin-bottom: 0px; margin-left: 0px; "&gt;We will =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">try to reproduce the =3D3D</DIV><DIV =
> style=3D3D"margin-top: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">problem on =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">our cluster. To help us on =
> looking&lt;/DIV&gt;&lt;DIV =3D3D</DIV><DIV =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">style=3D3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">margin-left: 0px; ">style=3D3D3D"margin-top: 0px; margin-right: =
> 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-bottom: 0px; =
> =3D3D</DIV><DIV style=3D3D"margin-top: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-right: 0px; margin-bottom: 0px; margin-left: =
> 0px; ">margin-left: =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">0px; =
> "&gt;into this problem, would you please let us know the =3D</DIV><DIV=
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">=3D3D</DIV><DIV style=3D3D"margin-top: 0px; =
> margin-right: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">margin-bottom: 0px; =
> margin-left: 0px; ">following:&lt;/DIV&gt;&lt;DIV =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">style=3D3D3D"margin-top: 0px; =
> margin-right: 0px; =3D3D</DIV><DIV =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">style=3D3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-left: =
> 0px; ">margin-bottom: 0px; margin-left: 0px; min-height: =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">14px; =
> "&gt;&lt;BR&gt;&lt;/DIV&gt;&lt;DIV =
> =3D3D</DIV><DIV =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">style=3D3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-left: 0px; =
> ">style=3D3D3D"margin-top: 0px; margin-right: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-bottom: 0px; =3D3D</DIV><DIV =
> style=3D3D"margin-top: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">margin-left: =3D</DIV><DIV=
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">0px; "&gt;1) The exact CFLAGS you are using when =
> configuring =3D3D</DIV><DIV=3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><SPAN =
> class=3D"Apple-converted-space">=A0</SPAN>style=3D3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">margin-left: 0px; ">mvapich2 (are you =
> using&lt;/DIV&gt;&lt;DIV =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">style=3D3D3D"margin-top: 0px; =3D3D</DIV><DIV =
> style=3D3D"margin-top: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">margin-right: =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">0px; margin-bottom: 0px; =
> margin-left: 0px; "&gt;default =3D3D</DIV><DIV =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">style=3D3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-left: =
> 0px; ">scripts provided by us?)&lt;/DIV&gt;&lt;DIV =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">style=3D3D3D"margin-top: 0px; =
> =3D3D</DIV><DIV style=3D3D"margin-top: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-right: 0px; margin-bottom: 0px; margin-left: =
> 0px; ">margin-right: =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">0px; =
> margin-bottom: 0px; margin-left: 0px; "&gt;2) Any =
> =3D3D</DIV><DIV =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">style=3D3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-left: 0px; ">runtime =
> environment variables you have setup =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">up?&lt;/DIV&gt;&lt;DIV =3D3D</DIV><DIV =
> style=3D3D"margin-top: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">">style=3D3D3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">=3D3D</DIV><DIV style=3D3D"margin-top: 0px; margin-right: =
> 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-bottom: 0px; margin-left: =
> 0px; ">margin-left: 0px; "&gt;3) On how =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">many nodes do you run the =
> test?&lt;/DIV&gt;&lt;DIV =3D3D</DIV><DIV =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">style=3D3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">margin-left: 0px; ">style=3D3D3D"margin-top: 0px; margin-right: =
> 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-bottom: 0px; =
> =3D3D</DIV><DIV style=3D3D"margin-top: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-right: 0px; margin-bottom: 0px; margin-left: =
> 0px; ">margin-left: =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">0px; =
> min-height: 14px; =
> "&gt;&lt;BR&gt;&lt;/DIV&gt;&lt;DIV =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">style=3D3D3D"margin-top: =3D3D</DIV><DIV =
> style=3D3D"margin-top: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-right: 0px; margin-bottom: 0px; margin-left: =
> 0px; =3D3D</DIV><DIV =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">style=3D3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-left: 0px; =
> ">"&gt;Thanks.&lt;/DIV&gt;&lt;DIV =
> style=3D3D3D"margin-top:=3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><SPAN =
> class=3D"Apple-converted-space">=A0</SPAN>0px; margin-right: 0px; =
> =3D3D</DIV><DIV style=3D3D"margin-top: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-right: 0px; margin-bottom: 0px; margin-left: =
> 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">">margin-bottom: 0px; =
> margin-left: 0px; min-height: 14px; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">"&gt;&lt;BR&gt;&lt;/DIV&gt;&lt;DIV =
> =3D3D</DIV><DIV style=3D3D"margin-top: =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">">style=3D3D3D"margin-top:=
> 0px; margin-right: 0px; margin-bottom: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">=3D3D</DIV><DIV style=3D3D"margin-top: 0px; =
> margin-right: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">margin-bottom: 0px; =
> margin-left: 0px; ">margin-left: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">"&gt;Regards,&lt;/DIV&gt;&lt;DIV =
> style=3D3D3D"margin-top: 0px; =3D3D</DIV><DIV=3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; "><SPAN =
> class=3D"Apple-converted-space">=A0</SPAN>style=3D3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">margin-left: 0px; ">margin-right: 0px; margin-bottom: 0px; =
> margin-left: =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">0px; "&gt;Wei =
> =3D3D</DIV><DIV style=3D3D"margin-top: 0px; margin-right: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-bottom: 0px; margin-left: =
> 0px; ">Huang&lt;/DIV&gt;&lt;DIV =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">style=3D3D3D"margin-top: 0px; margin-right: 0px; =
> =3D3D</DIV><DIV =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">style=3D3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-left: 0px; =
> ">margin-bottom: 0px; margin-left: 0px; min-height: =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">14px; =
> "&gt;&lt;BR&gt;&lt;/DIV&gt;&lt;DIV =
> =3D3D</DIV><DIV =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">style=3D3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-left: 0px; =
> ">style=3D3D3D"margin-top: 0px; margin-right: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-bottom: 0px; =3D3D</DIV><DIV =
> style=3D3D"margin-top: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">margin-left: =3D</DIV><DIV=
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">0px; "&gt;774 Dreese Lab, 2015 Neil =
> Ave,&lt;/DIV&gt;&lt;DIV =3D3D</DIV><DIV =3D</DIV><DIV=
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">style=3D3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-left: =
> 0px; ">style=3D3D3D"margin-top: 0px; margin-right: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-bottom: 0px; =3D3D</DIV><DIV =
> style=3D3D"margin-top: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">margin-left: =3D</DIV><DIV=
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">0px; "&gt;Dept. of Computer Science and =
> Engineering&lt;/DIV&gt;&lt;DIV =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">=3D3D</DIV><DIV style=3D3D"margin-top: 0px; =
> margin-right: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">margin-bottom: 0px; =
> margin-left: 0px; ">style=3D3D3D"margin-top: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-right: 0px; margin-bottom: 0px; =
> =3D3D</DIV><DIV style=3D3D"margin-top: =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">">margin-left: 0px; =
> "&gt;Ohio State University&lt;/DIV&gt;&lt;DIV =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">style=3D3D3D"margin-top: =
> =3D3D</DIV><DIV style=3D3D"margin-top: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-right: 0px; margin-bottom: 0px; margin-left: =
> 0px; ">0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; "&gt;OH =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">=3D3D</DIV><DIV style=3D3D"margin-top: 0px; =
> margin-right: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">margin-bottom: 0px; =
> margin-left: 0px; ">43210&lt;/DIV&gt;&lt;DIV =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">style=3D3D3D"margin-top: 0px; margin-right: 0px; =
> =3D3D</DIV><DIV =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">style=3D3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-left: 0px; =
> ">margin-bottom: 0px; margin-left: 0px; "&gt;Tel: =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">(614)292-8501&lt;/DIV&gt;&lt;DIV =
> =3D3D</DIV><DIV style=3D3D"margin-top: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-right: 0px; margin-bottom: 0px; margin-left: =
> 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">">style=3D3D3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">=3D3D</DIV><DIV style=3D3D"margin-top: 0px; =
> margin-right: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">margin-bottom: 0px; =
> margin-left: 0px; ">margin-left: 0px; min-height: =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">14px; =
> "&gt;&lt;BR&gt;&lt;/DIV&gt;&lt;DIV =
> style=3D3D3D"margin-top: =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">=3D3D</DIV><DIV style=3D3D"margin-top: 0px; margin-right: =
> 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-bottom: 0px; margin-left: =
> 0px; ">0px; margin-right: 0px; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">margin-bottom: 0px; margin-left: 0px; =3D3D</DIV><DIV =
> style=3D3D"margin-top: =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">">min-height: 14px; =
> "&gt;&lt;BR&gt;&lt;/DIV&gt;&lt;DIV =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">style=3D3D3D"margin-top: 0px; =3D3D</DIV><DIV=
> style=3D3D"margin-top: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">margin-right: =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">0px; margin-bottom: 0px; =
> margin-left: 0px; "&gt;On Wed, 21 =3D3D</DIV><DIV =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">style=3D3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">margin-left: 0px; ">Mar 2007, David Gunter =
> wrote:&lt;/DIV&gt;&lt;DIV =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">style=3D3D3D"margin-top: 0px; =3D3D</DIV><DIV =
> style=3D3D"margin-top: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">margin-right: =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">0px; margin-bottom: 0px; =
> margin-left: 0px; min-height: =3D3D</DIV><DIV =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">style=3D3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-left: =
> 0px; ">14px; "&gt;&lt;BR&gt;&lt;/DIV&gt; =
> &lt;BLOCKQUOTE =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; =
> ">type=3D3D3D"cite"&gt;&lt;DIV style=3D3D3D"margin-top: =
> =3D3D</DIV><DIV =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">style=3D3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-left: 0px; ">0px; =
> margin-right: 0px; margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">margin-left: 0px; "&gt;I have =3D3D</DIV><DIV =
> style=3D3D"margin-top: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">built =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">mvapich2 for an OFED-based IB Opteron =
> cluster.&lt;SPAN =3D3D</DIV><DIV =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">style=3D3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-left: =
> 0px; ">class=3D3D3D"Apple-converted-space"&gt;=3D3DA0 =3D</DIV><DIV=
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; =
> ">&lt;/SPAN&gt;When&lt;/DIV&gt;&lt;DIV =
> =3D3D</DIV><DIV style=3D3D"margin-top: =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">">style=3D3D3D"margin-top:=
> 0px; margin-right: 0px; margin-bottom: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">=3D3D</DIV><DIV style=3D3D"margin-top: 0px; =
> margin-right: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">margin-bottom: 0px; =
> margin-left: 0px; ">margin-left: 0px; "&gt;running =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">the Intel MPI Benchmarks (IMB3) I keep =
> =3D3D</DIV><DIV style=3D3D"margin-top: =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">seeing =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">the =
> following&lt;/DIV&gt;&lt;DIV style=3D3D3D"margin-top: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-right: =
> =3D3D</DIV><DIV style=3D3D"margin-top: 0px; margin-right: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-bottom: 0px; margin-left: =
> 0px; ">0px; margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">margin-left: 0px; "&gt;errors messages in many =
> =3D3D</DIV><DIV =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">style=3D3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-left: 0px; ">spots, =
> although the tests run to =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">completion:&lt;/DIV&gt;&lt;DIV =3D3D</DIV><DIV =
> style=3D3D"margin-top: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">">style=3D3D3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">=3D3D</DIV><DIV style=3D3D"margin-top: 0px; margin-right: =
> 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-bottom: 0px; margin-left: =
> 0px; ">margin-left: 0px; min-height: =3D</DIV><DIV style=3D"margin-top:=
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">14px; =
> "&gt;&lt;BR&gt;&lt;/DIV&gt;&lt;DIV =
> style=3D3D3D"margin-top: =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">=3D3D</DIV><DIV style=3D3D"margin-top: 0px; margin-right: =
> 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-bottom: 0px; margin-left: =
> 0px; ">0px; margin-right: 0px; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">margin-bottom: 0px; margin-left: 0px; "&gt;Internal =
> =3D3D</DIV><DIV =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">style=3D3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-left: 0px; ">error: =
> communicator is already on free =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">list&lt;/DIV&gt;&lt;DIV =
> style=3D3D3D"margin-top:=3D3D</DIV><DIV =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">style=3D3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-left: =
> 0px; "><SPAN =
> class=3D3D"Apple-converted-space">=3DA0</SPAN>0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-right: 0px; margin-bottom: 0px; margin-left: =
> 0px; =3D3D</DIV><DIV =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">style=3D3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-left: 0px; =
> ">min-height: 14px; =
> "&gt;&lt;BR&gt;&lt;/DIV&gt;&lt;DIV =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">style=3D3D3D"margin-top: 0px; =3D3D</DIV><DIV=
> style=3D3D"margin-top: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">margin-right: =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">0px; margin-bottom: 0px; =
> margin-left: 0px; "&gt;What is this =
> =3D3D</DIV><DIV=3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><SPAN =
> class=3D"Apple-converted-space">=A0</SPAN>style=3D3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">margin-left: 0px; ">referring to?&lt;/DIV&gt;&lt;DIV =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">style=3D3D3D"margin-top: 0px; =
> margin-right: 0px; =3D3D</DIV><DIV =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">style=3D3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-left: =
> 0px; ">margin-bottom: 0px; margin-left: 0px; min-height: =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">14px; =
> "&gt;&lt;BR&gt;&lt;/DIV&gt;&lt;DIV =
> =3D3D</DIV><DIV =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">style=3D3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-left: 0px; =
> ">style=3D3D3D"margin-top: 0px; margin-right: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-bottom: 0px; =3D3D</DIV><DIV =
> style=3D3D"margin-top: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">margin-left: =3D</DIV><DIV=
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">0px; "&gt;Thanks.&lt;/DIV&gt;&lt;DIV =
> style=3D3D3D"margin-top: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">=3D3D</DIV><DIV style=3D3D"margin-top: 0px; margin-right: =
> 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-bottom: 0px; margin-left: =
> 0px; ">margin-right: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">margin-bottom: 0px; margin-left: 0px; =3D3D</DIV><DIV =
> style=3D3D"margin-top: =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">">"&gt;--david&lt;/DIV&gt;&lt;DIV =
> style=3D3D3D"margin-top: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-right: =
> 0px; =3D3D</DIV><DIV style=3D3D"margin-top: 0px; margin-right: =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-bottom: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-left: 0px; min-height: 14px; =
> "&gt;&lt;BR&gt;&lt;/DIV&gt;&lt;DIV =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">=3D3D</DIV><DIV style=3D3D"margin-top: 0px; =
> margin-right: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">margin-bottom: 0px; =
> margin-left: 0px; ">style=3D3D3D"margin-top: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-right: 0px; margin-bottom: 0px; =
> =3D3D</DIV><DIV style=3D3D"margin-top: =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">">margin-left: 0px; =
> "&gt;--&lt;/DIV&gt;&lt;DIV style=3D3D3D"margin-top: =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">0px; margin-right: =
> =3D3D</DIV><DIV style=3D3D"margin-top: 0px; margin-right: =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">0px; margin-bottom: 0px; =
> margin-left: 0px; ">0px; margin-bottom: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-left: 0px; "&gt;David =
> Gunter&lt;/DIV&gt;&lt;DIV =3D3D</DIV><DIV =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">style=3D3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">margin-left: 0px; ">style=3D3D3D"margin-top: 0px; margin-right: =
> 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-bottom: 0px; =
> =3D3D</DIV><DIV style=3D3D"margin-top: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-right: 0px; margin-bottom: 0px; margin-left: =
> 0px; ">margin-left: =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">0px; =
> "&gt;HPC-4: HPC Environments: Parallel Tools =3D3D</DIV><DIV =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">style=3D3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">margin-left: 0px; ">Team&lt;/DIV&gt;&lt;DIV =
> style=3D3D3D"margin-top: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-right: =
> 0px; =3D3D</DIV><DIV style=3D3D"margin-top: 0px; margin-right: =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-bottom: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-left: 0px; "&gt;Los Alamos National =
> =3D3D</DIV><DIV =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">style=3D3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-left: 0px; =
> ">Laboratory&lt;/DIV&gt;&lt;DIV style=3D3D3D"margin-top: =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">0px; margin-right: 0px; =
> =3D3D</DIV><DIV style=3D3D"margin-top: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-right: 0px; margin-bottom: 0px; margin-left: =
> 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">">margin-bottom: 0px; =
> margin-left: 0px; min-height: 14px; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">"&gt;&lt;BR&gt;&lt;/DIV&gt;&lt;DIV =
> =3D3D</DIV><DIV style=3D3D"margin-top: =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">">style=3D3D3D"margin-top:=
> 0px; margin-right: 0px; margin-bottom: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">=3D3D</DIV><DIV style=3D3D"margin-top: 0px; =
> margin-right: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">margin-bottom: 0px; =
> margin-left: 0px; ">margin-left: 0px; min-height: =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">14px; =
> "&gt;&lt;BR&gt;&lt;/DIV&gt;&lt;DIV =
> style=3D3D3D"margin-top: =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">=3D3D</DIV><DIV style=3D3D"margin-top: 0px; margin-right: =
> 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-bottom: 0px; margin-left: =
> 0px; ">0px; margin-right: 0px; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">margin-bottom: 0px; margin-left: 0px; =3D3D</DIV><DIV =
> style=3D3D"margin-top: =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">">min-height: 14px; =
> "&gt;&lt;BR&gt;&lt;/DIV&gt; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">&lt;/BLOCKQUOTE&gt;&lt;DIV =
> style=3D3D3D"margin-top: =3D3D</DIV><DIV =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">style=3D3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-left: =
> 0px; ">0px; margin-right: 0px; margin-bottom: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-left: 0px; =3D3D</DIV><DIV =
> style=3D3D"margin-top: 0px; margin-right: =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">0px; margin-bottom: 0px; margin-left: 0px; =
> ">min-height: 14px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">"&gt;&lt;BR&gt;&lt;/DIV&gt; =3D3D</DIV><DIV=
> style=3D3D"margin-top: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; =
> ">">&lt;/BLOCKQUOTE&gt;&lt;/DIV&gt;&lt;BR&gt;&a=
> mp;lt;/DIV&gt;&lt;/BODY&gt;&lt;/H=3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">TML&gt;=3D3D</DIV><DIV =
> style=3D3D"margin-top: 0px; margin-right: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-bottom: 0px; margin-left: 0px; min-height: =
> 14px; "><BR></DIV><DIV =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">style=3D3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-left: =
> 0px; ">--Apple-Mail-1-849288998--</DIV><DIV =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">style=3D3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-left: =
> 0px; min-height: 14px; "><BR></DIV><DIV =
> style=3D3D"margin-top: =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; =
> ">">--=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=
> =3D3D=3D3D1750880957=3D3D=3D3D</DIV><DI=3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">V style=3D3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-left: =
> 0px; ">Content-Type: text/plain; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">charset=3D3D"us-ascii"</DIV><DIV style=3D3D"margin-top: 0px; =
> margin-right: =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">0px; margin-bottom: 0px; =
> margin-left: 0px; ">MIME-Version: 1.0</DIV><DIV =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">style=3D3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-left: =
> 0px; ">Content-Transfer-Encoding: 7bit</DIV><DIV =3D</DIV><DIV=
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">style=3D3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-left: =
> 0px; ">Content-Disposition: inline</DIV><DIV =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">style=3D3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-left: =
> 0px; min-height: 14px; "><BR></DIV><DIV =
> style=3D3D"margin-top: =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; =
> ">">_______________________________________________</DIV><DIV =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">style=3D3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">margin-left: 0px; ">mvapich-discuss mailing list</DIV><DIV =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">style=3D3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">margin-left: 0px; "><A =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">href=3D3D"<A =
> href=3D"mailto:mvapich-discuss at cse.ohio-state.edu">mailto:mvapich-discuss@=
> cse.ohio-state.edu</A>"><A =
> href=3D"mailto:mvapich-discuss at cse.ohi">mvapich-discuss at cse.ohi</A>=3D</DI=
> V><DIV style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">o-state.edu</A></DIV><DIV =
> style=3D3D"margin-top: 0px; margin-right: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-bottom: 0px; margin-left: 0px; "><A =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">href=3D3D"<A =
> href=3D"http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss">h=
> ttp://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss</A>">h=3D=
> </DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: =
> 0px; margin-left: 0px; =
> ">ttp://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss</A>=
> </DIV><D=3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">IV style=3D3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-left: 0px; min-height: 14px; =
> "><BR></DIV><DIV style=3D3D"margin-top: =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; =
> ">">--=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=
> =3D3D=3D3D1750880957=3D3D=3D3D--</DIV><=3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">DIV style=3D3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-left: =
> 0px; min-height: 14px; "><BR></DIV> =
> </BLOCKQUOTE><DIV =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">style=3D3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">margin-left: 0px; min-height: =
> 14px; "><BR></DIV> </BLOCKQUOTE><DIV =3D</DIV><DIV=
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">style=3D3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">margin-left: =
> 0px; min-height: 14px; "><BR></DIV><DIV =
> style=3D3D"margin-top: =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; =
> ">">_______________________________________________</DIV><DIV =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">style=3D3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">margin-left: 0px; ">mvapich-discuss mailing list</DIV><DIV =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">style=3D3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; =3D</DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">margin-left: 0px; "><A =3D</DIV><DIV style=3D"margin-top: 0px; =
> margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">href=3D3D"<A =
> href=3D"mailto:mvapich-discuss at cse.ohio-state.edu">mailto:mvapich-discuss@=
> cse.ohio-state.edu</A>"><A =
> href=3D"mailto:mvapich-discuss at cse.ohi">mvapich-discuss at cse.ohi</A>=3D</DI=
> V><DIV style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">o-state.edu</A></DIV><DIV =
> style=3D3D"margin-top: 0px; margin-right: 0px; =3D</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">margin-bottom: 0px; margin-left: 0px; "><A =
> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; ">href=3D3D"<A =
> href=3D"http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss">h=
> ttp://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss</A>">h=3D=
> </DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: =
> 0px; margin-left: 0px; =
> ">ttp://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss</A>=
> </DIV> =3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; =
> "></BLOCKQUOTE></DIV><BR></DIV></BODY></H=
> TML>=3D</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">--Apple-Mail-2-709635751--</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; min-height: 14px; "><BR></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">--=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D1220798408=3D=3D</DIV><DI=
> V style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">Content-Type: text/plain; =
> charset=3D"us-ascii"</DIV><DIV style=3D"margin-top: 0px; margin-right: =
> 0px; margin-bottom: 0px; margin-left: 0px; ">MIME-Version: 1.0</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">Content-Transfer-Encoding: 7bit</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">Content-Disposition: inline</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; min-height: 14px; "><BR></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">_______________________________________________</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; ">mvapich-discuss mailing list</DIV><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; "><A =
> href=3D"mailto:mvapich-discuss at cse.ohio-state.edu">mvapich-discuss at cse.ohi=
> o-state.edu</A></DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
> margin-bottom: 0px; margin-left: 0px; "><A =
> href=3D"http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss">h=
> ttp://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss</A></DIV><D=
> IV style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; min-height: 14px; "><BR></DIV><DIV style=3D"margin-top: =
> 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
> ">--=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D1220798408=3D=3D--</DIV><=
> DIV style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; min-height: 14px; "><BR></DIV> </BLOCKQUOTE><DIV =
> style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
> margin-left: 0px; min-height: 14px; "><BR></DIV> =
> </BLOCKQUOTE></DIV><BR></DIV></DIV></BODY></HTML>=
>
> --Apple-Mail-4-797936092--
>
> --===============1463864021==
> Content-Type: text/plain; charset="us-ascii"
> MIME-Version: 1.0
> Content-Transfer-Encoding: 7bit
> Content-Disposition: inline
>
> _______________________________________________
> mvapich-discuss mailing list
> mvapich-discuss at cse.ohio-state.edu
> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>
> --===============1463864021==--
>
More information about the mvapich-discuss
mailing list