[mvapich-discuss] (no subject)

Wed Feb 24 20:19:23 EST 2016

We had offline discussion with Nenad. We found that this issue goes away
when launching with mpirun_rsh. We will take a look at the issues with
Hydra.

Thanks,
Mingzhe

On Wed, Feb 24, 2016 at 1:06 PM, Nenad Vukicevic <nenad at intrepid.com> wrote:

> --===============2868158841024866563==
> Content-Type: multipart/alternative;
> boundary="089e0111ba8aa13d70052c87ece8"
>
> --089e0111ba8aa13d70052c87ece8
> Content-Type: text/plain; charset="UTF-8"
>
> I am using hydra with ssh, based on the logs.  While debugging I noticed
> that SLURM variables are not set, even though I have slurm on the system.
>
> Will send you example and some mpirun logs separately.
>
> On Wed, Feb 24, 2016 at 9:07 AM, Mingzhe Li <li.2192 at osu.edu> wrote:
>
> > Hi Nenad,
> >
> > Thanks for the note. Which launcher did you use? Is it possible to send
> us
> > a reproducer for this issue?
> >
> > Thanks,
> > Mingzhe
> >
> > On Wed, Feb 24, 2016 at 11:58 AM, Nenad Vukicevic <nenad at intrepid.com>
> > wrote:
> >
> >> X-MS-Exchange-Cros
> >> --===============3870460748841214172==
> >> Content-Type: multipart/alternative;
> >> boundary="001a11367c2ecaa3c8052c86fa95"
> >>
> >> --001a11367c2ecaa3c8052c86fa95
> >> Content-Type: text/plain; charset="UTF-8"
> >>
> >> I am running MVAPICH 2.2b on the Fedora FC23 IB cluster.  While running
> >> some accumulate tests I started getting segmentation faults in
> >> 'MPIDI_Fetch_and_op()' that are related to this line:
> >>
> >> 338         (win_ptr->create_flavor == MPI_WIN_FLAVOR_SHARED |
> >> 339         (win_ptr->shm_allocated == TRUE && orig_vc->node_id ==
> >> target_vc->node_id)))))
> >> 340     {
> >> 341         mpi_errno = MPIDI_CH3I_Shm_fop_op(origin_addr, result_addr,
> >> datatype,
> >> 342                                           target_rank, target_disp,
> >> op,
> >> win_ptr);
> >> 343         if (mpi_errno) MPIU_ERR_POP(mpi_errno);
> >> 344     }
> >>
> >> It turned out that node_id for orig_vc and target_vc are the same, even
> >> though these ranks are running on separate nodes.  I have 4 nodes and 5
> >> ranks, and rank 4 was instantiated on node 0, while the code above had
> >> node_id equal 3, which made it try to access shared space via shared
> >> memory.
> >>
> >> While further debugging the issue I noticed that thee are two places in
> >> the
> >> initialization code where hostname is being exchanged and 'node_id' set:
> >>
> >> mpid_vc.c:
> >> 1499 int MPIDI_Populate_vc_node_ids(MPIDI_PG_t *pg, int our_pg_rank)
> >>
> >> mpid_vc.c:
> >> 1363 int MPIDI_Get_local_host(MPIDI_PG_t *pg, int our_pg_rank)
> >>
> >> The first procedure above sets the node_id correctly, the second one
> does
> >> not.  If I comment out line 1444 in the same file, I am able to pass the
> >> test:
> >>
> >> 1444         // pg->vct[i].node_id = g_num_nodes - 1;
> >>
> >> Note that the second procedure is called only if "#if
> >> defined(CHANNEL_MRAIL) || defined(CHANNEL_PSM)" and I am now wondering
> if
> >> I
> >> have the right configuration.  While configuring mvapich I did not add
> any
> >> specific channel configuration, thinking that it will configure what is
> >> the
> >> best out of the box.
> >>
> >> I can provide some mpirun logs that show multiple hostname exchanges.
> >>
> >> Nenad
> >>
> >> --001a11367c2ecaa3c8052c86fa95
> >> Content-Type: text/html; charset="UTF-8"
> >> Content-Transfer-Encoding: quoted-printable
> >>
> >> <div dir=3D"ltr">I=C2=A0am running MVAPICH 2.2b on the Fedora FC23 IB
> >> clust=
> >> er.=C2=A0 While running some accumulate tests I started getting
> >> segmentatio=
> >> n faults in 'MPIDI_Fetch_and_op()' that are related to this
> >> line:<b=
> >> r><br>338 =C2=A0 =C2=A0 =C2=A0 =C2=A0 (win_ptr->create_flavor =3D=3D
> >> MPI=
> >> _WIN_FLAVOR_SHARED |<br>339 =C2=A0 =C2=A0 =C2=A0 =C2=A0
> >> (win_ptr->shm_al=
> >> located =3D=3D TRUE && orig_vc->node_id =3D=3D
> >> target_vc->nod=
> >> e_id)))))<br>340 =C2=A0 =C2=A0 {<br>341 =C2=A0 =C2=A0 =C2=A0 =C2=A0
> >> mpi_err=
> >> no =3D MPIDI_CH3I_Shm_fop_op(origin_addr, result_addr, datatype,<br>342
> >> =C2=
> >> =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0
> =C2=A0
> >> =
> >> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0
> >> targe=
> >> t_rank, target_disp, op, win_ptr);<br>343 =C2=A0 =C2=A0 =C2=A0 =C2=A0 if
> >> (m=
> >> pi_errno) MPIU_ERR_POP(mpi_errno);<br>344 =C2=A0 =C2=A0 }<br><br>It
> >> turned =
> >> out that node_id for orig_vc and target_vc are the same, even though
> >> these =
> >> ranks are running on separate nodes.=C2=A0 I have 4 nodes and 5 ranks,
> >> and =
> >> rank 4 was instantiated on node 0, while the code above had node_id
> equal
> >> 3=
> >> , which made it try to access shared space via shared
> >> memory.=C2=A0<div><br=
> >> ></div><div>While further debugging the issue I noticed that thee are
> two
> >> p=
> >> laces in the initialization code where hostname is being exchanged and
> >> &#39=
> >> ;node_id' set:<div><br></div><div>mpid_vc.c:</div><div>1499 int
> >> MPIDI_P=
> >> opulate_vc_node_ids(MPIDI_PG_t *pg, int
> >> our_pg_rank)<br><br>mpid_vc.c:</div=
> >> >
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> <div><span>1363 </span><span>int</span><span>
> >> MPIDI_Get_local_host(MPIDI_PG=
> >> _t *pg, </span><span>int</span><span>
> >> our_pg_rank)</span></div><div><br></d=
> >> iv><div>The first procedure above sets the node_id correctly, the second
> >> on=
> >> e does not.=C2=A0 If I comment out line 1444 in the same file, I am able
> >> to=
> >>  pass the test:</div><div><span><br>1444 </span><span>=C2=A0 =C2=A0
> >> =C2=A0 =
> >> =C2=A0 </span><span>//
> >> pg-></span><span>vct</span><span>[</span><span>i<=
> >> /span><span>].node_id =3D
> >> </span><span>g</span><span>_</span><span>num</spa=
> >> n><span>_nodes -
> >> 1;</span><br></div><div><span><br></span></div><div><span>=
> >> Note that the second procedure is called only if "</span>#if
> >> defined(C=
> >> HANNEL_MRAIL) || defined(CHANNEL_PSM)" and I am now wondering if I
> >> hav=
> >> e the right configuration.=C2=A0 While configuring mvapich I did not add
> >> an=
> >> y specific channel configuration, thinking that it will configure what
> is
> >> t=
> >> he best out of the box.</div><div><br></div><div>I can provide some
> >> mpirun =
> >> logs that show multiple hostname exchanges.=C2=A0</div>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> <div><br></div><div>Nenad<div>
> >> </div></div></div></div>
> >>
> >> --001a11367c2ecaa3c8052c86fa95--
> >>
> >> --===============3870460748841214172==
> >> Content-Type: text/plain; charset="us-ascii"
> >> MIME-Version: 1.0
> >> Content-Transfer-Encoding: 7bit
> >> Content-Disposition: inline
> >>
> >> _______________________________________________
> >> mvapich-discuss mailing list
> >> mvapich-discuss at cse.ohio-state.edu
> >> http://mailman.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
> >>
> >> --===============3870460748841214172==--
> >>
> >
> >
>
>
> --
> -- Nenad
>
> --089e0111ba8aa13d70052c87ece8
> Content-Type: text/html; charset="UTF-8"
> Content-Transfer-Encoding: quoted-printable
>
> <div dir=3D"ltr">I am using hydra with ssh, based on the logs.=C2=A0 While
> =
> debugging I noticed that SLURM variables are not set, even though I have
> sl=
> urm on the system.<div><br></div><div>Will send you example and some
> mpirun=
>  logs separately.</div></div><div class=3D"gmail_extra"><br><div
> class=3D"g=
> mail_quote">On Wed, Feb 24, 2016 at 9:07 AM, Mingzhe Li <span
> dir=3D"ltr">&=
> lt;<a href=3D"mailto:li.2192 at osu.edu" target=3D"_blank">li.2192 at osu.edu
> </a>=
> ></span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0
> 0=
>  0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr">Hi
> Ne=
> nad,<div><br></div><div>Thanks for the note. Which launcher did you use?
> Is=
>  it possible to send us a reproducer for this
> issue?</div><div><br></div><d=
> iv>Thanks,</div><div>Mingzhe</div></div><div class=3D"HOEnZb"><div
> class=3D=
> "h5"><div class=3D"gmail_extra"><br><div class=3D"gmail_quote">On Wed, Feb
> =
> 24, 2016 at 11:58 AM, Nenad Vukicevic <span dir=3D"ltr"><<a
> href=3D"mail=
> to:nenad at intrepid.com" target=3D"_blank">nenad at intrepid.com</a>></span>
> =
> wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0
> .8ex;bord=
> er-left:1px #ccc solid;padding-left:1ex">X-MS-Exchange-Cros<br>
>
> --=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D3870460748841214172=3D=3D<br=
> >
> Content-Type: multipart/alternative;
> boundary=3D"001a11367c2ecaa3c8052=
> c86fa95"<br>
> <br>
> --001a11367c2ecaa3c8052c86fa95<br>
> Content-Type: text/plain; charset=3D"UTF-8"<br>
> <br>
> I am running MVAPICH 2.2b on the Fedora FC23 IB cluster.=C2=A0 While
> runnin=
> g<br>
> some accumulate tests I started getting segmentation faults in<br>
> 'MPIDI_Fetch_and_op()' that are related to this line:<br>
> <br>
> 338=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(win_ptr->create_flavor =3D=3D
> MPI_=
> WIN_FLAVOR_SHARED |<br>
> 339=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(win_ptr->shm_allocated =3D=3D
> TRUE=
>  && orig_vc->node_id =3D=3D<br>
> target_vc->node_id)))))<br>
> 340=C2=A0 =C2=A0 =C2=A0{<br>
> 341=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mpi_errno =3D
> MPIDI_CH3I_Shm_fop_op(or=
> igin_addr, result_addr,<br>
> datatype,<br>
> 342=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
> =A0 =C2=A0target_rank, target_disp, op,<br>
> win_ptr);<br>
> 343=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0if (mpi_errno)
> MPIU_ERR_POP(mpi_errno)=
> ;<br>
> 344=C2=A0 =C2=A0 =C2=A0}<br>
> <br>
> It turned out that node_id for orig_vc and target_vc are the same, even<br>
> though these ranks are running on separate nodes.=C2=A0 I have 4 nodes and
> =
> 5<br>
> ranks, and rank 4 was instantiated on node 0, while the code above had<br>
> node_id equal 3, which made it try to access shared space via shared<br>
> memory.<br>
> <br>
> While further debugging the issue I noticed that thee are two places in
> the=
> <br>
> initialization code where hostname is being exchanged and
> 'node_id'=
>  set:<br>
> <br>
> mpid_vc.c:<br>
> 1499 int MPIDI_Populate_vc_node_ids(MPIDI_PG_t *pg, int our_pg_rank)<br>
> <br>
> mpid_vc.c:<br>
> 1363 int MPIDI_Get_local_host(MPIDI_PG_t *pg, int our_pg_rank)<br>
> <br>
> The first procedure above sets the node_id correctly, the second one
> does<b=
> r>
> not.=C2=A0 If I comment out line 1444 in the same file, I am able to pass
> t=
> he<br>
> test:<br>
> <br>
> 1444=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0// pg->vct[i].node_id =3D
> g_num_no=
> des - 1;<br>
> <br>
> Note that the second procedure is called only if "#if<br>
> defined(CHANNEL_MRAIL) || defined(CHANNEL_PSM)" and I am now
> wondering=
>  if I<br>
> have the right configuration.=C2=A0 While configuring mvapich I did not
> add=
>  any<br>
> specific channel configuration, thinking that it will configure what is
> the=
> <br>
> best out of the box.<br>
> <br>
> I can provide some mpirun logs that show multiple hostname exchanges.<br>
> <br>
> Nenad<br>
> <br>
> --001a11367c2ecaa3c8052c86fa95<br>
> Content-Type: text/html; charset=3D"UTF-8"<br>
> Content-Transfer-Encoding: quoted-printable<br>
> <br>
> <div dir=3D3D"ltr">I=3DC2=3DA0am running MVAPICH 2.2b on
> th=
> e Fedora FC23 IB clust=3D<br>
> er.=3DC2=3DA0 While running some accumulate tests I started getting
> segment=
> atio=3D<br>
> n faults in &#39;MPIDI_Fetch_and_op()&#39; that are related to
> this=
>  line:<b=3D<br>
> r><br>338 =3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0
> (win_ptr-&am=
> p;gt;create_flavor =3D3D=3D3D MPI=3D<br>
> _WIN_FLAVOR_SHARED |<br>339 =3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0
> =3DC2=3D=
> A0 (win_ptr-&gt;shm_al=3D<br>
> located =3D3D=3D3D TRUE &amp;&amp; orig_vc-&gt;node_id
> =3D3D=3D=
> 3D target_vc-&gt;nod=3D<br>
> e_id)))))<br>340 =3DC2=3DA0 =3DC2=3DA0 {<br>341 =3DC2=3DA0
> =3DC=
> 2=3DA0 =3DC2=3DA0 =3DC2=3DA0 mpi_err=3D<br>
> no =3D3D MPIDI_CH3I_Shm_fop_op(origin_addr, result_addr,
> datatype,<br&gt=
> ;342 =3DC2=3D<br>
> =3DA0 =3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0
> =3D=
> C2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =3D<br>
> =3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0
> =3DC2=3DA=
> 0 =3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0 targe=3D<br>
> t_rank, target_disp, op, win_ptr);<br>343 =3DC2=3DA0 =3DC2=3DA0
> =3DC2=
> =3DA0 =3DC2=3DA0 if (m=3D<br>
> pi_errno) MPIU_ERR_POP(mpi_errno);<br>344 =3DC2=3DA0 =3DC2=3DA0
> }<=
> br><br>It turned =3D<br>
> out that node_id for orig_vc and target_vc are the same, even though these
> =
> =3D<br>
> ranks are running on separate nodes.=3DC2=3DA0 I have 4 nodes and 5 ranks,
> =
> and =3D<br>
> rank 4 was instantiated on node 0, while the code above had node_id equal
> 3=
> =3D<br>
> , which made it try to access shared space via shared
> memory.=3DC2=3DA0<=
> div><br=3D<br>
> ></div><div>While further debugging the issue I noticed
> that=
>  thee are two p=3D<br>
> laces in the initialization code where hostname is being exchanged and
> &amp=
> ;#39=3D<br>
> ;node_id&#39;
> set:<div><br></div><div>mpid_vc.c=
> :</div><div>1499 int MPIDI_P=3D<br>
> opulate_vc_node_ids(MPIDI_PG_t *pg, int
> our_pg_rank)<br><br>mpi=
> d_vc.c:</div=3D<br>
> ><br>
> <br>
> <br>
> <br>
> <br>
> <br>
> <br>
> <br>
> <div><span>1363
> </span><span>int</span><sp=
> an> MPIDI_Get_local_host(MPIDI_PG=3D<br>
> _t *pg, </span><span>int</span><span>
> our_pg_rank)&=
> lt;/span></div><div><br></d=3D<br>
> iv><div>The first procedure above sets the node_id correctly, the
> =
> second on=3D<br>
> e does not.=3DC2=3DA0 If I comment out line 1444 in the same file, I am
> abl=
> e to=3D<br>
> =C2=A0pass the test:</div><div><span><br>1444
> </=
> span><span>=3DC2=3DA0 =3DC2=3DA0 =3DC2=3DA0 =3D<br>
> =3DC2=3DA0 </span><span>//
> pg-&gt;</span><span>=
> vct</span><span>[</span><span>i<=3D<br>
> /span><span>].node_id =3D3D
> </span><span>g</span&gt=
> ;<span>_</span><span>num</spa=3D<br>
> n><span>_nodes -
> 1;</span><br></div><div>&=
>
> lt;span><br></span></div><div><span>=3D<br=
> >
> Note that the second procedure is called only if
> &quot;</span>#if=
>  defined(C=3D<br>
> HANNEL_MRAIL) || defined(CHANNEL_PSM)&quot; and I am now wondering if
> I=
>  hav=3D<br>
> e the right configuration.=3DC2=3DA0 While configuring mvapich I did not
> ad=
> d an=3D<br>
> y specific channel configuration, thinking that it will configure what is
> t=
> =3D<br>
> he best out of the
> box.</div><div><br></div><div=
> >I can provide some mpirun =3D<br>
> logs that show multiple hostname exchanges.=3DC2=3DA0</div><br>
> <br>
> <br>
> <br>
> <br>
> <br>
> <br>
> <br>
> <div><br></div><div>Nenad<div><br>
> </div></div></div></div><br>
> <br>
> --001a11367c2ecaa3c8052c86fa95--<br>
> <br>
>
> --=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D3870460748841214172=3D=3D<br=
> >
> Content-Type: text/plain; charset=3D"us-ascii"<br>
> MIME-Version: 1.0<br>
> Content-Transfer-Encoding: 7bit<br>
> Content-Disposition: inline<br>
> <br>
> _______________________________________________<br>
> mvapich-discuss mailing list<br>
> <a href=3D"mailto:mvapich-discuss at cse.ohio-state.edu"
> target=3D"_blank">mva=
> pich-discuss at cse.ohio-state.edu</a><br>
> <a href=3D"
> http://mailman.cse.ohio-state.edu/mailman/listinfo/mvapich-discu=
> ss" rel=3D"noreferrer" target=3D"_blank">
> http://mailman.cse.ohio-state.edu/=
> mailman/listinfo/mvapich-discuss
> <http://mailman.cse.ohio-state.edu/=mailman/listinfo/mvapich-discuss>
> </a><br>
> <br>
>
> --=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D3870460748841214172=3D=3D--<=
> br>
> </blockquote></div><br></div>
> </div></div></blockquote></div><br><br clear=3D"all"><div><br></div>--
> <br>=
> <div class=3D"gmail_signature"><div dir=3D"ltr"><div><div
> dir=3D"ltr"><div>=
> <div dir=3D"ltr">-- Nenad</div></div></div></div></div></div>
> </div>
>
> --089e0111ba8aa13d70052c87ece8--
>
> --===============2868158841024866563==
> Content-Type: text/plain; charset="us-ascii"
> MIME-Version: 1.0
> Content-Transfer-Encoding: 7bit
> Content-Disposition: inline
>
> _______________________________________________
> mvapich-discuss mailing list
> mvapich-discuss at cse.ohio-state.edu
> http://mailman.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>
> --===============2868158841024866563==--
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20160224/c538286e/attachment-0001.html>