[mvapich-discuss] mvapich2 mpirun_rsh bug

[Jaidev Sridhar]

Hi Tatek,

On further investigation, we saw that the path handling was different
between MVAPICH and MVAPICH2 as you pointed out. Thanks for this report.
We've updated mpirun_rsh on MVAPICH2 to mimic the behavior on MVAPICH.

The update will be available in the upcoming release of MVAPICH2 1.2. It is
also available on the svn at:
    https://mvapich.cse.ohio-state.edu/svn/mpi/mvapich2/trunk

Regards,
Jaidev

On Fri, Oct 10, 2008 at 11:12:15PM +0000, Tatek wrote:
 > 
 >    Hello All,
 > 
 >    There seems to be a bug with mvapich2 and mpirun_rsh command.  Once I
 >    install mvapich2, to execute mpiun_rsh command, I have to give the
 >    full path of the mpirun_rsh or it will not execute.  For example, I
 >    was trying to run mpiBench and after setting the PATH for the bin and
 >    LD_LIBRARY_PATH, I would get the following
 > 
 >    [root at t2k-0007 mpibench-0.1-3]# mpirun_rsh -np 2 -hostfile hosts5
 >    ./mpiBench -i 100 Allreduce
 >    /usr/bin/env: mpispawn: No such file or directory
 >    Child exited abnormally!
 >    cleanupKilling remote processes.../usr/bin/env: mpispawn: No such file
 >    or directory DONE
 > 
 >    note that "which mpirun_rsh" shows the correct PATH so I had the PATH
 >    set correctly
 > 
 >    [root at t2k-0007 mpibench-0.1-3]# which mpirun_rsh
 >    /global/usr/mpi/mvapich2-1.2/pgi/bin/mpirun_rsh
 > 
 >    If I give the full PATH then no problem.
 > 
 >    [root at t2k-0007 mpibench-0.1-3]#
 >    /global/usr/mpi/mvapich2-1.2/pgi/bin/mpirun_rsh -np 2 -hostfile hosts5
 >    ./mpiBench -i 100 Allreduce START mpiBench 0 : ddn-0268
 >    1 : ddn-0302
 >    Allreduce Bytes: 0 Iters: 100 Avg: 2.9612 Min: 2.9612 Max: 2.9612 Dim:
 >    0 Ranks: 2
 >    Allreduce Bytes: 8 Iters: 100 Avg: 6.4301 Min: 6.4087 Max: 6.4516 Dim:
 >    0 Ranks: 2
 > 
 >    After some investigation, one of my colleagues figured out the issue
 >    was that MVAPICH2 mpirun_rsh command gets the directory name from the
 >    name of the executable (passed to the program as the first argument)
 >    and starts mpispawn using this path. If the full path name is not
 >    supplied, the mpispawn program is started without a path and can't be
 >    found unless it happens to be in the default login path of the
 >    user. We think this is a bug in MVAPICH2's mpirun_rsh. It is
 >    reasonable to expect that if the MVAPICH2 binary directory is the
 >    PATH, issuing an mpirun_rsh command should work.
 > 
 >    We have a workaround for now by either using the full PATH or using
 >    wrapper scripts but ideally, this should work same as it did with
 >    mvapich1.
 > 
 >    Best Regards,
 > 
 > 
 >    Tatek Getachew
 > 
 >    Solutions Engineer, HPC Solutions Engineering
 > 
 >    Appro International, Inc.
 > 
 >    446 South Abbott Ave
 > 
 >    Milpitas, CA 95035
 > 
 >    408-941-8100 x504 (office)
 > 
 >    408-640-9050 (cell)
 > 
 >    [1]tgetachew at appro.com
 > 
 > 
 > 
 > 
 >      _________________________________________________________________
 > 
 >    Get more out of the Web. Learn 10 hidden secrets of Windows Live.
 >    [2]Learn Now
 > 
 > References
 > 
 >    1. mailto:tgetachew at appro.com
 >    2. http://windowslive.com/connect/post/jamiethomson.spaces.live.com-Blog-cns!550F681DAD532637!5295.entry?ocid=TXT_TAGLM_WL_getmore_092008

 > _______________________________________________
 > mvapich-discuss mailing list
 > mvapich-discuss at cse.ohio-state.edu
 > http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss