[mvapich-discuss] delay in spawning

Jaidev Sridhar sridharj at cse.ohio-state.edu
Tue Feb 17 17:39:00 EST 2009 

Hossein,

Can you forward the exact command line you used to invoke mpirun_rsh? If
anything else is printed before or after the error message, please
forward them too.

Also, from the same environment and working directory can you send us
the output of the following commands? 
  $ which mpirun_rsh
  $ which mpispawn
  $ which env

I suspect a mixup between different versions of mpirun_rsh since the -v
version doesn't work for you. You can also try a running a simple MPI
program such as osu_latency between two nodes to see if your environment
is otherwise fine.

Note, if you use the -debug option, xterm needs to be installed / 
linked at /usr/X11R6/bin/xterm.

-Jaidev

On Tue, 2009-02-17 at 15:56 -0600, Hossein Pourreza wrote:
> I checked the FAQ and there is a *similar* problem. In my case the error message
> does not say anything about mpispawn it reads:
> /usr/bin/env: No such file or directory
> 
> I, however, included the full path to mpirun_sh and ran my program but there was
> no output (not even error message). I ran top command on the other window and
> it was showing many (maybe 10) ssh commands running.
> 
> I checked the debug output and it seems that execv is missing on my computer and
> it may cause the problem. I used your google-enabled search tool but there is
> nothing about "execv".
> 
> Also, as I mentioned earlier, running mpirun_rsh with -v option gives Unknown
> option error. Again, I searched your web site and there is no posting regarding
> this issue.
> 
> Thanks
> 
> On Tue, Feb 17, 2009 at 02:21:45PM -0500, Dhabaleswar Panda wrote:
> > Hossein,
> > 
> > This error and its solution are indicated in the FAQ page of the user
> > guide at the following URL:
> > 
> > http://mvapich.cse.ohio-state.edu/support/user_guide_mvapich2-1.2.html#x1-520009.2.3
> > 
> > Let us know if this solves your problem.
> > 
> > FYI, MVAPICH web site (including user guide and postings on
> > mvapich-discuss) are enabled by Google Search. If you encounter any
> > problem, please do a search first and you will get an answer quickly if
> > this has been reported/resolved earlier.
> > 
> > DK
> > 
> > On Tue, 17 Feb 2009, Hossein Pourreza wrote:
> > 
> > > Thanks for the reply.
> > >
> > > Situation got worse :)
> > >
> > > I am getting the following error:
> > >
> > > /usr/bin/env: No such file or directory
> > >
> > > When I ran the mpirun_rsh with -v option it gave me:
> > >
> > > Unknown option
> > >
> > > and finally with -debug option I am getting following output:
> > >
> > > $mpirun_rsh -debug -np 32 -hostfile hosts ./bin/ep.B.32
> > >
> > > execv: No such file or directory
> > > /usr/X11R6/bin/xterm -e /usr/bin/ssh -q power02 cd
> > > /stage/Benchmarks/NPB2.4/NPB2.4-MPI; /usr/bin/env
> > > LD_LIBRARY_PATH=/usr/mvapich/lib/shared:/stage/mvapich2.1/lib:
> > > MPISPAWN_MPIRUN_MPD=0 MPISPAWN_MPIRUN_HOST=power01 MPISPAWN_CHECKIN_PORT=33209
> > > MPISPAWN_MPIRUN_PORT=33209 MPISPAWN_GLOBAL_NPROCS=32 MPISPAWN_MPIRUN_ID=11731
> > > MPISPAWN_ARGC=2 MPISPAWN_ARGV_0=/usr/bin/gdb MPISPAWN_ARGV_1=./bin/ep.B.32
> > > MPISPAWN_GENERIC_ENV_COUNT=0  MPISPAWN_ID=0 MPISPAWN_LOCAL_NPROCS=4
> > > MPISPAWN_WORKING_DIR=/stage/Benchmarks/NPB2.4/NPB2.4-MPI
> > > MPISPAWN_MPIRUN_RANK_0=0 MPISPAWN_VIADEV_DEFAULT_PORT_0=-1
> > > MPISPAWN_MPIRUN_RANK_1=1 MPISPAWN_VIADEV_DEFAULT_PORT_1=-1
> > > MPISPAWN_MPIRUN_RANK_2=2 MPISPAWN_VIADEV_DEFAULT_PORT_2=-1
> > > MPISPAWN_MPIRUN_RANK_3=3 MPISPAWN_VIADEV_DEFAULT_PORT_3=-1  mpispawn execv: No
> > > such file or directory
> > > /usr/X11R6/bin/xterm -e /usr/bin/ssh -q power03 cd
> > > /stage/Benchmarks/NPB2.4/NPB2.4-MPI; /usr/bin/env
> > > LD_LIBRARY_PATH=/usr/mvapich/lib/shared:/stage/mvapich2.1/lib:
> > > MPISPAWN_MPIRUN_MPD=0 MPISPAWN_MPIRUN_HOST=power01 MPISPAWN_CHECKIN_PORT=33209
> > > MPISPAWN_MPIRUN_PORT=33209 MPISPAWN_GLOBAL_NPROCS=32 MPISPAWN_MPIRUN_ID=11731
> > > MPISPAWN_ARGC=2 MPISPAWN_ARGV_0=/usr/bin/gdb MPISPAWN_ARGV_1=./bin/ep.B.32
> > > MPISPAWN_GENERIC_ENV_COUNT=0  MPISPAWN_ID=1 MPISPAWN_LOCAL_NPROCS=4
> > > MPISPAWN_WORKING_DIR=/stage/Benchmarks/NPB2.4/NPB2.4-MPI
> > > MPISPAWN_MPIRUN_RANK_0=4 MPISPAWN_VIADEV_DEFAULT_PORT_0=-1
> > > MPISPAWN_MPIRUN_RANK_1=5 MPISPAWN_VIADEV_DEFAULT_PORT_1=-1
> > > MPISPAWN_MPIRUN_RANK_2=6 MPISPAWN_VIADEV_DEFAULT_PORT_2=-1
> > > MPISPAWN_MPIRUN_RANK_3=7 MPISPAWN_VIADEV_DEFAULT_PORT_3=-1  mpispawn execv: No
> > > such file or directory
> > > /usr/X11R6/bin/xterm -e /usr/bin/ssh -q power04 cd
> > > /stage/Benchmarks/NPB2.4/NPB2.4-MPI; /usr/bin/env
> > > LD_LIBRARY_PATH=/usr/mvapich/lib/shared:/stage/mvapich2.1/lib:
> > > MPISPAWN_MPIRUN_MPD=0 MPISPAWN_MPIRUN_HOST=power01 MPISPAWN_CHECKIN_PORT=33209
> > > MPISPAWN_MPIRUN_PORT=33209 MPISPAWN_GLOBAL_NPROCS=32 MPISPAWN_MPIRUN_ID=11731
> > > MPISPAWN_ARGC=2 MPISPAWN_ARGV_0=/usr/bin/gdb MPISPAWN_ARGV_1=./bin/ep.B.32
> > > MPISPAWN_GENERIC_ENV_COUNT=0  MPISPAWN_ID=2 MPISPAWN_LOCAL_NPROCS=4
> > > MPISPAWN_WORKING_DIR=/stage/Benchmarks/NPB2.4/NPB2.4-MPI
> > > MPISPAWN_MPIRUN_RANK_0=8 MPISPAWN_VIADEV_DEFAULT_PORT_0=-1
> > > MPISPAWN_MPIRUN_RANK_1=9 MPISPAWN_VIADEV_DEFAULT_PORT_1=-1
> > > MPISPAWN_MPIRUN_RANK_2=10 MPISPAWN_VIADEV_DEFAULT_PORT_2=-1
> > > MPISPAWN_MPIRUN_RANK_3=11 MPISPAWN_VIADEV_DEFAULT_PORT_3=-1  mpispawn execv: No
> > > such file or directory
> > > /usr/X11R6/bin/xterm -e /usr/bin/ssh -q power06 cd
> > > /stage/Benchmarks/NPB2.4/NPB2.4-MPI; /usr/bin/env
> > > LD_LIBRARY_PATH=/usr/mvapich/lib/shared:/stage/mvapich2.1/lib:
> > > MPISPAWN_MPIRUN_MPD=0 MPISPAWN_MPIRUN_HOST=power01 MPISPAWN_CHECKIN_PORT=33209
> > > MPISPAWN_MPIRUN_PORT=33209 MPISPAWN_GLOBAL_NPROCS=32 MPISPAWN_MPIRUN_ID=11731
> > > MPISPAWN_ARGC=2 MPISPAWN_ARGV_0=/usr/bin/gdb MPISPAWN_ARGV_1=./bin/ep.B.32
> > > MPISPAWN_GENERIC_ENV_COUNT=0  MPISPAWN_ID=4 MPISPAWN_LOCAL_NPROCS=4
> > > MPISPAWN_WORKING_DIR=/stage/Benchmarks/NPB2.4/NPB2.4-MPI
> > > MPISPAWN_MPIRUN_RANK_0=16 MPISPAWN_VIADEV_DEFAULT_PORT_0=-1
> > > MPISPAWN_MPIRUN_RANK_1=17 MPISPAWN_VIADEV_DEFAULT_PORT_1=-1
> > > MPISPAWN_MPIRUN_RANK_2=18 MPISPAWN_VIADEV_DEFAULT_PORT_2=-1
> > > MPISPAWN_MPIRUN_RANK_3=19 MPISPAWN_VIADEV_DEFAULT_PORT_3=-1  mpispawn execv: No
> > > such file or directory
> > > /usr/X11R6/bin/xterm -e /usr/bin/ssh -q power05 cd
> > > /stage/Benchmarks/NPB2.4/NPB2.4-MPI; /usr/bin/env
> > > LD_LIBRARY_PATH=/usr/mvapich/lib/shared:/stage/mvapich2.1/lib:
> > > MPISPAWN_MPIRUN_MPD=0 MPISPAWN_MPIRUN_HOST=power01 MPISPAWN_CHECKIN_PORT=33209
> > > MPISPAWN_MPIRUN_PORT=33209 MPISPAWN_GLOBAL_NPROCS=32 MPISPAWN_MPIRUN_ID=11731
> > > MPISPAWN_ARGC=2 MPISPAWN_ARGV_0=/usr/bin/gdb MPISPAWN_ARGV_1=./bin/ep.B.32
> > > MPISPAWN_GENERIC_ENV_COUNT=0  MPISPAWN_ID=3 MPISPAWN_LOCAL_NPROCS=4
> > > MPISPAWN_WORKING_DIR=/stage/Benchmarks/NPB2.4/NPB2.4-MPI
> > > MPISPAWN_MPIRUN_RANK_0=12 MPISPAWN_VIADEV_DEFAULT_PORT_0=-1
> > > MPISPAWN_MPIRUN_RANK_1=13 MPISPAWN_VIADEV_DEFAULT_PORT_1=-1
> > > MPISPAWN_MPIRUN_RANK_2=14 MPISPAWN_VIADEV_DEFAULT_PORT_2=-1
> > > MPISPAWN_MPIRUN_RANK_3=15 MPISPAWN_VIADEV_DEFAULT_PORT_3=-1  mpispawn execv: No
> > > such file or directory
> > > /usr/X11R6/bin/xterm -e /usr/bin/ssh -q power07 cd
> > > /stage/Benchmarks/NPB2.4/NPB2.4-MPI; /usr/bin/env
> > > LD_LIBRARY_PATH=/usr/mvapich/lib/shared:/stage/mvapich2.1/lib:
> > > MPISPAWN_MPIRUN_MPD=0 MPISPAWN_MPIRUN_HOST=power01 MPISPAWN_CHECKIN_PORT=33209
> > > MPISPAWN_MPIRUN_PORT=33209 MPISPAWN_GLOBAL_NPROCS=32 MPISPAWN_MPIRUN_ID=11731
> > > MPISPAWN_ARGC=2 MPISPAWN_ARGV_0=/usr/bin/gdb MPISPAWN_ARGV_1=./bin/ep.B.32
> > > MPISPAWN_GENERIC_ENV_COUNT=0  MPISPAWN_ID=5 MPISPAWN_LOCAL_NPROCS=4
> > > MPISPAWN_WORKING_DIR=/stage/Benchmarks/NPB2.4/NPB2.4-MPI
> > > MPISPAWN_MPIRUN_RANK_0=20 MPISPAWN_VIADEV_DEFAULT_PORT_0=-1
> > > MPISPAWN_MPIRUN_RANK_1=21 MPISPAWN_VIADEV_DEFAULT_PORT_1=-1
> > > MPISPAWN_MPIRUN_RANK_2=22 MPISPAWN_VIADEV_DEFAULT_PORT_2=-1
> > > MPISPAWN_MPIRUN_RANK_3=23 MPISPAWN_VIADEV_DEFAULT_PORT_3=-1  mpispawn execv: No
> > > such file or directory
> > > /usr/X11R6/bin/xterm -e /usr/bin/ssh -q power08 cd
> > > /stage/Benchmarks/NPB2.4/NPB2.4-MPI; /usr/bin/env
> > > LD_LIBRARY_PATH=/usr/mvapich/lib/shared:/stage/mvapich2.1/lib:
> > > MPISPAWN_MPIRUN_MPD=0 MPISPAWN_MPIRUN_HOST=power01 MPISPAWN_CHECKIN_PORT=33209
> > > MPISPAWN_MPIRUN_PORT=33209 MPISPAWN_GLOBAL_NPROCS=32 MPISPAWN_MPIRUN_ID=11731
> > > MPISPAWN_ARGC=2 MPISPAWN_ARGV_0=/usr/bin/gdb MPISPAWN_ARGV_1=./bin/ep.B.32
> > > MPISPAWN_GENERIC_ENV_COUNT=0  MPISPAWN_ID=6 MPISPAWN_LOCAL_NPROCS=4
> > > MPISPAWN_WORKING_DIR=/stage/Benchmarks/NPB2.4/NPB2.4-MPI
> > > MPISPAWN_MPIRUN_RANK_0=24 MPISPAWN_VIADEV_DEFAULT_PORT_0=-1
> > > MPISPAWN_MPIRUN_RANK_1=25 MPISPAWN_VIADEV_DEFAULT_PORT_1=-1
> > > MPISPAWN_MPIRUN_RANK_2=26 MPISPAWN_VIADEV_DEFAULT_PORT_2=-1
> > > MPISPAWN_MPIRUN_RANK_3=27 MPISPAWN_VIADEV_DEFAULT_PORT_3=-1  mpispawn execv: No
> > > such file or directory
> > > /usr/X11R6/bin/xterm -e /usr/bin/ssh -q power09 cd
> > > /stage/Benchmarks/NPB2.4/NPB2.4-MPI; /usr/bin/env
> > > LD_LIBRARY_PATH=/usr/mvapich/lib/shared:/stage/mvapich2.1/lib:
> > > MPISPAWN_MPIRUN_MPD=0 MPISPAWN_MPIRUN_HOST=power01 MPISPAWN_CHECKIN_PORT=33209
> > > MPISPAWN_MPIRUN_PORT=33209 MPISPAWN_GLOBAL_NPROCS=32 MPISPAWN_MPIRUN_ID=11731
> > > MPISPAWN_ARGC=2 MPISPAWN_ARGV_0=/usr/bin/gdb MPISPAWN_ARGV_1=./bin/ep.B.32
> > > MPISPAWN_GENERIC_ENV_COUNT=0  MPISPAWN_ID=7 MPISPAWN_LOCAL_NPROCS=4
> > > MPISPAWN_WORKING_DIR=/stage/Benchmarks/NPB2.4/NPB2.4-MPI
> > > MPISPAWN_MPIRUN_RANK_0=28 MPISPAWN_VIADEV_DEFAULT_PORT_0=-1
> > > MPISPAWN_MPIRUN_RANK_1=29 MPISPAWN_VIADEV_DEFAULT_PORT_1=-1
> > > MPISPAWN_MPIRUN_RANK_2=30 MPISPAWN_VIADEV_DEFAULT_PORT_2=-1
> > > MPISPAWN_MPIRUN_RANK_3=31 MPISPAWN_VIADEV_DEFAULT_PORT_3=-1  mpispawn
> > > Child exited abnormally!
> > > cleanupKilling remote processes...DONE
> > >
> > > On Tue, Feb 17, 2009 at 12:38:34PM -0500, Dhabaleswar Panda wrote:
> > > > You are using the old MPD-based start-up. Starting with MVAPICH2 1.2, a
> > > > new scalable and robust mpirun_rsh framework (non-MPD-based) job launching
> > > > mechanism has been added. This framework reduces job start-up time
> > > > considerably and is also scalable to multi-thousand core clusters. Please
> > > > use this and you should see considerable improvement in your job launching
> > > > framework. Please refer to the MVAPICH2 user guide from the following URL
> > > > for details on using the mpirun_rsh framework:
> > > >
> > > > http://mvapich.cse.ohio-state.edu/support/user_guide_mvapich2-1.2.html
> > > >
> > > > If the problem persists with the new mpirun_rsh framework, let us know.
> > > >
> > > > DK
> > > >
> > > > On Tue, 17 Feb 2009, Hossein Pourreza wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I used to experience with a very long delay before execution of my tasks with
> > > > > MVAPICH-0.9.8 and reported that problem. I was hopping to see a fix in the new
> > > > > version but I still have the same problem with MVAPICH-1.2. The configuration of
> > > > > my machine is: (I use SDR infiniband on PCI_X bus)
> > > > >
> > > > >
> > > > > uname -v = Generic_137112-07
> > > > >
> > > > > /usr/bin/uname -p = i386
> > > > > /bin/uname -X     = System = SunOS
> > > > > Node = power
> > > > > Release = 5.10
> > > > > KernelID = Generic_137112-07
> > > > > Machine = i86pc
> > > > > BusType = <unknown>
> > > > > Serial = <unknown>
> > > > > Users = <unknown>
> > > > > OEM# = 0
> > > > > Origin# = 1
> > > > > NumCPU = 4
> > > > >
> > > > > /bin/arch              = i86pc
> > > > > /usr/bin/arch -k       = i86pc
> > > > > /usr/convex/getsysinfo = unknown
> > > > > hostinfo               = unknown
> > > > > /bin/machine           = unknown
> > > > >
> > > > > If I try to run my program with processes more than the number of cores of one
> > > > > node, I experience a very long delay before my task gets started. This delay
> > > > > increases by increasing number of processes. I run my tasks using:
> > > > >
> > > > > mpiexec -np xx ./myprog
> > > > >
> > > > > I am using mpd as processes placement daemon.
> > > > >
> > > > > Any help will be greatly appreciated.
> > > > > _______________________________________________
> > > > > mvapich-discuss mailing list
> > > > > mvapich-discuss at cse.ohio-state.edu
> > > > > http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
> > > > >
> > >
>

More information about the mvapich-discuss mailing list