[mvapich-discuss] delay in spawning

Hossein Pourreza pourreza at cs.umanitoba.ca
Tue Feb 17 16:56:23 EST 2009 

I checked the FAQ and there is a *similar* problem. In my case the error message
does not say anything about mpispawn it reads:
/usr/bin/env: No such file or directory

I, however, included the full path to mpirun_sh and ran my program but there was
no output (not even error message). I ran top command on the other window and
it was showing many (maybe 10) ssh commands running.

I checked the debug output and it seems that execv is missing on my computer and
it may cause the problem. I used your google-enabled search tool but there is
nothing about "execv".

Also, as I mentioned earlier, running mpirun_rsh with -v option gives Unknown
option error. Again, I searched your web site and there is no posting regarding
this issue.

Thanks

On Tue, Feb 17, 2009 at 02:21:45PM -0500, Dhabaleswar Panda wrote:
> Hossein,
> 
> This error and its solution are indicated in the FAQ page of the user
> guide at the following URL:
> 
> http://mvapich.cse.ohio-state.edu/support/user_guide_mvapich2-1.2.html#x1-520009.2.3
> 
> Let us know if this solves your problem.
> 
> FYI, MVAPICH web site (including user guide and postings on
> mvapich-discuss) are enabled by Google Search. If you encounter any
> problem, please do a search first and you will get an answer quickly if
> this has been reported/resolved earlier.
> 
> DK
> 
> On Tue, 17 Feb 2009, Hossein Pourreza wrote:
> 
> > Thanks for the reply.
> >
> > Situation got worse :)
> >
> > I am getting the following error:
> >
> > /usr/bin/env: No such file or directory
> >
> > When I ran the mpirun_rsh with -v option it gave me:
> >
> > Unknown option
> >
> > and finally with -debug option I am getting following output:
> >
> > $mpirun_rsh -debug -np 32 -hostfile hosts ./bin/ep.B.32
> >
> > execv: No such file or directory
> > /usr/X11R6/bin/xterm -e /usr/bin/ssh -q power02 cd
> > /stage/Benchmarks/NPB2.4/NPB2.4-MPI; /usr/bin/env
> > LD_LIBRARY_PATH=/usr/mvapich/lib/shared:/stage/mvapich2.1/lib:
> > MPISPAWN_MPIRUN_MPD=0 MPISPAWN_MPIRUN_HOST=power01 MPISPAWN_CHECKIN_PORT=33209
> > MPISPAWN_MPIRUN_PORT=33209 MPISPAWN_GLOBAL_NPROCS=32 MPISPAWN_MPIRUN_ID=11731
> > MPISPAWN_ARGC=2 MPISPAWN_ARGV_0=/usr/bin/gdb MPISPAWN_ARGV_1=./bin/ep.B.32
> > MPISPAWN_GENERIC_ENV_COUNT=0  MPISPAWN_ID=0 MPISPAWN_LOCAL_NPROCS=4
> > MPISPAWN_WORKING_DIR=/stage/Benchmarks/NPB2.4/NPB2.4-MPI
> > MPISPAWN_MPIRUN_RANK_0=0 MPISPAWN_VIADEV_DEFAULT_PORT_0=-1
> > MPISPAWN_MPIRUN_RANK_1=1 MPISPAWN_VIADEV_DEFAULT_PORT_1=-1
> > MPISPAWN_MPIRUN_RANK_2=2 MPISPAWN_VIADEV_DEFAULT_PORT_2=-1
> > MPISPAWN_MPIRUN_RANK_3=3 MPISPAWN_VIADEV_DEFAULT_PORT_3=-1  mpispawn execv: No
> > such file or directory
> > /usr/X11R6/bin/xterm -e /usr/bin/ssh -q power03 cd
> > /stage/Benchmarks/NPB2.4/NPB2.4-MPI; /usr/bin/env
> > LD_LIBRARY_PATH=/usr/mvapich/lib/shared:/stage/mvapich2.1/lib:
> > MPISPAWN_MPIRUN_MPD=0 MPISPAWN_MPIRUN_HOST=power01 MPISPAWN_CHECKIN_PORT=33209
> > MPISPAWN_MPIRUN_PORT=33209 MPISPAWN_GLOBAL_NPROCS=32 MPISPAWN_MPIRUN_ID=11731
> > MPISPAWN_ARGC=2 MPISPAWN_ARGV_0=/usr/bin/gdb MPISPAWN_ARGV_1=./bin/ep.B.32
> > MPISPAWN_GENERIC_ENV_COUNT=0  MPISPAWN_ID=1 MPISPAWN_LOCAL_NPROCS=4
> > MPISPAWN_WORKING_DIR=/stage/Benchmarks/NPB2.4/NPB2.4-MPI
> > MPISPAWN_MPIRUN_RANK_0=4 MPISPAWN_VIADEV_DEFAULT_PORT_0=-1
> > MPISPAWN_MPIRUN_RANK_1=5 MPISPAWN_VIADEV_DEFAULT_PORT_1=-1
> > MPISPAWN_MPIRUN_RANK_2=6 MPISPAWN_VIADEV_DEFAULT_PORT_2=-1
> > MPISPAWN_MPIRUN_RANK_3=7 MPISPAWN_VIADEV_DEFAULT_PORT_3=-1  mpispawn execv: No
> > such file or directory
> > /usr/X11R6/bin/xterm -e /usr/bin/ssh -q power04 cd
> > /stage/Benchmarks/NPB2.4/NPB2.4-MPI; /usr/bin/env
> > LD_LIBRARY_PATH=/usr/mvapich/lib/shared:/stage/mvapich2.1/lib:
> > MPISPAWN_MPIRUN_MPD=0 MPISPAWN_MPIRUN_HOST=power01 MPISPAWN_CHECKIN_PORT=33209
> > MPISPAWN_MPIRUN_PORT=33209 MPISPAWN_GLOBAL_NPROCS=32 MPISPAWN_MPIRUN_ID=11731
> > MPISPAWN_ARGC=2 MPISPAWN_ARGV_0=/usr/bin/gdb MPISPAWN_ARGV_1=./bin/ep.B.32
> > MPISPAWN_GENERIC_ENV_COUNT=0  MPISPAWN_ID=2 MPISPAWN_LOCAL_NPROCS=4
> > MPISPAWN_WORKING_DIR=/stage/Benchmarks/NPB2.4/NPB2.4-MPI
> > MPISPAWN_MPIRUN_RANK_0=8 MPISPAWN_VIADEV_DEFAULT_PORT_0=-1
> > MPISPAWN_MPIRUN_RANK_1=9 MPISPAWN_VIADEV_DEFAULT_PORT_1=-1
> > MPISPAWN_MPIRUN_RANK_2=10 MPISPAWN_VIADEV_DEFAULT_PORT_2=-1
> > MPISPAWN_MPIRUN_RANK_3=11 MPISPAWN_VIADEV_DEFAULT_PORT_3=-1  mpispawn execv: No
> > such file or directory
> > /usr/X11R6/bin/xterm -e /usr/bin/ssh -q power06 cd
> > /stage/Benchmarks/NPB2.4/NPB2.4-MPI; /usr/bin/env
> > LD_LIBRARY_PATH=/usr/mvapich/lib/shared:/stage/mvapich2.1/lib:
> > MPISPAWN_MPIRUN_MPD=0 MPISPAWN_MPIRUN_HOST=power01 MPISPAWN_CHECKIN_PORT=33209
> > MPISPAWN_MPIRUN_PORT=33209 MPISPAWN_GLOBAL_NPROCS=32 MPISPAWN_MPIRUN_ID=11731
> > MPISPAWN_ARGC=2 MPISPAWN_ARGV_0=/usr/bin/gdb MPISPAWN_ARGV_1=./bin/ep.B.32
> > MPISPAWN_GENERIC_ENV_COUNT=0  MPISPAWN_ID=4 MPISPAWN_LOCAL_NPROCS=4
> > MPISPAWN_WORKING_DIR=/stage/Benchmarks/NPB2.4/NPB2.4-MPI
> > MPISPAWN_MPIRUN_RANK_0=16 MPISPAWN_VIADEV_DEFAULT_PORT_0=-1
> > MPISPAWN_MPIRUN_RANK_1=17 MPISPAWN_VIADEV_DEFAULT_PORT_1=-1
> > MPISPAWN_MPIRUN_RANK_2=18 MPISPAWN_VIADEV_DEFAULT_PORT_2=-1
> > MPISPAWN_MPIRUN_RANK_3=19 MPISPAWN_VIADEV_DEFAULT_PORT_3=-1  mpispawn execv: No
> > such file or directory
> > /usr/X11R6/bin/xterm -e /usr/bin/ssh -q power05 cd
> > /stage/Benchmarks/NPB2.4/NPB2.4-MPI; /usr/bin/env
> > LD_LIBRARY_PATH=/usr/mvapich/lib/shared:/stage/mvapich2.1/lib:
> > MPISPAWN_MPIRUN_MPD=0 MPISPAWN_MPIRUN_HOST=power01 MPISPAWN_CHECKIN_PORT=33209
> > MPISPAWN_MPIRUN_PORT=33209 MPISPAWN_GLOBAL_NPROCS=32 MPISPAWN_MPIRUN_ID=11731
> > MPISPAWN_ARGC=2 MPISPAWN_ARGV_0=/usr/bin/gdb MPISPAWN_ARGV_1=./bin/ep.B.32
> > MPISPAWN_GENERIC_ENV_COUNT=0  MPISPAWN_ID=3 MPISPAWN_LOCAL_NPROCS=4
> > MPISPAWN_WORKING_DIR=/stage/Benchmarks/NPB2.4/NPB2.4-MPI
> > MPISPAWN_MPIRUN_RANK_0=12 MPISPAWN_VIADEV_DEFAULT_PORT_0=-1
> > MPISPAWN_MPIRUN_RANK_1=13 MPISPAWN_VIADEV_DEFAULT_PORT_1=-1
> > MPISPAWN_MPIRUN_RANK_2=14 MPISPAWN_VIADEV_DEFAULT_PORT_2=-1
> > MPISPAWN_MPIRUN_RANK_3=15 MPISPAWN_VIADEV_DEFAULT_PORT_3=-1  mpispawn execv: No
> > such file or directory
> > /usr/X11R6/bin/xterm -e /usr/bin/ssh -q power07 cd
> > /stage/Benchmarks/NPB2.4/NPB2.4-MPI; /usr/bin/env
> > LD_LIBRARY_PATH=/usr/mvapich/lib/shared:/stage/mvapich2.1/lib:
> > MPISPAWN_MPIRUN_MPD=0 MPISPAWN_MPIRUN_HOST=power01 MPISPAWN_CHECKIN_PORT=33209
> > MPISPAWN_MPIRUN_PORT=33209 MPISPAWN_GLOBAL_NPROCS=32 MPISPAWN_MPIRUN_ID=11731
> > MPISPAWN_ARGC=2 MPISPAWN_ARGV_0=/usr/bin/gdb MPISPAWN_ARGV_1=./bin/ep.B.32
> > MPISPAWN_GENERIC_ENV_COUNT=0  MPISPAWN_ID=5 MPISPAWN_LOCAL_NPROCS=4
> > MPISPAWN_WORKING_DIR=/stage/Benchmarks/NPB2.4/NPB2.4-MPI
> > MPISPAWN_MPIRUN_RANK_0=20 MPISPAWN_VIADEV_DEFAULT_PORT_0=-1
> > MPISPAWN_MPIRUN_RANK_1=21 MPISPAWN_VIADEV_DEFAULT_PORT_1=-1
> > MPISPAWN_MPIRUN_RANK_2=22 MPISPAWN_VIADEV_DEFAULT_PORT_2=-1
> > MPISPAWN_MPIRUN_RANK_3=23 MPISPAWN_VIADEV_DEFAULT_PORT_3=-1  mpispawn execv: No
> > such file or directory
> > /usr/X11R6/bin/xterm -e /usr/bin/ssh -q power08 cd
> > /stage/Benchmarks/NPB2.4/NPB2.4-MPI; /usr/bin/env
> > LD_LIBRARY_PATH=/usr/mvapich/lib/shared:/stage/mvapich2.1/lib:
> > MPISPAWN_MPIRUN_MPD=0 MPISPAWN_MPIRUN_HOST=power01 MPISPAWN_CHECKIN_PORT=33209
> > MPISPAWN_MPIRUN_PORT=33209 MPISPAWN_GLOBAL_NPROCS=32 MPISPAWN_MPIRUN_ID=11731
> > MPISPAWN_ARGC=2 MPISPAWN_ARGV_0=/usr/bin/gdb MPISPAWN_ARGV_1=./bin/ep.B.32
> > MPISPAWN_GENERIC_ENV_COUNT=0  MPISPAWN_ID=6 MPISPAWN_LOCAL_NPROCS=4
> > MPISPAWN_WORKING_DIR=/stage/Benchmarks/NPB2.4/NPB2.4-MPI
> > MPISPAWN_MPIRUN_RANK_0=24 MPISPAWN_VIADEV_DEFAULT_PORT_0=-1
> > MPISPAWN_MPIRUN_RANK_1=25 MPISPAWN_VIADEV_DEFAULT_PORT_1=-1
> > MPISPAWN_MPIRUN_RANK_2=26 MPISPAWN_VIADEV_DEFAULT_PORT_2=-1
> > MPISPAWN_MPIRUN_RANK_3=27 MPISPAWN_VIADEV_DEFAULT_PORT_3=-1  mpispawn execv: No
> > such file or directory
> > /usr/X11R6/bin/xterm -e /usr/bin/ssh -q power09 cd
> > /stage/Benchmarks/NPB2.4/NPB2.4-MPI; /usr/bin/env
> > LD_LIBRARY_PATH=/usr/mvapich/lib/shared:/stage/mvapich2.1/lib:
> > MPISPAWN_MPIRUN_MPD=0 MPISPAWN_MPIRUN_HOST=power01 MPISPAWN_CHECKIN_PORT=33209
> > MPISPAWN_MPIRUN_PORT=33209 MPISPAWN_GLOBAL_NPROCS=32 MPISPAWN_MPIRUN_ID=11731
> > MPISPAWN_ARGC=2 MPISPAWN_ARGV_0=/usr/bin/gdb MPISPAWN_ARGV_1=./bin/ep.B.32
> > MPISPAWN_GENERIC_ENV_COUNT=0  MPISPAWN_ID=7 MPISPAWN_LOCAL_NPROCS=4
> > MPISPAWN_WORKING_DIR=/stage/Benchmarks/NPB2.4/NPB2.4-MPI
> > MPISPAWN_MPIRUN_RANK_0=28 MPISPAWN_VIADEV_DEFAULT_PORT_0=-1
> > MPISPAWN_MPIRUN_RANK_1=29 MPISPAWN_VIADEV_DEFAULT_PORT_1=-1
> > MPISPAWN_MPIRUN_RANK_2=30 MPISPAWN_VIADEV_DEFAULT_PORT_2=-1
> > MPISPAWN_MPIRUN_RANK_3=31 MPISPAWN_VIADEV_DEFAULT_PORT_3=-1  mpispawn
> > Child exited abnormally!
> > cleanupKilling remote processes...DONE
> >
> > On Tue, Feb 17, 2009 at 12:38:34PM -0500, Dhabaleswar Panda wrote:
> > > You are using the old MPD-based start-up. Starting with MVAPICH2 1.2, a
> > > new scalable and robust mpirun_rsh framework (non-MPD-based) job launching
> > > mechanism has been added. This framework reduces job start-up time
> > > considerably and is also scalable to multi-thousand core clusters. Please
> > > use this and you should see considerable improvement in your job launching
> > > framework. Please refer to the MVAPICH2 user guide from the following URL
> > > for details on using the mpirun_rsh framework:
> > >
> > > http://mvapich.cse.ohio-state.edu/support/user_guide_mvapich2-1.2.html
> > >
> > > If the problem persists with the new mpirun_rsh framework, let us know.
> > >
> > > DK
> > >
> > > On Tue, 17 Feb 2009, Hossein Pourreza wrote:
> > >
> > > > Hi,
> > > >
> > > > I used to experience with a very long delay before execution of my tasks with
> > > > MVAPICH-0.9.8 and reported that problem. I was hopping to see a fix in the new
> > > > version but I still have the same problem with MVAPICH-1.2. The configuration of
> > > > my machine is: (I use SDR infiniband on PCI_X bus)
> > > >
> > > >
> > > > uname -v = Generic_137112-07
> > > >
> > > > /usr/bin/uname -p = i386
> > > > /bin/uname -X     = System = SunOS
> > > > Node = power
> > > > Release = 5.10
> > > > KernelID = Generic_137112-07
> > > > Machine = i86pc
> > > > BusType = <unknown>
> > > > Serial = <unknown>
> > > > Users = <unknown>
> > > > OEM# = 0
> > > > Origin# = 1
> > > > NumCPU = 4
> > > >
> > > > /bin/arch              = i86pc
> > > > /usr/bin/arch -k       = i86pc
> > > > /usr/convex/getsysinfo = unknown
> > > > hostinfo               = unknown
> > > > /bin/machine           = unknown
> > > >
> > > > If I try to run my program with processes more than the number of cores of one
> > > > node, I experience a very long delay before my task gets started. This delay
> > > > increases by increasing number of processes. I run my tasks using:
> > > >
> > > > mpiexec -np xx ./myprog
> > > >
> > > > I am using mpd as processes placement daemon.
> > > >
> > > > Any help will be greatly appreciated.
> > > > _______________________________________________
> > > > mvapich-discuss mailing list
> > > > mvapich-discuss at cse.ohio-state.edu
> > > > http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
> > > >
> >

-- 
Hossein Pourreza		 			e-mail:<pourreza AT(@) cs DOT umanitoba DOT ca>    
Department of Computer Science		URL: http://www.cs.umanitoba.ca/~pourreza
University of Manitoba  			Phone: 204-474-8391            
Winnipeg, Manitoba, Canada R3T 2N2

More information about the mvapich-discuss mailing list