[mvapich-discuss] mpirun_rsh and Chelsio cluster running RHEL5

[Bryan Putnam]

Hi All,

When attempting to use "mpirun_rsh" (from mvapich2-1.4 or mvapich2-1.4rc2) 
on a Chelsio cluster, I get errors which are reproduced below. Note that I 
don't see these errors with mvapich2 when using MPD/mpiexec or the 
mpiexec.hydra launcher from the mpich2 distribution.

Thanks,
Bryan
===============================================

coates-a012 1005% mpif90 hellof.f -o hellof
coates-a012 1006% cat $PBS_NODEFILE
coates-a012
coates-a012
coates-a012
coates-a012
coates-a040
coates-a040
coates-a040
coates-a040
coates-a012 1007% mpirun_rsh -hostfile $PBS_NODEFILE -np 8 ./hellof
Fatal error in MPI_Init:
Error message texts are not available
Exit code -5 signaled from coates-a040
MPI process (rank: 4) terminated unexpectedly on 
coates-a040.rcac.purdue.edu
Fatal error in MPI_Init:
Error message texts are not available
Fatal error in MPI_Init:
Error message texts are not available
MPI process (rank: 0) terminated unexpectedly on 
coates-a012.rcac.purdue.edu
forrtl: error (69): process interrupted (SIGINT)
Image              PC                Routine            Line        Source             
libc.so.6          000000341045492F  Unknown               Unknown  
Unknown
libc.so.6          0000003410463C15  Unknown               Unknown  
Unknown
libc.so.6          000000341045EF68  Unknown               Unknown  
Unknown
libmthca-rdmav2.s  00002B000ADB7C26  Unknown               Unknown  
Unknown
libibverbs.so.1    000000341180722C  Unknown               Unknown  
Unknown
libibverbs.so.1    00000034118082A3  Unknown               Unknown  
Unknown
libibverbs.so.1    000000341180707B  Unknown               Unknown  
Unknown
libmpich.so.1.1    00002B0009C76298  Unknown               Unknown  
Unknown
libmpich.so.1.1    00002B0009C3C992  Unknown               Unknown  
Unknown
libmpich.so.1.1    00002B0009C751DC  Unknown               Unknown  
Unknown
libmpich.so.1.1    00002B0009BFBB6D  Unknown               Unknown  
Unknown
libmpich.so.1.1    00002B0009C4B715  Unknown               Unknown  
Unknown
libmpich.so.1.1    00002B0009C46116  Unknown               Unknown  
Unknown
libmpich.so.1.1    00002B0009C4566A  Unknown               Unknown  
Unknown
libmpich.so.1.1    00002B0009C45560  Unknown               Unknown  
Unknown
hellof             0000000000400F61  Unknown               Unknown  
Unknown
hellof             0000000000400EFC  Unknown               Unknown  
Unknown
libc.so.6          000000341041D994  Unknown               Unknown  
Unknown
hellof             0000000000400E09  Unknown               Unknown  
Unknown
forrtl: error (69): process interrupted (SIGINT)
Image              PC                Routine            Line        Source             
libpthread.so.0    0000003410C0D590  Unknown               Unknown  
Unknown
libibverbs.so.1    00000034118090D4  Unknown               Unknown  
Unknown
libcxgb3-rdmav2.s  00002B7457929C78  Unknown               Unknown  
Unknown
libibverbs.so.1    000000341180722C  Unknown               Unknown  
Unknown
libibverbs.so.1    00000034118082A3  Unknown               Unknown  
Unknown
libibverbs.so.1    000000341180707B  Unknown               Unknown  
Unknown
libmpich.so.1.1    00002B74565E1298  Unknown               Unknown  
Unknown
libmpich.so.1.1    00002B74565A7992  Unknown               Unknown  
Unknown
libmpich.so.1.1    00002B74565E01DC  Unknown               Unknown  
Unknown
libmpich.so.1.1    00002B7456566B6D  Unknown               Unknown  
Unknown
libmpich.so.1.1    00002B74565B6715  Unknown               Unknown  
Unknown
libmpich.so.1.1    00002B74565B1116  Unknown               Unknown  
Unknown
libmpich.so.1.1    00002B74565B066A  Unknown               Unknown  
Unknown
libmpich.so.1.1    00002B74565B0560  Unknown               Unknown  
Unknown
hellof             0000000000400F61  Unknown               Unknown  
Unknown
hellof             0000000000400EFC  Unknown               Unknown  
Unknown
libc.so.6          000000341041D994  Unknown               Unknown  
Unknown
hellof             0000000000400E09  Unknown               Unknown  
Unknown
forrtl: error (69): process interrupted (SIGINT)
Image              PC                Routine            Line        Source             
ld-linux-x86-64.s  0000003410014BA6  Unknown               Unknown  
Unknown
ld-linux-x86-64.s  000000341000610F  Unknown               Unknown  
Unknown
ld-linux-x86-64.s  0000003410007D33  Unknown               Unknown  
Unknown
ld-linux-x86-64.s  0000003410010C4D  Unknown               Unknown  
Unknown
ld-linux-x86-64.s  000000341000CE96  Unknown               Unknown  
Unknown
ld-linux-x86-64.s  000000341001064C  Unknown               Unknown  
Unknown
libdl.so.2         0000003410800F9A  Unknown               Unknown  
Unknown
ld-linux-x86-64.s  000000341000CE96  Unknown               Unknown  
Unknown
libdl.so.2         000000341080150D  Unknown               Unknown  
Unknown
libdl.so.2         0000003410800F11  Unknown               Unknown  
Unknown
libibverbs.so.1    0000003411807151  Unknown               Unknown  
Unknown
libibverbs.so.1    0000003411807410  Unknown               Unknown  
Unknown
libibverbs.so.1    000000341180825F  Unknown               Unknown  
Unknown
libibverbs.so.1    000000341180707B  Unknown               Unknown  
Unknown
libmpich.so.1.1    00002BA2E8701298  Unknown               Unknown  
Unknown
libmpich.so.1.1    00002BA2E86C7992  Unknown               Unknown  
Unknown
libmpich.so.1.1    00002BA2E87001DC  Unknown               Unknown  
Unknown
libmpich.so.1.1    00002BA2E8686B6D  Unknown               Unknown  
Unknown
libmpich.so.1.1    00002BA2E86D6715  Unknown               Unknown  
Unknown
libmpich.so.1.1    00002BA2E86D1116  Unknown               Unknown  
Unknown
libmpich.so.1.1    00002BA2E86D066A  Unknown               Unknown  
Unknown
libmpich.so.1.1    00002BA2E86D0560  Unknown               Unknown  
Unknown
hellof             0000000000400F61  Unknown               Unknown  
Unknown
hellof             0000000000400EFC  Unknown               Unknown  
Unknown
libc.so.6          000000341041D994  Unknown               Unknown  
Unknown
hellof             0000000000400E09  Unknown               Unknown  
Unknown
coates-a012 1008%