[mvapich-discuss] Running more than 72 tasks with mvapich 0.9.5

[Otheus]

Greetings,

We have a new cluster using IB and MVAPICH 0.9.5 (I'd upgrade to 0.9.6, 
but this is what the vendor is currently supporting.) When I run an MPI 
job, such as xhpl or even a hello-world I wrote, using more than 75 
tasks, the job fails. I have not been able to find this problem 
addressed on your mailing list. Help is greatly appreciated.

-bash-3.00$ mpirun_rsh -rsh -np 80 -hostfile cluster-of-10.txt $PWD/xhpl
[30] Abort: [k04:30] Got completion with error, code=VAPI_RETRY_EXC_ERR, 
vendor code=81, dest rank=14
 at line 372 in file viacheck.c
mpirun_rsh: Abort signaled from [30]
[47] Abort: [k06:47] Got completion with error, code=VAPI_RETRY_EXC_ERR, 
vendor code=81, dest rank=0
 at line 372 in file viacheck.c
[63] Abort: [k08:63] Got completion with error, code=VAPI_RETRY_EXC_ERR, 
vendor code=81, dest rank=3
 at line 372 in file viacheck.c
[21] Abort: [k03:21] Got completion with error, code=VAPI_RETRY_EXC_ERR, 
vendor code=81, dest rank=11
 at line 372 in file viacheck.c
[23] Abort: [k03:23] Got completion with error, code=VAPI_RETRY_EXC_ERR, 
vendor code=81, dest rank=0
 at line 372 in file viacheck.c
[31] Abort: [k04:31] Got completion with error, code=VAPI_RETRY_EXC_ERR, 
vendor code=81, dest rank=15
 at line 372 in file viacheck.c
[52] Abort: [k07:52] Got completion with error, code=VAPI_RETRY_EXC_ERR, 
vendor code=81, dest rank=13
 at line 372 in file viacheck.c
[44] Abort: [k06:44] Got completion with error, code=VAPI_RETRY_EXC_ERR, 
vendor code=81, dest rank=0
 at line 372 in file viacheck.c
[29] Abort: [k04:29] Got completion with error, code=VAPI_RETRY_EXC_ERR, 
vendor code=81, dest rank=0
 at line 372 in file viacheck.c
[46] Abort: [k06:46] Got completion with error, code=VAPI_RETRY_EXC_ERR, 
vendor code=81, dest rank=0
 at line 372 in file viacheck.c
[28] Abort: [k04:28] Got completion with error, code=VAPI_RETRY_EXC_ERR, 
vendor code=81, dest rank=2
 at line 372 in file viacheck.c
[57] Abort: [k08:57] Got completion with error, code=VAPI_RETRY_EXC_ERR, 
vendor code=81, dest rank=1
 at line 372 in file viacheck.c
[66] Abort: [k09:66] Got completion with error, code=VAPI_RETRY_EXC_ERR, 
vendor code=81, dest rank=15
 at line 372 in file viacheck.c
[74] Abort: [k10:74] Got completion with error, code=VAPI_RETRY_EXC_ERR, 
vendor code=81, dest rank=2
 at line 372 in file viacheck.c
[76] Abort: [k10:76] Got completion with error, code=VAPI_RETRY_EXC_ERR, 
vendor code=81, dest rank=3
 at line 372 in file viacheck.c
[65] Abort: [k09:65] Got completion with error, code=VAPI_RETRY_EXC_ERR, 
vendor code=81, dest rank=0
 at line 372 in file viacheck.c    
 at line 1835 in file viacheck.c
[75] Abort: [k10:75] Got completion with error, code=VAPI_RETRY_EXC_ERR, 
vendor code=81
 at line 1835 in file viacheck.c
[1] Abort: [k01:1] Got completion with error, code=VAPI_RETRY_EXC_ERR, 
vendor code=81
 at line 1835 in file viacheck.c
[6] Abort: [k01:6] Got completion with error, code=VAPI_RETRY_EXC_ERR, 
vendor code=81
 at line 1835 in file viacheck.c
[7] Abort: [k01:7] Got completion with error, code=VAPI_RETRY_EXC_ERR, 
vendor code=81
 at line 1835 in file viacheck.c
[0] Abort: [k01:0] Got completion with error, code=VAPI_RETRY_EXC_ERR, 
vendor code=81
 at line 1835 in file viacheck.c
[4] Abort: [k01:4] Got completion with error, code=VAPI_RETRY_EXC_ERR, 
vendor code=81
 at line 1835 in file viacheck.c
[2] Abort: [k01:2] Got completion with error, code=VAPI_RETRY_EXC_ERR, 
vendor code=81, dest rank=62
 at line 372 in file viacheck.c
[71] Abort: [k09:71] Got completion with error, code=VAPI_RETRY_EXC_ERR, 
vendor code=81
 at line 1835 in file viacheck.c
[9] Abort: [k02:9] Got completion with error, code=VAPI_RETRY_EXC_ERR, 
vendor code=81
 at line 1835 in file viacheck.c
[12] Abort: [k02:12] Got completion with error, code=VAPI_RETRY_EXC_ERR, 
vendor code=81
 at line 1835 in file viacheck.c
[8] Abort: [k02:8] Got completion with error, code=VAPI_RETRY_EXC_ERR, 
vendor code=81
 at line 1835 in file viacheck.c
[15] Abort: [k02:15] Got completion with error, code=VAPI_RETRY_EXC_ERR, 
vendor code=81
 at line 1835 in file viacheck.c
[11] Abort: [k02:11] Got completion with error, code=VAPI_RETRY_EXC_ERR, 
vendor code=81
 at line 1835 in file viacheck.c
[13] Abort: [k02:13] Got completion with error, code=VAPI_RETRY_EXC_ERR, 
vendor code=81
 at line 1835 in file viacheck.c
[14] Abort: [k02:14] Got completion with error, code=VAPI_RETRY_EXC_ERR, 
vendor code=81
 at line 1835 in file viacheck.c
[10] Abort: [k02:10] Got completion with error, code=VAPI_RETRY_EXC_ERR, 
vendor code=81
 at line 1835 in file viacheck.c
[38] Abort: [k05:38] Got completion with error, code=VAPI_RETRY_EXC_ERR, 
vendor code=81
 at line 1835 in file viacheck.c
[33] Abort: [k05:33] Got completion with error, code=VAPI_RETRY_EXC_ERR, 
vendor code=81, dest rank=15
 at line 372 in file viacheck.c
[35] Abort: [k05:35] Got completion with error, code=VAPI_RETRY_EXC_ERR, 
vendor code=81
 at line 1835 in file viacheck.c
[3] Abort: [k01:3] Got completion with error, code=VAPI_RETRY_EXC_ERR, 
vendor code=81
 at line 1835 in file viacheck.c
[5] Abort: [k01:5] Got completion with error, code=VAPI_RETRY_EXC_ERR, 
vendor code=81
 at line 1835 in file viacheck.c

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20060508/7affacfa/attachment.html