[mvapich-discuss] mvapich2 IB problems with transfers over ~10KB.

[Mike Houston]

We've hit an odd snag with using mvapich2.  We can't seem to reliably 
send messages > 10KB.  If we break up all large messages into 8KB blocks 
and send, things work just fine, but as expected, performance is awful.  
Under mpich2 with GigE and IPoIB, large messages seem to work just 
fine.  Both MPI_Send and MPI_Put seem to exhibit the same behavior.  I 
should note that the one oddity of our system implementation is that we 
have a posted MPI_IRecv waiting while doing the large transfers.  
Open-MPI flips out when we do this, even in tcp mode.

We have PCI-X SDR 4X boards, running the latest IB Gold release (1.8.3) 
ontop of the latest RHEL4 SMP x86 kernel (32-bit).  The boards have 
slightly older firmware, 3.3.3, but I'm hesitant to flash up unless 
there are known issues with that firmware...  We built using the 
defaults in make.mvapich2.vapi.  Any suggestions on where to look or 
what to update?  It seems *very* odd that large transfers aren't working 
for us...

Thanks!

-Mike