[mvapich-discuss] Problems with RDMA

[Dhabaleswar Panda]

Thanks for your report. A couple of questions:

 - Did you build/install the MPI on these systems or QLogic did?

 - Are you able to run basic MPI benchmarks (such as OSU MPI
   benchmarks)?

 - When you are saying QLogic IB, are the adapters Mellanox-based
   or QLogic-based? These two adapters are very different and provide
   different interfaces to the MPI layer. For Mellanox-based
   adapters, OpenFabrics-Gen2 interface needs to be used for
   both MVAPICH 1 and MVAPICH2. MVAPICH 1 also has a separate
   MVAPICH-PSM interface for QLogic IB adapters supporting
   PSM. Similar interface is being designed for MVAPICH2 and
   will be available in the upcoming MVAPICH2 release.

 - The error report suggests that the application is not able to
   register buffers. Take a look at Section 9.3.4 of MVAPICH2 1.2
   user guide for details. The URL is as follows:

http://mvapich.cse.ohio-state.edu/support/user_guide_mvapich2-1.2.html#x1-580009.3.4

Thanks,

DK

On Fri, 1 May 2009, Katherine Holcomb wrote:

> We have a new cluster with Qlogics IB.  Basic diagnostics provided by
> the vendor work fine, but when I try to run MPI using MVAPICH (1 or 2) I
> get
>
> [vbuf.c 230] Cannot register vbuf region
> Internal Error: invalid error code ffffffff (Ring Index out of range) in
> MPIDI_C
> H3I_RDMA_init:505
> Fatal error in MPI_Init:
> Other MPI error, error stack:
> MPIR_Init_thread(310)....: Initialization failed
> MPID_Init(113)...........: channel initialization failed
> MPIDI_CH3_Init(168)......:
> MPIDI_CH3I_RDMA_init(505):
> [vbuf.c 230] Cannot register vbuf region
> Internal Error: invalid error code ffffffff (Ring Index out of range) in
> MPIDI_C
> H3I_RDMA_init:505
>
> repeated once for each process.
>
> We don't have any experience with IB so we are baffled, and I have not
> found much useful information on Google.
>
> Thanks for any help.
>
> --
> Katherine Holcomb, Ph.D.                kholcomb at virginia.edu
> UVACSE
> I046 BSEL, Clark Hall                          (434) 982-5948
> University of Virginia 22904
>
> _______________________________________________
> mvapich-discuss mailing list
> mvapich-discuss at cse.ohio-state.edu
> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>