[mvapich-discuss] Using mvapich2 on a partitioned infiniband network

[Jesper Larsen]

Hi All

We have a new InfiniBand (IB) network which is partitioned into a development part and a production part - both on the same IB switch. There are some nodes and a frontend in each part of the network. This allows us to change stuff in the development partition without having to worry about messing up the production partition (which would be really bad).

The way the partitioning works is essentially that we configure the subnet manager to disallow communication using the default pkey (except with the switch which runs the subnet manager). The development partition then has its own pkey with the development nodes and the development frontend as members. The same goes for the production partition. IPoIB is enabled.

As far as I understand the MPI communication is done directly using a pkey. Normally the default pkey. I have to use another pkey and have tried to set the variable: MV2_DEFAULT_PKEY to the pkey of the development partition. But without luck. Any ideas what I am doing wrong? Or how to see which pkeys are actually used in the communication?

My system is:

$ mpiname -a
MVAPICH2 1.7 Thu Oct 13 17:31:44 EDT 2011 ch3:mrail

Compilation
CC: gcc    -DNDEBUG -DNVALGRIND -O2
CXX: c++   -DNDEBUG -DNVALGRIND -O2
F77: ifort -i-dynamic   -O2
FC: ifort -i-dynamic   -O2

Configuration
--prefix=/usr/mpi/intel/mvapich2-1.7 --enable-shared F77=ifort -i-dynamic FC=ifort -i-dynamic


Best regards,
Jesper