[mvapich-discuss] MVAPICH2 compiler wrappers

[Adam T. Moody]

Hello MVAPICH team,
I'm working to install MVAPICH2-1.5 across the set of our clusters.  We 
have three different target architectures: Mellanox, QLogic, and shared 
memory.  This requires us to build three different versions of 
MVAPICH2-1.5.  Ideally, it would be best if the MPI library could 
determine which channels to load at runtime.  In this case, we could 
make a single build and simply configure it differently for each 
architecture.

In the mean time, we try to make life easier for the application teams 
so that they don't need to build three different versions of their app.  
The trick we play is to ask the application team to link to the shared 
object of the MPI library, instead of the static library.  Then, we 
install the MVAPICH2 build that is appropriate for a given architecture 
into the same directory prefix on each machine.  This way, an 
application team can build a single copy of their app and then run it on 
any of the three architectures.  For this to work, the application must 
only be dependent on the MPI library.  In particular, it can't have 
load-time dependencies on architecture-specific libraries.  For example, 
if an application specifies "-libverbs" at link time, the app will fail 
to load on the shared memory machine where libibverbs does not exist.  
Unfortunately, the default MPI compiler wrappers provided with MVAPICH2 
add architecture-specific dependencies.  For instace, here are the build 
lines from mpicc on each of our machines.

Mellanox:
 >>: mpicc -o mpiBench mpiBench.c -show
gcc -o mpiBench mpiBench.c -I/usr/local/tools/mvapich2-gnu-1.5/include 
-L/usr/local/tools/mvapich2-gnu-1.5/lib 
-L/usr/local/tools/mvapich2-gnu-1.5/lib 
-Wl,-rpath,/usr/local/tools/mvapich2-gnu-1.5/lib -lmpich -lopa -lpmi 
-lpthread -lrdmacm -libverbs -libumad -lrt

QLogic:
 >>: mpicc -o mpiBench mpiBench.c -show
gcc -lpsm_infinipath -o mpiBench mpiBench.c 
-I/usr/local/tools/mvapich2-gnu-1.5/include 
-L/usr/local/tools/mvapich2-gnu-1.5/lib 
-L/usr/local/tools/mvapich2-gnu-1.5/lib 
-Wl,-rpath,/usr/local/tools/mvapich2-gnu-1.5/lib -lmpich -lopa -lpmi 
-lpthread -lpsm_infinipath -lpthread

Shared memory:
 >>: mpicc -o mpiBench mpiBench.c -show
gcc -o mpiBench mpiBench.c -I/usr/local/tools/mvapich2-gnu-1.5/include 
-L/usr/local/tools/mvapich2-gnu-1.5/lib 
-L/usr/local/tools/mvapich2-gnu-1.5/lib 
-Wl,-rpath,/usr/local/tools/mvapich2-gnu-1.5/lib -lmpich -lopa -lpmi 
-lpthread -lrt -lrt

When linking to the shared object, we'd like all three of these to be 
something like the following:
gcc -o mpiBench mpiBench.c -I/usr/local/tools/mvapich2-gnu-1.5/include 
-L/usr/local/tools/mvapich2-gnu-1.5/lib 
-Wl,-rpath,/usr/local/tools/mvapich2-gnu-1.5/lib -lmpich

The shared object of the MPI library can then encode the various 
architecture-specific libraries as dependencies.  Of course, when 
linking to the static library, one still needs to specify the 
architecture-specific libraries.  Thus, the solution is to modify the 
mpicc wrappers to specify one set of libraries for static builds and one 
for shared object builds.  For now, we locally patch the MPI wrappers to 
hardcode what we need, but I suspect other sites would find this feature 
helpful.  I'd be willing to help rewrite the compiler wrappers, but I 
don't want to do this work if you've already started.

I suppose this work could be propogated up to the MPICH level, so I CC'd 
Pavan.
-Adam