[mvapich-discuss] Verify the application is really running over IB

[Jeff Squyres]

On Sep 1, 2007, at 1:48 AM, wgy at altair.com.cn wrote:

> 3)when I try to use hp mpi 2.2.5 over the IB network I got the  
> following:
> [radioss at hpc-node-01 job1]$ /opt/hpmpi/bin/mpirun -stdio=i0
> -cpu_bind=cyclic -VAPI  -f appfile < PFTANKD01
> dlopen test for MPI_ICLIB_VAPI__VAPI_MAIN could not open libs in list
> libmtl_common.so   libmpga.so      libmosal.so     libvapi.so:
> /usr/local/topspin/lib64/libmosal.so: undefined symbol: pthread_create
> dlopen test for MPI_ICLIB_VAPI__VAPI_CISCO could not open libs in list
> libpthread.so     libmosal.so     libvapi.so: /usr/lib64/ 
> libpthread.so:
> invalid ELF header
> mpid: MPI BUG: VAPI requested but not available
> what does it probably indicate? anything is wrong with the IB  
> configuration?

This is a question for HP MPI support.  Please contact them with your  
questions about HP MPI.

> 4)You guys suggested me to use HP MPI (no native mvapich) and OFED IB
> stack if possible.

HP MPI was not recommended on this list; you must be referring to  
some other conversation...?

Cisco QA qualifies both HP MPI and MVAPICH (and other MPI's) on our  
hardware.

> now I have some questiosn hope you can have a quick comment or  
> refer me
> some website link so that I can read through:
> i)how to verify which IB stack is used here, OFED or Cisco/Topspin IB
> stack?

Your RPMs are named "topspin"; hence you are using the Cisco/Topspin  
IB stack.

> what's the advantages of OFED IB stack over Cisco/Topspin IB stack?

There are several:

- The OpenFabrics stack is developed by the community of all  
InfiniBand vendors to provide a single set of drivers and tools that  
are compatible across all IB hardware
- The OpenFabrics stack is open source, the kernel parts of which are  
included in Linus' git tree (and therefore are in main-line Linux  
distros)
- All new work is occurring (and has been occurring for 1-2 years) in  
the OpenFabrics stack for HPC customers
- The OpenFabrics stack has several low-level features that the  
Topspin IB stack does not (features that MPI implementations can take  
advantage of)
- MPI implementations are (and have been for quite a while) focusing  
on the OFED stack rather than the mVAPI-based stacks.  For example: I  
have not worked on the Topspin IB stack in about 1.5 years.

Binary distributions of OFED are available from Cisco under the  
somewhat-confusing moniker of "Server networking and virtualization  
software", here:

     http://www.cisco.com/kobayashi/sw-center/snv.shtml

You need to have a [free] www.cisco.com username/password in order to  
reach the download page.

(if the above link doesn't work for some reason, go to www.cisco.com - 
 > support -> download software -> server networking and  
virtualization software -> linux host driver software packages)

If you have any questions / problems with this procedure, please  
contact me off-list; we're straying a bit off-topic here...

> ii)what's the advatages of HP HPI over "native mvapich"?

Every MPI claims to have advantages over its competitors; you should  
probably ask your HP MPI contact for their marketing information.   
Some obvious differences:

- MVAPICH is open source; HP MPI is closed source
- MVAPICH is supported by the community; HP MPI has paid/contract  
support

> what means by
> "native mvapich"? the one shipped with Cisco/Topspin? is it enough to
> upgrade mvapich to the latest one which is availabel on mvapich  
> website?

Cisco certifies the versions of MVAPICH that are included with OFED  
distributions.  The version of MVAPICH included in OFED v1.2 is  
*much* more recent compared to the version included in the Topspin IB  
stack.  Cisco does not certify or support other versions of MVAPICH,  
but the community (e.g., this mailing list) provides good support.

-- 
Jeff Squyres
Cisco Systems