[mvapich-discuss] mvapich2 + PGI + TCP/IPoIB

[Dhabaleswar Panda]

> Howdy all.
> 
> > It doesn't appear that /usr/pgi/linux86-64/6.1/license.dat exists, so
> > double-check to make sure you have set the LM_LICENSE_FILE environment
> > variable correctly. You may need to set this value to a FLEXlm license
> > server, depending on your setup.
> 
> Well, if this does not make me look like a total ass I don't know what will:-) 
> The LM_LICENSE_FILE env variable was set in my .bashrc file, but as I was 
> doing this as root everything went bananas.
> 
> > Please let us know if you have any other questions or if this does not
> > solve your problem.
>
> MVAPICH2 installes and runs smoothly. However, the results for
> running the MM5 weather model on 20 CPU's are discouraging. On
> average I'm only getting about 35% CPU usage and the model is veeery
> slow.
>
> My "educated" guess is that MVAPICH2 is not taking advantage of the Topspin 
> IPoIB software. Any thoughts on this???

Glad to know that you are able to install MVAPICH2 and run it
smoothly.

Are you configuring MVAPICH2 to run with the underlying IPoIB stack?
The latest release of MVAPICH2 has support for multiple underlying
protocol stacks/interfaces: VAPI, uDAPL, OpenIB/Gen2 and IPoIB. All
these are possible with InfiniBand fabric. You need to configure it
properly to select the appropriate underlying interface/stack. The
user guide indicates how to select the appropriate underlying stack.

FYI, IPoIB stack is not a high performance communication stack with
InfiniBand. You need to use low-level interfaces (VAPI or OpenIB/Gen2)
to get the maximum performance.

Thus, I am suspecting that you have configured MVAPICH2 to run with
IPoIB (even though you are not sure). If this is true (please verify),
you will not get good performance. You need to configure it with VAPI
or OpenIB/Gen2 to get better performance numbers.

To verify on which stack your installation is running, please feel
free to run OSU MPI micro-benchmarks (available with the
distribution).  It has several tests including latency, bandwidth,
bi-directional bandwidth and multicast tests. MVAPICH2 distribution
also has additional one-sided tests. With VAPI or OpenIB/Gen2
installation, you should be able to get very good performance
numbers. You can verify these numbers with the numbers available on
the MVAPICH web page.  Alternatively, please run these tests and
report back the numbers on this mailing list so that we can tell you
whether you are running it on the low level VAPI or Gen2 interface or
not.

> Using Topspin MPI, I'm getting approximately 89% and 83% scaling with 12 and 
> 20 CPUs, respectively (as compared to running MM5 on a single CPU).

FYI, Topspin MPI is based on MVAPICH. To the best of our knowledge, it
is based on MVAPICH 0.9.5-xxx (where xxx indicates the patch number)
and the default underlying interface is on VAPI not on IPoIB. Please
check this. It does not include MVAPICH2 support yet. May be somebody
from Topspin/Cisco can clarify this.

Also, FYI, MVAPICH 0.9.5/0.9.6 have advanced support for collectives,
scalability, etc. Thus, you are welcome to try MVAPICH or MVAPICH2
with VAPI or Gen2 interface and see what results you get.

Hope this helps. 

DK

> All the best, Ólafur.
> -- 
> Ólafur Rögnvaldsson 
> or at os.is
> http://www.os.is/~or/rev
> Phone: +(354) 569 6000
>            +(354) 864 4943
> 
> _______________________________________________
> mvapich-discuss mailing list
> mvapich-discuss at cse.ohio-state.edu
> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>