[mvapich-discuss] How to check if MVAPICH is using IB network but not ethernetwork?

[Divi Venkateswarlu]

       Hello all:
  
       Good morning! 
       I set up a 64-core cluster based on ROCKS-5.0 using eight Dell PE2900 boxes.
       All are dual-processor QC machines.

       compiled MVAPICH-1.0 (using intel compiler) with default parameters in make.mvapich.gen2
       IB stack is OFED-1.2.5.5. 

        My MD program (PMEMD/AMBER) is compiled with no errors with IFORT/MKL libraries and
       I could run the code on all 64 cores, but the scaling from 16 to 32 to 64 is terrible. I am inclosing
       the benchmarks on a test run.

              # of CPUs/cores   Time (sec)         Nodes (load-balanced)      Scaling (%)
                 8                            82                      8                                        100 
               16                            49                      8                                          84            
              32                            42                      8                                           49
              64                            39                      8                                           26

           In contrast, on single box, I get a reasonable scaling.

       # cores     time (sec)
          2              284    (100%)
          4              164   (87%
          8              107    (65%)

       For some reason, I suspect, MPI traffic is not going over IB net.

           MVAPICH is built using make.mvapich.gen2 with F77=ifort and CC=gcc

     mpif77 -link_info is:

     /state/partition1/fc91052/bin/ifort -L/usr/local/ofed/lib64 -L/usr/local/mvapich/lib 
     -lmpich -L/usr/local/ofed/lib64 -Wl,-rpath=/usr/local/ofed/lib64 -libverbs  
    -libumad -lpthread -lpthread -lrt


     How can I be sure that MPI traffic is going through IB network rather than ethernet?
     Are there any specific checks I should perform?

     Thanks a lot for your help.

      Divi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20080607/0ed904bc/attachment-0001.html