[mvapich-discuss] Is there a way to make ch3:psm work on SLURM-based systems?

[Gunter, David O]

We have a slurm-based system running Livermore's CHAOS stack. This system also uses the Qlogic Infinipath card and I know that we should be configuring mvapich2-1.8 with the '-with-device=ch3:psm' option, but when we do this we get an error that slurm is not supported.  So we configure the old way, '--with-pmi=slurm  --with-pm=no --with-rdma=gen2' and things seem to work.  However, when testing via the OSU benmarks we see a huge output of warning messages tell us to re-configure the library with the '--with-device=ch3:psm' configure option. (See below)

Is there a way to either use the ch3:psm device with slurm or is there a way to turn off this message to keep it from appearing?

Thanks,
david

(dog at ml005 12%) srun -N 16 -n 64 osu_mbw_mr
[ml005.localdomain:mpi_rank_0][rdma_find_network_type] QLogic IB card detected in system
[ml005.localdomain:mpi_rank_0][rdma_find_network_type] Please re-configure the library with the '--with-device=ch3:psm' configure option for best performance
[ml027.localdomain:mpi_rank_63][rdma_find_network_type] QLogic IB card detected in system
[many lines removed]
[ml017.localdomain:mpi_rank_21][rdma_find_network_type] Please re-configure the library with the '--with-device=ch3:psm' configure option for best performance
# OSU MPI Multiple Bandwidth / Message Rate Test v3.6
# [ pairs: 32 ] [ window size: 64 ]
# Size                  MB/s        Messages/s
1                     116.11      116111578.70
2                     270.93      135466560.35
4                     542.55      135637685.02
8                    1066.58      133321971.01
16                   2150.17      134385710.14
32                   4343.15      135723409.57
64                   8584.57      134133894.32
128                 16496.80      128881239.19
256                 31527.22      123153184.11
512                 57838.59      112965999.37
1024                95724.16       93480624.57
2048               171346.90       83665477.67
4096               259874.23       63445857.09
8192               345351.12       42157119.12
16384              351755.78       21469469.11
32768              264519.29        8072488.10
65536              331644.50        5060493.44
131072             356726.41        2721606.55
262144             337601.17        1287846.27
524288             331887.68         633025.51
1048576            334439.17         318946.04
2097152            334085.32         159304.30
4194304            313676.98          74786.42
(dog at ml005 13%) 

--
David Gunter
HPC-3: Infrastructure Team
Los Alamos National Laboratory