[mvapich-discuss] Options to let MPI use different HCAs or IB ports on different servers

[Hari Subramoni]

Hello,

I do not completely understand what you mean by "IB driver". If you mean
that you have multiple HCAs, and you want MVAPICH2 to utilize both of them,
then you can to use one of the following commands

mpirun_rsh -np 2 n0 n1 MV2_IBA_HCA=mlx5_0:mlx5_1 prog

mpirun_rsh -np 2 n0 n1 MV2_NUM_HCAS=2 prog

The equivalent of name "mlx5_0" for your system can be obtained from the
"ibstat" command

[subramon at haswell1 ~]$ ibstat
CA 'mlx5_0'
        CA type: MT4119
        Number of ports: 1
        Firmware version: 16.18.0160
<...snip...>

MVAPICH2 has several levels of support for hardware configurations that
have more than one InfiniBand end point (referred to as 'multi-rail'
configuration by MVAPICH2). For example, MVAPICH2 has support to use
multiple HCAs, multiple ports per HCA and multiple InfiniBand queue pairs
per port.

Our userguide has a lot of information about the various multi-rail modes
supported by MVAPICH2 and how to use them. I've put links to the relevant
sections of the userguide below. Please browse through them for detailed
information.

http://mvapich.cse.ohio-state.edu/static/media/mvapich/mvapich2-2.2-userguide.html#x1-660006.12
http://mvapich.cse.ohio-state.edu/static/media/mvapich/mvapich2-2.2-userguide.html#x1-670006.13

I hope I was able to address your concerns. Please feel to let me know if
you have any follow-up questions.

Regards,
Hari.

On Fri, Mar 3, 2017 at 11:42 PM, Junjie Qian <junjieqian at outlook.com> wrote:

> Hi List,
>
>
> I am new to MVAPICH2, and have some questions regarding using options to
> run.
>
>
> The issue is that, there are two servers, each server has two IB ports
> with NUMA and the IB driver version may be different. Are there options to
> make MPI run on different server knows which IB driver version and port
> should be used on different server.
>
>
> My current choice is, specify both IB driver versions with
> (MV2_IBA_HCA=driver_0 and MV2_IBA_HCA=driver_1) and ignoring "
> MV2_DEFAULT_PORT" in the MPI command.
>
>
> Is there any suggestions on this?
>
> Thank you
>
> Best
>
> Junjie Qian
>
> _______________________________________________
> mvapich-discuss mailing list
> mvapich-discuss at cse.ohio-state.edu
> http://mailman.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20170304/97200bee/attachment-0001.html>