[mvapich-discuss] hang while trying to run MPI in a Heterogeneous network (One Node with 2 HCA and the other Node with a single HCA)

[Rajeev.c.p]

Hari,

Thanks for the quick response. 
Is there any other way to use both the HCA's in the Hetrogeneous network. Can we have 2 instance of the same process each bound to each of the HCA's and then send data to the Node2. So that still the outgoing BandWidth can be increased.
Also can we take advantage of the 2 HCA's in the Sender Node if we have 2 Nodes in the receiving side each with a single HCA?

IF the above is possible please help us with the mpi command line which will achieve the above functionality.

Thanks and Regards
Rajeev




On Thursday, February 27, 2014 1:22 PM, Hari Subramoni <subramoni.1 at osu.edu> wrote:
 
Hello Rajeev,


On heterogeneous networks, MVAPICH2 will fall back to the lowest 
common denominator in terms of number of HCA's (in this case, one HCA). 
So MVAPICH2 will not be able to take advantage of both HCAs on node #1 as you desire. You will have to use MV2_NUM_HCAS=1 MV2_IB_HCA=mlx4_0 to avoid the hang.


MVAPICH2-2.0b should not have any hangs / issues on heterogeneous networks.


All MPI calls in MVAPICH2 (including MPI_Send / MPI_Receive) will utilize RDMA capabilities of the underlying network if
  - 1. MVAPICH2 was configured to run on RDMA enabled networks (this is the default config) and

  - 2. You are running on a system with the appropriate hardware / software installed and working


Please let us know if you've any further questions / issues and we will be glad to help.

Regards,
Hari.




On Thu, Feb 27, 2014 at 3:27 PM, Rajeev.c.p <rajeevcp at yahoo.com> wrote:

Hi,
>
>
>We are facing issues while trying to run MPI in a hetrogeneous IB network. The configuration we have is given below
>We have 2 Linux Nodes
>Node1 with 2 FDR HCA's  = 192.168.1.4
>Node 2 with 1 FDR HCA  = 192.168.1.3  . All the HCA cards are from Mellanox and they are connected to a FDR IB switch.
>
>
>We are trying to run a bandwidth program using MPI_SEND and MPI_RECEIVE with the following command line
>/home/klac/mvapich2-2.2.0/bin/mpiexec -np 2 -hosts 192.168.1.4,192.168.1.3 -env MV2_NUM_HCAS=2,MV2_IB_HCA=mlx4_0:mlx4_1 ./RunMPIBWTest 1024 1024 . This hangs indefinitely and does not returns. Below is the last set of traces from the by enabling verbose
>
>
>[mpiexec at IMCRecipe00] [pgid: 0] got PMI command: cmd=barrier_in
>[proxy:0:1 at IMCNode001] got pmi command (from 4): barrier_in
>
>
>[proxy:0:1 at IMCNode001] forwarding command (cmd=barrier_in) upstream
>
>
>I have 3 questions 
>(1) Is the above command line correct to use both the HCA's to send Data to the client node so that we get increased bandwidth
>(2) There was earlier posts which was talking about hangs in the heterogeneous networks is that resolved with MVapich 2.2
>(3) Are the MV2_NUM_HCA and MV2_IB_HCA environment variables  applicable to MPI_SEND and MPI_RECEIVE command or does it work only with RDMA?
>
>
>Thanks and Regards
>Rajeev
>
>
>_______________________________________________
>mvapich-discuss mailing list
>mvapich-discuss at cse.ohio-state.edu
>http://mailman.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20140227/89b439a4/attachment-0001.html>