[mvapich-discuss] Specifying multiple HCAs in hostfile

[Dalai, Santak]

Yes, that’s what I have been doing till now. In my case, I am using many processes and hence specifying the list of HCAs in the command-line crowds it. I was thinking if I can do that using a hostfile and keep the command-line clean and readable.

Thanks,
Santak Dalai

From: Jonathan Perkins [mailto:perkinjo at cse.ohio-state.edu]
Sent: Tuesday, March 10, 2015 6:02 PM
To: Dalai, Santak; mvapich-discuss at cse.ohio-state.edu
Subject: Re: [mvapich-discuss] Specifying multiple HCAs in hostfile

I'm sorry, but at this time the hostfile format doesn't allow for multiple HCAs to be specified.  Perhaps you can try using MPMD if you need to have each process to have distinct HCAs specified by having the executable be something like "env MV2_IBA_HCA=hcalist ./mpi-app".

Please look for Job Launch Using MPMD section available at http://mvapich.cse.ohio-state.edu/static/media/mvapich/mvapich2-2.1rc1-userguide.html#x1-240005.2.1.

I hope this helps.  Let us know if you have any further questions.

On Tue, Mar 10, 2015 at 8:14 AM Dalai, Santak <Santak.Dalai at kla-tencor.com<mailto:Santak.Dalai at kla-tencor.com>> wrote:
Hello all,

I saw the format of hostfile in the userguide and it says a way to specify the host, the ranks and the HCA.

Hostfile Format
The mpirun_rsh hostfile format allows for users to specify hostnames, one per line, optionally with a multiplier, and HCA specification.
The multiplier allows you to save typing by allowing you to specify blocked distribution of MPI ranks using one line per hostname. The HCA specification allows you to force an MPI rank to use a particular HCA.
The optional components are delimited by a ‘:’. Comments and empty lines are also allowed. Comments start with ‘#’ and continue to the next newline.

Sample hostfile

$ cat hosts
# sample hostfile for mpirun_rsh
host1           # rank 0 will be placed on host1
host2:2         # rank 1 and 2 will be placed on host 2
host3:hca1      # rank 3 will be on host3 and will used hca1
host4:4:hca2    # ranks 4 through 7 will be on host4 and use hca2

# if the number of processes specified for this job is greater than 8
# then the additional ranks will be assigned to the hosts in a cyclic
# fashion.  For example, rank 8 will be on host1 and ranks 9 and 10 will
# be on host2.


Is there a way to specify multiple HCAs for a rank? Let say for the above example I want to run rank3 on host3 with2 hcas-hca1 and hca2. How to achieve it?

Thanks,
Santak Dalai

_______________________________________________
mvapich-discuss mailing list
mvapich-discuss at cse.ohio-state.edu<mailto:mvapich-discuss at cse.ohio-state.edu>
http://mailman.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20150310/ad7ee1db/attachment-0001.html>