[mvapich-discuss] mvapich2 2.3a on ppc64le

[Sourav Chakraborty]

Hi Jamil,

The combination MV2_CPU_BINDING_LEVEL=numanode
MV2_CPU_BINDING_POLICY=scatter is not currently supported.

If you want to bind the processes to individual cores, you can use
MV2_CPU_BINDING_LEVEL=core
MV2_CPU_BINDING_POLICY=scatter

If you want to bind the processes to numanodes, you can specify it in the
following fashion: MV2_CPU_MAPPING=0-79:80-159
MV2_CPU_BINDING_POLICY=scatter
This setting will work for 2 processes. For more processes, you can modify
the mapping according to the userguide:
http://mvapich.cse.ohio-state.edu/static/media/mvapich/mvapich2-2.3a-userguide.html#x1-620006.5.2

Thanks,
Sourav



On Thu, Jun 29, 2017 at 8:07 AM, Hari Subramoni <subramoni.1 at osu.edu> wrote:

> Thanks for getting back quickly. It is surprising that you are facing the
> registration cache issue with osu microbenchmarks.
>
> We will take a look at it and get back to you shortly.
>
> Thx,
> Hari.
>
> On Thu, Jun 29, 2017 at 8:03 AM, Jamil Appa <jamil.appa at zenotech.com>
> wrote:
>
>> Hi Hari
>>
>>  Thanks for the quick reply. I get this error running any of the osu
>> benchmarks that are distributed with mvapich2 as well as my own application.
>>
>>   The applications work if I have 4 threads per core so a total 80
>> threads per node.  It looks like there is a limit to the maximum number of
>> threads fixed by the size of the mask.
>>
>>  Let me know if you want me to run with different switches to generate
>> more output.
>>
>>  Jamil
>>
>>
>> On Thu, 29 Jun 2017 at 12:49 Hari Subramoni <subramoni.1 at osu.edu> wrote:
>>
>>> Hello,
>>>
>>> Sorry to hear that you are facing issues. These are two separate issues
>>> actually. Could you please let us know what program you are running? That
>>> will help us narrow the issue down further.
>>>
>>> Thx,
>>> Hari.
>>>
>>>
>>> On Jun 29, 2017 6:49 AM, "Jamil Appa" <jamil.appa at zenotech.com> wrote:
>>>
>>> Hi
>>>
>>>     I am trying to use mvapich2 2.3a on a 2 node ppc64le system with 8
>>> threads per core (total threads per node of 160)
>>>
>>>     There appears to be a bug in ptmalloc that prevents correct startup
>>> related to setting the affinity.
>>>
>>>    WARNING: Error in initializing MVAPICH2 ptmalloc library.Continuing
>>> without InfiniBand registration cache support.
>>> Warning! : Core id -1 does not exist on this architecture!
>>> CPU Affinity is undefined
>>> Error parsing CPU mapping string
>>> INTERNAL ERROR: invalid error code ffffffff (Ring Index out of range) in
>>> MPIDI_CH3I_set_affinity:2673
>>> Warning! : Core id -1 does not exist on this architecture!
>>> CPU Affinity is undefined
>>> Error parsing CPU mapping string
>>> INTERNAL ERROR: invalid error code ffffffff (Ring Index out of range) in
>>> MPIDI_CH3I_set_affinity:2673
>>> [cli_2]: aborting job:
>>> Fatal error in PMPI_Init_thread:
>>> Other MPI error, error stack:
>>> MPIR_Init_thread(490):
>>> MPID_Init(386).......:
>>>
>>> [cli_0]: aborting job:
>>> Fatal error in PMPI_Init_thread:
>>> Other MPI error, error stack:
>>> MPIR_Init_thread(490):
>>> MPID_Init(386).......:
>>>
>>>  mpiexec -env MV2_NUM_HCAS 1 -env MV2_NUM_PORTS 1 -env
>>> MV2_USE_THREAD_WARNING 0 -env MV2_SHOW_HCA_BINDING 0 -env
>>> MV2_CPU_BINDING_LEVEL numanode -env MV2_CPU_BINDING_POLICY scatter
>>>
>>>   cat /etc/redhat-release
>>> Red Hat Enterprise Linux Server release 7.2 (Maipo)
>>>
>>>  uname -a
>>>
>>>  Linux nux gpu02.cluster 3.10.0-327.el7.ppc64le #1 SMP Thu Oct 29
>>> 17:31:13 EDT 2015 ppc64le ppc64le ppc64le GNU/Linux
>>>
>>>
>>>
>>>
>>> *Jamil Appa* | Co-Founder and Director | Zenotech
>>> [image: Papercut]
>>> [image: Tel:] +44 (0)7747 606 788 <+44%207747%20606788> [image:
>>> Zenotech LTD - Simulation Unlimited] <http://www.zenotech.com/>
>>> [image: Email:] jamil.appa at zenotech.com
>>> [image: Web:] www.zenotech.com
>>> [image: Papercut]
>>> [image: linkedin:] <http://uk.linkedin.com/pub/jamil-appa/1/165/120>[image:
>>> Twitter:] <https://twitter.com/zenotech>[image: Location:]
>>> <https://www.google.co.uk/maps/place/Bristol+%26+Bath+Science+Park/@51.500921,-2.478567,17z/data=!3m1!4b1!4m2!3m1!1s0x48719ab86a5a9f7d:0xd17394f3400abb0a>
>>>
>>> Company Registration No : 07926926 | VAT No : 128198591
>>>
>>> Registered Office : 1 Larkfield Grove, Chepstow, Monmouthshire, NP16
>>> 5UF, UK
>>>
>>> Address : Bristol & Bath Science Park, Dirac Cres, Emersons Green,
>>> Bristol BS16 7FR
>>>
>>>
>>> _______________________________________________
>>> mvapich-discuss mailing list
>>> mvapich-discuss at cse.ohio-state.edu
>>> http://mailman.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>>>
>>>
>>>
>
> _______________________________________________
> mvapich-discuss mailing list
> mvapich-discuss at cse.ohio-state.edu
> http://mailman.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20170629/b4ba0512/attachment-0001.html>