Why do I need MV2_CPU_MAPPING? (was :Re: [mvapich-discuss] MV2_CPU_MAPPING doesn't work on all the nodes)

[Devendar Bureddy]

Hi Craig

On Mon, Jun 25, 2012 at 3:13 PM, Craig Tierney <craig.tierney at noaa.gov> wrote:
> On 6/25/12 12:55 PM, Devendar Bureddy wrote:
>>
>> Hi Craig
>>
>> We also do not recommend to use MV2_CPU_MAPPING explicitly until
>> unless if user want some specific layout of cpu affinity across all
>> the nodes and do not want to modify hostfile.  MVAPICH2 also
>> automatically decides the affinity internally and users should not
>> worry much about it. There is also some high level abstract run-time
>> options (MV2_CPU_BINDING_POLICY,  MV2_CPU_BINDING_LEVEL)  to change
>> the default affinity settings if required. There is certainly no need
>> to explicitly specify affinity with MV2_CPU_MAPPING
>>
>> -Devendar
>>
>
> Devnedar,
>
> Thanks for the quick response.  I am currently building mvapich-1.8 for a
> new Sandy Bridge cluster and will be testing all this stuff out again.  We
> don't have many hybrid-codes, but I could never leave it up to mvapich2
> (from 1.4 to 1.6 era) and get good performance.  I had to write my own
> wrapper script that got the processor and memory affinity correct.
>
> When was MV2_CPU_BINDING_LEVEL added?  I am now starting to build version
> 1.8 for the new cluster.  We will backport the build to our older systems.
>  It seems that the solution to my issue is by setting MV2_CPU_BINDING_LEVEL
> to socket or numanode.

We enhanced CPU affinity options in 1.8 release.  Please find more
details in user guide section :
http://mvapich.cse.ohio-state.edu/support/user_guide_mvapich2-1.8.html#x1-520006.3
. Let us know if you see any issues

- Devendar

>
> So it seems that you fixed my problem before I asked it!
>  MV2_CPU_BINDING_LEVEL seems to be the answer.
>
> Thanks,
> Craig
>
>
>
>> On Mon, Jun 25, 2012 at 1:45 PM, Craig Tierney<craig.tierney at noaa.gov>
>>  wrote:
>>>
>>> Mvapich2 development team,
>>>
>>> This thread got me thinking again regarding one issue I have always had
>>> with
>>> process affinity and mvapich2.  Why do even need to use MV2_CPU_MAPPING?
>>>  For 99.9% of the cases that I could ever think of for MPI applications,
>>> there is one correct way to layout processes.  If I have the machine
>>> file, I
>>> would know what to do.  If I have OMP_NUM_THREADS, I would know what to
>>> do.
>>>  If for some reason I need a different mapping on different hosts, I
>>> would
>>> be able to deduce the correct layout from the machine file (which we do
>>> need
>>> to do and MV2_CPU_MAPPING cannot support this).
>>>
>>> The only rule I can think of is distribute process evenly across sockets,
>>> specify memory affinity should try and stay local.  The only other choice
>>> there is should you use a scatter or compact distribution.  That could be
>>> specified by a variable.
>>>
>>> Of course there are many things that could complicate my "simple" view of
>>> the world.  Besides launching multiple jobs on a node, what are they and
>>> how
>>> often do they really happen? For multiple jobs, you would probably
>>> through
>>> out affinity because there would be no consistent way by a user to ensure
>>> that two jobs are not stepping on each other (you would need a batch
>>> system
>>> to do this).
>>>
>>> Other MPI stacks do not require the use of something like MV2_CPU_MAPPING
>>> (Intel MPI and SGI MPI have methods built in).  Honestly most of my users
>>> wouldn't get the use of MV2_CPU_MAPPING correct.  This is something the
>>> system should be doing for them and not figure it out for themselves.
>>>
>>> Thanks,
>>> Craig
>>> _______________________________________________
>>> mvapich-discuss mailing list
>>> mvapich-discuss at cse.ohio-state.edu
>>> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>>
>>
>>
>>
>



-- 
Devendar