[mvapich-discuss] MVAPITCH environment variables for preconnecting paths and for process-to-core mappings

[Subramoni, Hari]

Dear, Tom.

Clarification on Question 1: To the best of my knowledge, none of the MPI libraries you mentioned (CrayMPI, IntelMPI, HPE/SGI MPT) "precompute communication paths between different ranks ahead of time". The environment variables you have mentioned here is one that controls when a process creates a network end point to communicate with a peer process (either at job startup or when a process really wants to send a message to a peer process). This feature is typically applicable only for communication operations that rely on the network interconnect. The corresponding environment variable for MVAPICH2, as you have correctly noted is MV2_ON_DEMAND_THRESHOLD.

The following section of the MVAPICH2 user guide has more details on this parameter

http://mvapich.cse.ohio-state.edu/static/media/mvapich/mvapich2-2.3.4-userguide.html#x1-22500011.43

This threshold also has an impact on speed at which jobs can be started. The following link has more details about the impact MV2_ON_DEMAND_THRESHOLD has on the performance of job startup.

http://mvapich.cse.ohio-state.edu/performance/job-startup/

MVAPICH2 has several environment variables to control how processes are bound to physical cores. There are higher-level/abstract setting that allow you to distribute processes automatically. The environment variable to manually specify the process to core binding policy is MV2_CPU_MAPPING (e.g. MV2_CPU_MAPPING=0:1:2:...). You can use MV2_SHOW_CPU_BINDING environment variable to make display the process to core binding policy that was actually chosen for the execution.

These details are covered in slides #31 - #38 of the MVAPICH2 tutorial (link below) that we conducted as part of the Annual MVAPICH User Group (MUG) Meeting (http://mug.mvapich.cse.ohio-state.edu/).

http://mug.mvapich.cse.ohio-state.edu/static/media/mug/presentations/19/subramoni-mug-19.pdf

The following section of the MVAPICH2 user guide also has more details on how to map processes to cores.

http://mvapich.cse.ohio-state.edu/static/media/mvapich/mvapich2-2.3.4-userguide.html#x1-640006.5

Please let us know if you have any questions about MVAPICH2 and we will be glad to assist.

Best,
Hari.

-----Original Message-----
From: mvapich-discuss-bounces at cse.ohio-state.edu <mvapich-discuss-bounces at mailman.cse.ohio-state.edu> On Behalf Of Oppe, Thomas C ERDC-RDE-ITL-MS Contractor
Sent: Friday, July 24, 2020 10:51 AM
To: mvapich-discuss at cse.ohio-state.edu <mvapich-discuss at mailman.cse.ohio-state.edu>
Subject: [mvapich-discuss] MVAPITCH environment variables for preconnecting paths and for process-to-core mappings

Dear Sir:

I am working with MVAPITCH2 libraries on a Cray CS500 with 64 1.5 GHz AMD EPYC 7542 Rome cores per node, 2 sockets per node, 32 cores per socket.

Question 1:  Is there an environment variable for pre-computing all possible pairwise paths from each rank to every other rank rather than computing the paths dynamically as needed when a communication request is made?  Many other MPI implementations have this feature, such as

Intel MPI:  export I_MPI_DYNAMIC_CONNECTION=0

Cray MPICH:  export MPICH_GNI_DYNAMIC_CONN=disabled

HPE/SGI MPT:  export MPI_CONNECTIONS_THRESHOLD=<val>, where <val> is greater than the number of ranks being used.

Open MPI:  export OMPI_MCA_mpi_preconnect_mpi=1

I think the variable for MVAPITCH may be

export MV2_ON_DEMAND_THRESHOLD=<val>, where <val> is greater than the number of ranks being used.  Is that correct?

Question 2:  Is there an MVAPITCH environment variable for a process-to-core placement mapping?  For example, if I want to space out the processes on a node in the case that a node is only 1/4 populated, say putting 16 processes on a 64-core node, I can do this:

Intel MPI:  export I_MPI_PIN_PROCESSOR_LIST="shift=4"

Cray MPICH:   aprun -d 4 -n <procs>

HPE/SGI MPT:  export MPI_DSM_CPULIST="0-60/4:allhosts"   or
                           export MPI_DSM_CPULIST="0,4,8,12,16,...,60:allhosts"

Open MPI:  not sure, maybe
export OMPI_MCA_rmaps_base_mapping_policy=ppr:16:node:PE=1
export OMPI_MCA_rmaps_base_n_pernode=16

What is the MVAPITCH environment variable to process pinning and process-to-core mapping?

Thank you for any information.

Tom Oppe

-----------------------------------------------------------------------------
Thomas C. Oppe
HPCMP Benchmarking Team
HITS Team SAIC
Thomas.C.Oppe at erdc.dren.mil
Work:  (601) 634-2797
Cell:    (601) 642-6391
-----------------------------------------------------------------------------

_______________________________________________
mvapich-discuss mailing list
mvapich-discuss at cse.ohio-state.edu
http://mailman.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss