[mvapich-discuss] Error in initializing MVAPICH2 ptmalloc library with mpi4py

[Subramoni, Hari]

Hi, Jonatan.

Glad to hear that you’re able to run successfully with the mpiexec launcher.

Could you please let us know if the original performance issues have gone away now?

In order to improve the job startup performance and make the system administrators aware of how this can be done, we had added the following warning in the code if MVAPICH2 is being run on a homogeneous (in terms of processor and HCA type) cluster without setting MV2_HOMOGENEOUS_CLUSTER=1.

[c-206-4:mpi_rank_0][rdma_param_handle_heterogeneity] All nodes involved in the job were detected to be homogeneous in terms of processors and interconnects. Setting MV2_HOMOGENEOUS_CLUSTER=1 can improve job startup performance on such systems. The following link has more details on enhancing job startup performance. http://mvapich.cse.ohio-state.edu/performance/job-startup/.

Setting that environment variable to "1" allows MVAPICH2 to skip certain message exchanges to identify heterogeneity (in terms of CPU, Network Adapter etc) in HPC clusters so that the library uses the least common denominator. This is set by default to 0 so that we work on all systems.

I hope this explanation clarifies the situation.

Best,
Hari.

From: Jonatan Martín <martin at ice.csic.es>
Sent: Tuesday, December 15, 2020 11:52 AM
To: Subramoni, Hari <subramoni.1 at osu.edu>
Cc: mvapich-discuss at cse.ohio-state.edu <mvapich-discuss at mailman.cse.ohio-state.edu>
Subject: Re: [mvapich-discuss] Error in initializing MVAPICH2 ptmalloc library with mpi4py

Hi Hari,

I've tried with the following script and I could run it without warnings finally.

#!/bin/bash
#SBATCH -N 2
LD_PRELOAD=<full path to libmpi.so> mpiexec -f hosts -n 40 -env MV2_HOMOGENEOUS_CLUSTER=1 python myprogram.py

Thanks for your help! As a final question, what's the impact of setting MV2_HOMOGENEOUS_CLUSTER=1?

Regards,

Jonatan

Missatge de Subramoni, Hari <subramoni.1 at osu.edu<mailto:subramoni.1 at osu.edu>> del dia dt., 15 de des. 2020 a les 16:15:
Hi, Jonatan.

There could be some firewall issue preventing mpirun_rsh from launching on multiple nodes. Can you please try to launch with mpiexec instead to see if it works?

mpiexec -n <num_procs> -f <hostfile> -env <env vars if any> <executable>

If you would like to launch with srun, then yes, you should reconfigure MVAPICH2 with --with-pm=slurm and --with-pmi=pmix and provide the pmi option as you mentioned when launching.

Thx,
Hari.

From: Jonatan Martín <martin at ice.csic.es<mailto:martin at ice.csic.es>>
Sent: Tuesday, December 15, 2020 6:30 AM
To: Subramoni, Hari <subramoni.1 at osu.edu<mailto:subramoni.1 at osu.edu>>
Cc: mvapich-discuss at cse.ohio-state.edu<mailto:mvapich-discuss at cse.ohio-state.edu> <mvapich-discuss at mailman.cse.ohio-state.edu<mailto:mvapich-discuss at mailman.cse.ohio-state.edu>>
Subject: Re: [mvapich-discuss] Error in initializing MVAPICH2 ptmalloc library with mpi4py

Hi Hari,

Thank you very much for your fast reply. I did both tests, but I couldn't solve the problem. Actually, I realized that the code gets stuck at some point in a loop where there is exchange of information between CPUs. It's really weird. Let me show you some examples.

We use the queue system Slurm to run our jobs. The process manager interface is PMIx. If I use the script:

#!/bin/bash
#SBATCH -n 40
#SBATCH - N 1
mpirun python myprogram.py

The program starts and finishes well, but with the following warnings:

- WARNING: Error in initializing MVAPICH2 ptmalloc library. Continuing without InfiniBand registration cache support.
- [hidra1:mpi_rank_0][rdma_param_handle_heterogeneity] All nodes involved in the job were detected to be homogeneous in terms of processors and interconnects. Setting MV2_HOMOGENEOUS_CLUSTER=1 can improve job startup performance on such systems.

The first warning seems to be solved when we set “LD_PRELOAD=<full path to libmpi.so>” and the second by setting "MV2_HOMOGENEOUS_CLUSTER=1".

But when I want to use more than one node (let's say 2),

#!/bin/bash
#SBATCH - N 2
mpirun_rsh -hostfile hosts -n 40 python myprogram.py

Then the program gets stuck. I'm wondering if we should run the code with srun and install mvapich2 with another configuration different from the default one (maybe with --with-pm=slurm and --with-pmi=pmix). We tried before, but we couldn't manage to run the code. Do you know if we use srun we must give explicitly the option --pmi=pmix?

Thank you again and regards,

Jonatan


Missatge de Subramoni, Hari <subramoni.1 at osu.edu<mailto:subramoni.1 at osu.edu>> del dia dl., 14 de des. 2020 a les 17:37:
Hi, Jonatan.

Can you please try the following options one by one and let us know what performance you obseve?


  1.  Add “LD_PRELOAD=<full path to libmpi.so>” before your launch command

     *   E.g. LD_PRELOAD=/home/subramon/MVAPICH2/git/ install-dir/lib/libmpi.so mpirun_rsh -np xx -hostfile <hostfile> <path_to_executable>

  1.  Set MV2_VBUF_TOTAL_SIZE & MV2_IBA_EAGER_THRESHOLD to 128 KB

Thx,
Hari.

From: mvapich-discuss-bounces at cse.ohio-state.edu<mailto:mvapich-discuss-bounces at cse.ohio-state.edu> <mvapich-discuss-bounces at mailman.cse.ohio-state.edu<mailto:mvapich-discuss-bounces at mailman.cse.ohio-state.edu>> On Behalf Of Jonatan Martín
Sent: Monday, December 14, 2020 10:59 AM
To: mvapich-discuss at cse.ohio-state.edu<mailto:mvapich-discuss at cse.ohio-state.edu> <mvapich-discuss at mailman.cse.ohio-state.edu<mailto:mvapich-discuss at mailman.cse.ohio-state.edu>>
Subject: [mvapich-discuss] Error in initializing MVAPICH2 ptmalloc library with mpi4py

Hi,

I'm trying to run a Python code with mvapich2 2.3.5 with multiple nodes, but I get the following warning:

WARNING: Error in initializing MVAPICH2 ptmalloc library. Continuing without InfiniBand registration cache support.

The execution goes on, but with poor performance. I've read in another discussion (http://mailman.cse.ohio-state.edu/pipermail/mvapich-discuss/2015-April/005532.html) that this issue was supposed to be solved in the version 2.2. I wanted to ask if this issue was already solved or it is really a problem with the configuration of mvapich2 in the cluster I'm using. Can you confirm this?

Also it is stated that you can modify the variables "MV2_VBUF_TOTAL_SIZE" and "MV2_IBA_EAGER_THRESHOLD" to get better performance, but I'm not sure how to tune these variables. What's the maximum recommended size you can use there? Or how do you know the optimal size?

Thank you very much and regards,

Jonatan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20201215/3551b622/attachment-0001.html>