[Mvapich-discuss] Help with PSM2_MQ_RECVREQS_MAX and GATHERV

[Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC]]

Greetings MVAPICH List,

It was about time for my yearly "Try my code GEOS with MVAPICH2" (Hi Hari! :) ) and so I built up an MVAPICH2 2.3.6 stack. I encountered the same error as last time (dies in MPI_Put), but since then, one of our in-house MPI gurus created a different IO path that avoids the MPI_Put call. I try that and it works. Huzzah!

But at the end of a simulation day when our checkpoints write, I get:

Exhausted 1048576 MQ irecv request descriptors, which usually indicates a user program error or insufficient request descriptors (PSM2_MQ_RECVREQS_MAX=1048576)

Obviously I can do what it says in the message and up PSM2_MQ_RECVREQS_MAX (and I'm trying that now), but a bit of Googling around led to a page on the Intel MPI Forum:

https://urldefense.com/v3/__https://community.intel.com/t5/Intel-oneAPI-HPC-Toolkit/PSM2-MQ-RECVREQS-MAX-limit-reached/m-p/1087818__;!!KGKeukY!i1OtU098CxepUnnbIYeuybpbxu0e4yEPUxyQAfRTWNmKuLAYL7kB5y8OQlTQ5g2vlxoc5vNILQ$ 

where the proposed solution there was to use I_MPI_ADJUST_GATHERV=3. And guess what setting we have to run with when using Intel MPI? Yup, I_MPI_ADJUST_GATHERV=3!

So, while I'm looking at increasing PSM2_MQ_RECVREQS_MAX, I was wondering if there is a similar GATHERV tune for MVAPICH2? I don't see anything explicit in the User's Guide, but I figured I'd ask the experts here as maybe there is some setting that does something similar (or maybe a more "MVAPICH-y" way to change that PSM2 variable)?

Thanks,
Matt

--
Matt Thompson, SSAI, Ld Scientific Programmer/Analyst
NASA GSFC,    Global Modeling and Assimilation Office
Code 610.1,  8800 Greenbelt Rd,  Greenbelt,  MD 20771
Phone: 301-614-6712                 Fax: 301-614-6246
https://urldefense.com/v3/__http://science.gsfc.nasa.gov/sed/bio/matthew.thompson__;!!KGKeukY!i1OtU098CxepUnnbIYeuybpbxu0e4yEPUxyQAfRTWNmKuLAYL7kB5y8OQlTQ5g2vlxpFO49wjg$ 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osu.edu/pipermail/mvapich-discuss/attachments/20210622/071741ea/attachment-0021.html>