[mvapich-discuss] Listing of "hidden" MV2 environment variables?

[Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC]]

On 01/21/2015 03:23 PM, Hari Subramoni wrote:
> Hello Matt,
>
> Good to know that the problem got solved by using blocking mode. Would
> it be possible for you to give us a reproducer so that we can try to
> debug the problem locally?

Well, I've settled at just turning off the shared memory with 
MV2_USE_SHARED_MEM=0. It seems to have the least side effects 
(MV2_USE_SHARED_BLOCKING=0 seemed to change a few other things).

As for a reproducer, well...about the only reproducer we have is the 
model itself. I think MVAPICH2 has some developers on NCCS' system, if 
so, we could work on getting them running with it.

>
> We are continually working to expose the environment varibales in the
> MVAPICH2 code base through the MV2_SHOW_ENV_INFO option. We hope to have
> this option display all enviroment variables in one of our future releases.

Please do! The more you can expose, the better.

>
> Thx,
> Hari.
>
> On Tue, Jan 20, 2015 at 12:09 PM, Thompson, Matt (GSFC-610.1)[SCIENCE
> SYSTEMS AND APPLICATIONS INC] <matthew.thompson at nasa.gov
> <mailto:matthew.thompson at nasa.gov>> wrote:
>
>     All,
>
>     In my recent attempts to diagnose and workaround this issue [1]:
>
>         http://mailman.cse.ohio-state.__edu/pipermail/mvapich-discuss/__2015-January/005296.html
>         <http://mailman.cse.ohio-state.edu/pipermail/mvapich-discuss/2015-January/005296.html>
>
>
>     I've been twiddling various MV2_ environmental knobs by looking at
>     the output of MV2_SHOW_ENV_INFO=2 and I eventually realized that
>     that setting does not show all environment variables.
>
>     For example, it turns out that, at least for one case (the above 336
>     process one), MV2_USE_BLOCKING=1 seems to workaround the issue.
>
>     However, in a "stock option" run of MVAPICH2 2.1rc1--where
>     MV2_USE_BLOCKING=0 is the default setting--MV2_USE_BLOCKING does not
>     show up in the listing of MV2_SHOW_ENV_INFO=2. It's only shown if it
>     is toggled.
>
>     So, I was wondering, does anyone have a listing of what knobs are
>     turnable in MVAPICH2 (say in the User's Guide) but aren't included
>     in MV2_SHOW_ENV_INFO=2?
>
>     Or, perhaps, is there a higher value of MV2_SHOW_ENV_INFO that shows
>     them (a la I_MPI_DEBUG in Intel MPI)? I tried 3 but nothing really
>     happened save the =1 prints.
>
>     (Note: Some further testing seems to show that it's actually
>     MV2_USE_SHARED_MEM=0 set by MV2_USE_BLOCKING=1 that's helping. More
>     to test!)
>
>     Matt
>
>     [1] Update: looks like changing MV2_SMP_NUM_SEND_BUFFER and
>     MV2_EAGERSIZE_1SC[2] to their Westmere/DDR values do not always
>     solve the problem. On 336 cores, it seems to be more reliably
>     halting at all times.
>
>     [2] Does anyone know what MV2_EAGERSIZE_1SC does? It does not seem
>     to exist in the User's Guide.
>
>     --
>     Matt Thompson          SSAI, Sr Software Test Engr
>     NASA GSFC, Global Modeling and Assimilation Office
>     Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771
>     Phone: 301-614-6712 <tel:301-614-6712>              Fax:
>     301-614-6246 <tel:301-614-6246>
>     _________________________________________________
>     mvapich-discuss mailing list
>     mvapich-discuss at cse.ohio-__state.edu
>     <mailto:mvapich-discuss at cse.ohio-state.edu>
>     http://mailman.cse.ohio-state.__edu/mailman/listinfo/mvapich-__discuss
>     <http://mailman.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss>
>
>


-- 
Matt Thompson          SSAI, Sr Software Test Engr
NASA GSFC, Global Modeling and Assimilation Office
Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771
Phone: 301-614-6712              Fax: 301-614-6246