[mvapich-discuss] mvapich2 debug/verbose mode?

[Belgin, Mehmet]

Hi Hari,

Thank you for your suggestion. Do you think it makes a difference to to install a debug version of mvapich2, then use it to launch the code/dependencies that are NOT compiled with that stack? This code has a lot of dependencies and recompilation is a really messy task (but it’s possible, of course).

Thanks a lot again!

-Memo

=========================================
Mehmet Belgin, Ph.D.
Scientific Computing Consultant
Partnership for an Advanced Computing Environment (PACE)
Georgia Institute of Technology
258 4th Street NW, Rich Building, #326
Atlanta, GA  30332-0700
Office: (404) 385-0665



On Dec 19, 2016, at 6:08 PM, Hari Subramoni <subramoni.1 at osu.edu<mailto:subramoni.1 at osu.edu>> wrote:

Hi Mehmet,

For maximum debugging information and error checking, please configure MVAPICH2 with --enable-g=all and --enable-fast=none and run with MV2_DEBUG_SHOW_BACKTRACE=2. This should dump very verbose debugging information like the nodes where the failure occurred etc and core files (if applicable and allowed). If reconfiguration is not an option, please try running with just the environment variable.

Regards,
Hari.

On Mon, Dec 19, 2016 at 5:59 PM, Belgin, Mehmet <mehmet.belgin at oit.gatech.edu<mailto:mehmet.belgin at oit.gatech.edu>> wrote:
Dear all,

We are trying to troubleshoot an application that crashes only with large number of cores (>1024) and after more than a day of runtime, which makes the process very difficult. Unfortunately we can't get a lot of insight from the error messages mvapich2 is generating. We are interested in things like which nodes pair(s) were involved in the failed message passing operation (similar to OpenMPI's crash messages).

I've been looking at runtime options and found "MV2_DEBUG_SHOW_BACKTRACE". What other parameters would you recommend to maximize the verbosity of mvapich2 to see if we can catch any clues? We are using intel/15.0 with mvapich2/2.1, and the usual hydra launcher.

I'll appreciate any suggestions you may have.

Thank you in advance and happy holidays,

-Mehmet

=========================================
Mehmet Belgin, Ph.D.
Scientific Computing Consultant
Partnership for an Advanced Computing Environment (PACE)
Georgia Institute of Technology
258 4th Street NW, Rich Building, #326
Atlanta, GA  30332-0700
Office: (404) 385-0665<tel:(404)%20385-0665>




_______________________________________________
mvapich-discuss mailing list
mvapich-discuss at cse.ohio-state.edu<mailto:mvapich-discuss at cse.ohio-state.edu>
http://mailman.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20161219/a5668103/attachment.html>