[mvapich-discuss] mvapich2 debug/verbose mode?

[Hari Subramoni]

Hi Mehmet,

For maximum debugging information and error checking, please configure
MVAPICH2 with --enable-g=all and --enable-fast=none and run with
MV2_DEBUG_SHOW_BACKTRACE=2.
This should dump very verbose debugging information like the nodes where
the failure occurred etc and core files (if applicable and allowed). If
reconfiguration is not an option, please try running with just the
environment variable.

Regards,
Hari.

On Mon, Dec 19, 2016 at 5:59 PM, Belgin, Mehmet <
mehmet.belgin at oit.gatech.edu> wrote:

> Dear all,
>
> We are trying to troubleshoot an application that crashes only with large
> number of cores (>1024) and after more than a day of runtime, which makes
> the process very difficult. Unfortunately we can't get a lot of insight
> from the error messages mvapich2 is generating. We are interested in things
> like which nodes pair(s) were involved in the failed message passing
> operation (similar to OpenMPI's crash messages).
>
> I've been looking at runtime options and found "MV2_DEBUG_SHOW_BACKTRACE".
> What other parameters would you recommend to maximize the verbosity of
> mvapich2 to see if we can catch any clues? We are using intel/15.0 with
> mvapich2/2.1, and the usual hydra launcher.
>
> I'll appreciate any suggestions you may have.
>
> Thank you in advance and happy holidays,
>
> -Mehmet
>
> =========================================
> Mehmet Belgin, Ph.D.
> Scientific Computing Consultant
> Partnership for an Advanced Computing Environment (PACE)
> Georgia Institute of Technology
> 258 4th Street NW, Rich Building, #326
> Atlanta, GA  30332-0700
> Office: (404) 385-0665
>
>
>
>
> _______________________________________________
> mvapich-discuss mailing list
> mvapich-discuss at cse.ohio-state.edu
> http://mailman.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20161219/35baa1cb/attachment-0001.html>