[mvapich-discuss] [SPAM] Maximum number of pending MPI_Isend calls? What signal is thrown by INTERNAL ERRORs?

[Devendar Bureddy]

Hi Tim

On Thu, Mar 24, 2011 at 10:09 AM, Tim Hartley <poobahtim at gmail.com> wrote:

> Hi,
>
> I'm curious if there is a hard or soft limit on the number of pending
> MPI_Isend calls, either from a single MPI process, or to a single MPI
> process.
>
> I am running a multithreaded test application on 32 nodes of the OSU
> BMI department. We use middleware called DataCutter which is MPI-aware
> (so only one thread - the "runtime" thread - does any MPI calls). I'm
> fiddling around quite a bit and I am able to get the following fail
> messages:
>
>
> Fatal error in MPI_Recv: Other MPI error
> [cli_32]: aborting job:
> Fatal error in MPI_Recv: Other MPI error
> INTERNAL ERROR: Invalid error class (66) encountered while returning from
> MPI_Test.  Please file a bug report.
> Fatal error in MPI_Test: Unknown error.  Please file a bug report., error
> stack:
> (unknown)(): connection failure
> [cli_31]: aborting job:
> Fatal error in MPI_Test: Unknown error.  Please file a bug report., error
> stack:
> (unknown)(): connection failure
>
> etc.
>
> However, if I simply throttle the number of pending Isends, this
> behavior goes away. This absolutely may be a bug in my application
> code, but I haven't been able to get a corefile or catch the signal
> with a signal handler or an already-attached gdb.
>
> What signal will this kind of error throw?
>


As this is a fatal error,  it calls exit() to terminate the process. If you
would like get a core file, put a break point at "MPID_Abort",
"MPIDI_CH3_Abort".  You can run "gcore" under gdb when it break at these
functions.

It looks like you are running out of request handles when running Isends
with out polling.


-Devendar

>
> Thanks a lot,
> Tim
> _______________________________________________
> mvapich-discuss mailing list
> mvapich-discuss at cse.ohio-state.edu
> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20110324/86a40985/attachment.html