[mvapich-discuss] Application aborts at runtime.

[Hari Subramoni]

Hi Chaitra,

I dug a little deeper into this issue. As you may know MVAPICH2 is a
derivative of MPICH i.e - we inherit some of the higher level code from
MPICH.

This issue looks to come from higher MPICH level code. Although the upper
bound for the tag is initialized to be INT_MAX, MPICH reserves some tag
space for internal use.

With mpich-3.1.4, I see the following in File: "src/mpi/init/initthread.c".
This gives you a max value of  536870911 that users can specify at the
application level with threaded support. MVAPICH2 inherits this limit.

503     /* Set aside tag space for tagged collectives and failure
notification */
504     MPIR_Process.attrs.tag_ub     >>= 2;

With mpich-3.2b2, I see the following in File: "src/mpi/init/initthread.c".
This gives you a max value of  268435455 that users can specify at the
application level with threaded support.

503     /* Set aside tag space for tagged collectives and failure
notification */
504     MPIR_Process.attrs.tag_ub     >>= 3;

The reason why your application passes with OpenMPI could be because
they've a higher upper limit for the tag.

In this context, I would recommend that you follow up with the MPICH team
at "discuss at mpich.org".

In the short term, I would recommend modifying your application to use
smaller tags to make progress with MPICH derivatives.

Regards,
Hari.

On Thu, Jun 4, 2015 at 4:44 AM, Chaitra Kumar <chaitragkumar at gmail.com>
wrote:

> Hi Hari,
>
> I installed MVAPICH2-2.1 with TCP/IP nemesis channel.  Even with this
> release the problem persists.
>
> Is there any other way to solve this problem?
>
> Regards,
> Chaitra
>
> On Thu, Jun 4, 2015 at 8:18 AM, Hari Subramoni <subramoni.1 at osu.edu>
> wrote:
>
>> Hi Chaitra,
>>
>> It seems like you are using the socket channel here. In this case, you
>> can directly use the TCP/IP nemesis channel.
>>
>> Please refer to the following section of the userguide for more
>> information.
>>
>>
>> http://mvapich.cse.ohio-state.edu/static/media/mvapich/mvapich2-2.1-userguide.html#x1-170004.9
>>
>> Alternately, you can also try to use MPICH directly.
>>
>> On a side note, can you please try to use MVAPICH2-2.1 and see if the
>> error persists?
>>
>> Regards,
>> Hari.
>>
>> On Wed, Jun 3, 2015 at 8:56 PM, Chaitra Kumar <chaitragkumar at gmail.com>
>> wrote:
>>
>>> Hi Hari,
>>>
>>> Thanks for your reply.
>>>
>>> Below is the output of mpiname -a
>>> MVAPICH2 2.0 Fri Jun 20 20:00:00 EDT 2014 ch3:sock
>>>
>>> Compilation
>>> CC: gcc    -DNDEBUG -DNVALGRIND -O2
>>> CXX: g++   -DNDEBUG -DNVALGRIND
>>> F77: gfortran   -O2
>>> FC: gfortran
>>>
>>> Configuration
>>> --prefix=/home/chaitra/mvapich/mvapich2.2 --with-device=ch3:sock
>>> --enable-cxx --enable-threads=multiple --disable-mcast
>>>
>>> I am trying to run a multithreaded application using MVAPICH2. The
>>> application tries to traverse a huge graph.  Is it possible to increase the
>>> INT_MAX limit?
>>>
>>> The application runs without any errors using openmpi.
>>>
>>> Please let me know if you need more information.
>>>
>>> Regards,
>>> Chaitra
>>>
>>>
>>> On Thu, Jun 4, 2015 at 1:10 AM, Hari Subramoni <subramoni.1 at osu.edu>
>>> wrote:
>>>
>>>> Hello Chaitra,
>>>>
>>>> From the error, it looks like the value of tag you're passing to
>>>> MPI_Isend is incorrect, possibly its < 0 or > INT_MAX (as defined on your
>>>> system). However, without more information on how you configured the MPI
>>>> library (output of mpiname -a), some more details of the application, it
>>>> will be hard to give an exact answer.
>>>>
>>>> Regards,
>>>> Hari.
>>>>
>>>> On Wed, Jun 3, 2015 at 10:23 AM, Chaitra Kumar <chaitragkumar at gmail.com
>>>> > wrote:
>>>>
>>>>> Hi Team,
>>>>>
>>>>> I am trying to run an application using MVAPICH2-2.0.  The application
>>>>> compiles without any error, but at runtime, aborts with following error:
>>>>>
>>>>> Fatal error in PMPI_Isend:Invalid tag, error stack:
>>>>> PMPI_Isend(149): MPI_Isend(buf=0x7f59b5124300, count=8,
>>>>> MPI_UNSIGNED_LONG, dest=0, tag=1073741824, comm=0x84000001,
>>>>> request=0x132ae24) failed
>>>>> PMPI_Isend(98).: Invalid tag, value is 1073741824
>>>>>
>>>>> Can you please help me in resolving this.
>>>>>
>>>>> Thanks in advance.
>>>>>
>>>>> Regards,
>>>>> Chaitra
>>>>>
>>>>> _______________________________________________
>>>>> mvapich-discuss mailing list
>>>>> mvapich-discuss at cse.ohio-state.edu
>>>>> http://mailman.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20150604/95bf7ca7/attachment.html>