[mvapich-discuss] MVAPICH error question

[Karl Schulz]

Martin,

Also, just FYI there is a newer version of MVAPICH2 available on Ranger.
If you want to try it out as part of your threading notes, let us know and
I can point you to the module.

Thanks,

-karl

On 12/6/10 12:23 PM, "Jonathan Perkins" <perkinjo at cse.ohio-state.edu>
wrote:

>Hi Martin.
>
>Below is a snippet of documentation related to MPI_Init_thread.
>http://www.mpi-forum.org/docs/mpi22-report/node260.htm#Node260
>
>int MPI_Init_thread(int *argc, char *((*argv)[]), int required, int
>*provided)
>
>MPI_THREAD_SINGLE
>Only one thread will execute.
>
>MPI_THREAD_FUNNELED
>The process may be multi-threaded, but the application must
>ensure that only the main thread makes MPI calls (for the denition of
>main thread,
>see MPI_IS_THREAD_MAIN on page 386).
>
>MPI_THREAD_SERIALIZED
>The process may be multi-threaded, and multiple threads may
>make MPI calls, but only one at a time: MPI calls are not made
>concurrently from
>two distinct threads (all MPI calls are \serialized").
>
>MPI_THREAD_MULTIPLE
>Multiple threads may call MPI, with no restrictions.
>
>When passing in one of the different thread levels, you're telling the
>MPI implementation what level of support is required based on the way
>your code is designed.  If you have multiple threads making MPI calls
>simultaneously you must have MPI_THREAD_MULTIPLE support.  If your
>threads have locks that ensure that only one thread is making a call
>at a time, then you can use MPI_THREAD_SERIALIZED support.
>
>It sounds like you might need MPI_THREAD_MULTIPLE support since you
>mentioned that there is no locking going on.  Do you have multiple
>threads making MPI calls or only one?  Are you checking the return
>value from MPI_Init_thread (the provided parameter)?  I believe that
>you may not be getting the threaded level that your app requires.
>Can you let us know what version of mvapich2 you are using?
>
>Let me know if this information helps or if you have any further
>questions.
>
>On Mon, Dec 6, 2010 at 12:12 PM, Martin Schatz <mschatz at cs.utexas.edu>
>wrote:
>> Hello,
>> My name is Martin Schatz and I am currently working on a C++ project
>>that
>> uses the mvapich2 library.  I have been running into some problems while
>> running my code and cannot for the life of me figure out what is going
>>on.
>>  I was hoping that maybe someone on this list could help out.  If this
>>is
>> the wrong mailing list, please refer me to the correct one.
>> For background, the project I am working on requires a thread-safe
>> implementation of MPI as multiple threads (pthreads are used) are
>>making MPI
>> calls at the same time.  Currently, I have the level of thread safety
>>set to
>> MPI_THREAD_SERIALIZED, though I have tried the same code with the higher
>> MPI_THREAD_MULTIPLE safety level (but I do not think the MULTIPLE level
>>is
>> necessary as I am ok with the calls occurring in some serial fashion).
>>When
>> running my program using mpirun with debugging turned on, I see no
>> indication that something is wrong, however when I run the program in an
>> large parallel environment (TACC machines), the following error occurs:
>> Exit code -5 signaled from i182-111.ranger.tacc.utexas.edu
>> Killing remote processes...current bytes -56, total bytes 0, remote id
>>11
>> Assertion failed in file ch3_smp_progress.c at line 2328:
>> s_current_bytes[vc->smp.local_nodes] == 0
>> internal ABORT - process 12MPI process terminated unexpectedly
>> DONE
>> This error does not occur in a specific place during the execution
>> (everything seems to be working fine when it occurs).  I was wondering
>>if
>> anyone on this list could explain what this error means and perhaps some
>> tips on tracking down the bug.  Also, I am not using explicit locks in
>>the
>> program around the MPI calls.  I thought that by specifying the level of
>> thread safety in MPI_thread_init, it would be taken care of for me.  Is
>>this
>> in error?  Thanks for any guidance you can provide.
>>
>> Martin Schatz
>> _______________________________________________
>> mvapich-discuss mailing list
>> mvapich-discuss at cse.ohio-state.edu
>> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>>
>>
>
>
>
>-- 
>Jonathan Perkins
>http://www.cse.ohio-state.edu/~perkinjo
>
>_______________________________________________
>mvapich-discuss mailing list
>mvapich-discuss at cse.ohio-state.edu
>http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss