[mvapich-discuss] multithreaded win_lock

[Thiago Ize]

Hi,

I have a multithreaded program that will seg fault when I do concurrent 
mpi_get operations.  For instance:

All processes have a single thread do the following:
  initialize with MPI_THREAD_MULTIPLE
  localData = MPI::Alloc_mem();
  MPI::Win win = win::Create(localData, ...);

Then I'll have each thread in each process do:
    int owner = something that varies per thread.
    MPI_Win_lock(MPI_LOCK_EXCLUSIVE, owner, 0, win); //shared seg faults 
as well
    MPI_Get(threadLocalData, ..., owner, win);
    MPI_Win_unlock(owner, win); //Seg faults here.

The owner variable can very among the various threads so that thread 1 
might want to get data from process 2 while thread 2 wants data from 
process 5, and so on.  The idea is that all my threads are reading data 
from various nodes in my cluster and I would like this to all happen at 
the same time.  Since no one ever writes to the data in localData, this 
should all be safe and valid code assuming mpi allows this.

With debugging turned on, sometimes I'll get errors such as:
Fatal error in MPI_Win_unlock: Other MPI error, error stack:
MPI_Win_unlock(117)...: MPI_Win_unlock(rank=1, win=0xa0000001) failed
MPIDI_Win_unlock(2551): Wrong synchronization of RMA calls

I'm guessing this is due to the concurrent window locking of different 
processes?  Shouldn't this be allowed if I have MPI_THREAD_MULTIPLE and 
I'm just doing MPI_Gets?  Furthermore, shouldn't the EXCLUSIVE lock 
handle this if it's not allowed?

If I put a mutex around the lock/get/unlock mpi code then everything 
works fine but of course it doesn't scale well to take advantage of my 
multicore nodes.

Any suggestions about what I could be doing wrong and if there is a way 
around this?  

Thanks,
Thiago