[mvapich-discuss] waitsome/testsome memory allocation

[Justin]

Hi,

I am tracking down some memory issues in our code.  And I am finding 
strange memory allocations occurring within MPI_Waitsome and 
MPI_Testsome.  In one section of our code we use MPI_Pack and MPI_Unpack 
to combine a bunch of small messages.  We then send out the packed 
messages using isend.  The receiving processors post irecvs.  To 
complete the communication we use both testsome and waitsome.  What we 
are seeing is processors start by allocating a small amount of memory 
but as the code marches forward in time processors will allocate more 
memory within one of these mpi calls.  Processors continue allocating 
larger and larger amounts of memory as time goes on.  For example early 
on the allocation might be  a couple KB but eventually it will get to 
around 1MB and i've even seen it as high as 14MB.  I predict that if I 
ran it further it would allocate a much larger amount that 14MB.  
Processors are not all allocating this memory at the same time.   In 
other parts of the code we do not use packing and we do not see this 
allocation behavior.  I'm guessing that somewhere we are either 
miss-using packing or some other MPI feature and are causing MPI to leak.

I was wondering if you could tell me why testsome/waitsome would 
allocate memory as that could provide a good hint as to how we are 
miss-using mpi.

Currently we are using mvapich version 0.9.9  on Atlas at LLNL

Thanks,
Justin