[mvapich-discuss] Memory issue with MPI_Ineighbor_alltoallv

[Liu, Fang]

Dear all,

I am getting following error when running the application with 216 processes, but there is no problem
with 64 processes, and I double checked the total memory usage which doesn't exceed the request memory
on PBS system. I am wondering if this is known issue and I suspect it may be related to some internal memory
setting for MPI_Ineighbor_alltoallv function? Any suggestion on resolving this issue would be really appreciated.  

[cli_189]: aborting job:
Fatal error in PMPI_Ineighbor_alltoallv:
Other MPI error, error stack:
PMPI_Ineighbor_alltoallv(229).......:  MPI_Ineighbor_alltoallv(sendbuf=0x2aa
ac902a4f0, sendcounts=0x2aaac9004b50, sdispls=0x2aaac9003670, sendtype=MPI_D
OUBLE, recvbuf=0x2aaac901ab20, recvcounts=0x2aaac9007510, rdispls=0x2aaac9006030, recvtype=MPI_DOUBLE, comm=comm=0x84000006, request=0x7fffff
PMPI_Ineighbor_alltoallv(215).......:
MPIR_Ineighbor_alltoallv_impl(112)..:
MPIR_Ineighbor_alltoallv_default(78):
MPID_Sched_recv(599)................:
MPIDU_Sched_add_entry(425)..........: Out of memory

---------
Fang (Cherry) Liu, Ph.D. (fang.liu at oit.gatech.edu)

Research Scientist II
Partnership for an Advanced Computing Environment
Georgia Institute of Technology
258 Fourth Street, Rich Building, Room 327
Atlanta, GA  30332-0700
Office: (404) 385-0045