[mvapich-discuss] Error (PMPI_Init_thread) when running MPI

[Junjie Qian]

Hi Hari,


Thank you for your help and response!

The application has no failure before (fresh start in a new container). It fails with the same error msg after the /dev/shm is cleaned.


I will try MVAPICH2-Virt, thank you for your suggestion.


Thanks

Junjie Qian

________________________________
From: hari.subramoni at gmail.com <hari.subramoni at gmail.com> on behalf of Hari Subramoni <subramoni.1 at osu.edu>
Sent: Wednesday, March 15, 2017 3:14:45 PM
To: Junjie Qian
Cc: mvapich-discuss at cse.ohio-state.edu
Subject: Re: [mvapich-discuss] Error (PMPI_Init_thread) when running MPI

Hello,

It looks like /dev/shm folder on your system is full. This is possibly leading to this issue. Ideally, this should not happen... especially if your application is exiting cleanly. Can you please let us know if there have been any failures at the application level previously?

As a workaround, please try to clear the files in /dev/shm and retry.

On a different note, I see that you maybe running in container environment. We have a version of mvapich2 specifically optimized for delivering bare metal virtualized environments called MVAPICH2-Virt. I'd recommend that you use it.

Regards,
Hari.


On Mar 15, 2017 5:25 PM, "Junjie Qian" <junjieqian at outlook.com<mailto:junjieqian at outlook.com>> wrote:

Hi List,


Recently I have a problem with MVAPICH2 running, which did not occur before. The error log is as:


[cli_0]: aborting job:
Fatal error in PMPI_Init_thread:
Other MPI error, error stack:
MPIR_Init_thread(514)..........:
MPID_Init(365).................: channel initialization failed
MPIDI_CH3_Init(404)............:
MPIDI_CH3I_SHMEM_Helper_fn(929): write: No space left on device
[:mpispawn_0][readline] Unexpected End-Of-File on file descriptor 8. MPI process died?
[mpispawn_0][mtpmi_processops] Error while reading PMI socket. MPI process died?

My command line is as: mpirun_rsh -np 2 container_1 container_2 MV2_NUM_PORTS=1 MV2_IBA_HCA=mlx4_0 MV2_DEFAULT_PORT=1 MV2_SMP_USE_CMA=0 MV2_ENABLE_AFFINITY=0 ./exec

Can you give me some suggestion on how to solve this issue? I tried different combinations of command options in the user guide, but none help.

Thank you
Junjie Qian

_______________________________________________
mvapich-discuss mailing list
mvapich-discuss at cse.ohio-state.edu<mailto:mvapich-discuss at cse.ohio-state.edu>
http://mailman.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20170315/2fe290dd/attachment.html>