[mvapich-discuss] Error (PMPI_Init_thread) when running MPI

[Xiaoyi Lu]

(Just for closing this thread.)

By having more discussions with the user, we found the issue can be fixed by specifying a different shared memory path through the runtime parameter. 

The parameter information can be found through the following link:
http://mvapich.cse.ohio-state.edu/static/media/mvapich/mvapich2-2.2-userguide.html#x1-22900011.63

Thanks,
Xiaoyi

> On Mar 15, 2017, at 7:06 PM, Junjie Qian <junjieqian at outlook.com> wrote:
> 
> Hi Hari,
> 
> Thank you for your help and response!
> The application has no failure before (fresh start in a new container). It fails with the same error msg after the /dev/shm is cleaned.
> 
> I will try MVAPICH2-Virt, thank you for your suggestion.
> 
> Thanks
> Junjie Qian
> From: hari.subramoni at gmail.com <hari.subramoni at gmail.com> on behalf of Hari Subramoni <subramoni.1 at osu.edu>
> Sent: Wednesday, March 15, 2017 3:14:45 PM
> To: Junjie Qian
> Cc: mvapich-discuss at cse.ohio-state.edu
> Subject: Re: [mvapich-discuss] Error (PMPI_Init_thread) when running MPI
>  
> Hello, 
> 
> It looks like /dev/shm folder on your system is full. This is possibly leading to this issue. Ideally, this should not happen... especially if your application is exiting cleanly. Can you please let us know if there have been any failures at the application level previously? 
> 
> As a workaround, please try to clear the files in /dev/shm and retry. 
> 
> On a different note, I see that you maybe running in container environment. We have a version of mvapich2 specifically optimized for delivering bare metal virtualized environments called MVAPICH2-Virt. I'd recommend that you use it. 
> 
> Regards, 
> Hari.
> 
> 
> On Mar 15, 2017 5:25 PM, "Junjie Qian" <junjieqian at outlook.com> wrote:
> Hi List,
> 
> Recently I have a problem with MVAPICH2 running, which did not occur before. The error log is as:
> 
> 
> [cli_0]: aborting job:
> Fatal error in PMPI_Init_thread:
> Other MPI error, error stack:
> MPIR_Init_thread(514)..........:
> MPID_Init(365).................: channel initialization failed
> MPIDI_CH3_Init(404)............:
> MPIDI_CH3I_SHMEM_Helper_fn(929): write: No space left on device
> [:mpispawn_0][readline] Unexpected End-Of-File on file descriptor 8. MPI process died?
> [mpispawn_0][mtpmi_processops] Error while reading PMI socket. MPI process died?
> 
> My command line is as: mpirun_rsh -np 2 container_1 container_2 MV2_NUM_PORTS=1 MV2_IBA_HCA=mlx4_0 MV2_DEFAULT_PORT=1 MV2_SMP_USE_CMA=0 MV2_ENABLE_AFFINITY=0 ./exec
> 
> Can you give me some suggestion on how to solve this issue? I tried different combinations of command options in the user guide, but none help.
> 
> Thank you
> Junjie Qian
> 
> 
> _______________________________________________
> mvapich-discuss mailing list
> mvapich-discuss at cse.ohio-state.edu
> http://mailman.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
> 
> 
> _______________________________________________
> mvapich-discuss mailing list
> mvapich-discuss at cse.ohio-state.edu
> http://mailman.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss