<html><head></head><body><div class="ydp40e89ec0yahoo-style-wrap" style="font-family:Helvetica Neue, Helvetica, Arial, sans-serif;font-size:16px;"><div></div>
<div dir="ltr" data-setdir="false"><span>I get that after setting MV2_SHOW_ENV_INFO=3</span><br> </div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false"><div><div> MVAPICH2-2.3.5 Parameters</div><div>---------------------------------------------------------------------</div><div> PROCESSOR ARCH NAME : MV2_ARCH_AMD_EPYC_7742_128</div><div> PROCESSOR FAMILY NAME : MV2_CPU_FAMILY_AMD</div><div> PROCESSOR MODEL NUMBER : 49</div><div> HCA NAME : MV2_HCA_MLX_CX_EDR</div><div> HETEROGENEOUS HCA : NO</div><div> MV2_VBUF_TOTAL_SIZE : 16384</div><div> MV2_IBA_EAGER_THRESHOLD : 16384</div><div> MV2_RDMA_FAST_PATH_BUF_SIZE : 5120</div><div> MV2_PUT_FALLBACK_THRESHOLD : 8192</div><div> MV2_GET_FALLBACK_THRESHOLD : 0</div><div> MV2_EAGERSIZE_1SC : 8192</div><div> MV2_SMP_EAGERSIZE : 16385</div><div> MV2_SMP_QUEUE_LENGTH : 262144</div><div> MV2_SMP_NUM_SEND_BUFFER : 32</div><div> MV2_SMP_BATCH_SIZE : 8</div><div> Tuning Table: : MV2_ARCH_AMD_EPYC_7742_128 MV2_HCA_MLX_CX_EDR</div><div>---------------------------------------------------------------------</div><div><br></div><div> MVAPICH2 All Parameters</div><div> MPIRUN_RSH_LAUNCH : 0</div><div> MV2_SHMEM_BACKED_UD_CM : 0</div><div> MV2_3DTORUS_SUPPORT : 0</div><div> MV2_NUM_SA_QUERY_RETRIES : 20</div><div> MV2_NUM_SLS : 8</div><div> MV2_DEFAULT_SERVICE_LEVEL : 0</div><div> MV2_PATH_SL_QUERY : 0</div><div> MV2_USE_QOS : 0</div><div> MV2_USE_MCAST : 0</div><div> MV2_USE_RDMA_CM_MCAST : 0</div><div> MV2_MCAST_BCAST_MIN_MSG : 1</div><div> MV2_MCAST_BCAST_MAX_MSG : 524288</div><div> MV2_ALLGATHER_BRUCK_THRESHOLD : 524288</div><div> MV2_ALLGATHER_RD_THRESHOLD : 81920</div><div> MV2_ALLGATHER_REVERSE_RANKING : 1</div><div> MV2_ALLGATHERV_RD_THRESHOLD : 0</div><div> MV2_ALLREDUCE_2LEVEL_MSG : 262144</div><div> MV2_ALLREDUCE_SHORT_MSG : 2048</div><div> MV2_ALLTOALL_MEDIUM_MSG : 16384</div><div> MV2_ALLTOALL_SMALL_MSG : 2048</div><div> MV2_ALLTOALL_THROTTLE_FACTOR : 32</div><div> MV2_BCAST_TWO_LEVEL_SYSTEM_SIZE : 64</div><div> MV2_GATHER_SWITCH_PT : 0</div><div> MV2_INTRA_SHMEM_REDUCE_MSG : 2048</div><div> MV2_KNOMIAL_2LEVEL_BCAST_MESSAGE_SIZE_THRESHOLD : 2048</div><div> MV2_KNOMIAL_2LEVEL_BCAST_SYSTEM_SIZE_THRESHOLD : 64</div><div> MV2_KNOMIAL_INTER_LEADER_THRESHOLD : 65536</div><div> MV2_KNOMIAL_INTER_NODE_FACTOR : 4</div><div> MV2_KNOMIAL_INTRA_NODE_FACTOR : 4</div><div> MV2_KNOMIAL_INTRA_NODE_THRESHOLD : 131072</div><div> MV2_RED_SCAT_LARGE_MSG : 524288</div><div> MV2_RED_SCAT_SHORT_MSG : 64</div><div> MV2_REDUCE_2LEVEL_MSG : 16384</div><div> MV2_REDUCE_SHORT_MSG : 8192</div><div> MV2_SCATTER_MEDIUM_MSG : 0</div><div> MV2_SCATTER_SMALL_MSG : 0</div><div> MV2_SHMEM_ALLREDUCE_MSG : 32768</div><div> MV2_SHMEM_COLL_MAX_MSG_SIZE : 131072</div><div> MV2_SHMEM_COLL_NUM_COMM : 32</div><div> MV2_SHMEM_COLL_NUM_PROCS : 128</div><div> MV2_SHMEM_COLL_SPIN_COUNT : 5</div><div> MV2_SHMEM_REDUCE_MSG : 4096</div><div> MV2_USE_BCAST_SHORT_MSG : 16384</div><div> MV2_USE_DIRECT_GATHER : 1</div><div> MV2_USE_DIRECT_GATHER_SYSTEM_SIZE_MEDIUM : 1024</div><div> MV2_USE_DIRECT_GATHER_SYSTEM_SIZE_SMALL : 384</div><div> MV2_USE_DIRECT_SCATTER : 1</div><div> MV2_USE_OSU_COLLECTIVES : 1</div><div> MV2_USE_OSU_NB_COLLECTIVES : 1</div><div> MV2_USE_KNOMIAL_2LEVEL_BCAST : 1</div><div> MV2_USE_KNOMIAL_INTER_LEADER_BCAST : 1</div><div> MV2_USE_SCATTER_RD_INTER_LEADER_BCAST : 1</div><div> MV2_USE_SCATTER_RING_INTER_LEADER_BCAST : 1</div><div> MV2_USE_SHMEM_ALLREDUCE : 1</div><div> MV2_USE_SHMEM_BARRIER : 1</div><div> MV2_USE_SHMEM_BCAST : 1</div><div> MV2_USE_SHMEM_COLL : 1</div><div> MV2_USE_SHMEM_REDUCE : 1</div><div> MV2_USE_TWO_LEVEL_GATHER : 1</div><div> MV2_USE_TWO_LEVEL_SCATTER : 1</div><div> MV2_USE_XOR_ALLTOALL : 1</div><div> MV2_ENABLE_SOCKET_AWARE_COLLECTIVES : 1</div><div> MV2_USE_SOCKET_AWARE_ALLREDUCE : 1</div><div> MV2_USE_SOCKET_AWARE_BARRIER : 1</div><div> MV2_USE_SOCKET_AWARE_SHARP_ALLREDUCE : 0</div><div> MV2_SOCKET_AWARE_ALLREDUCE_MAX_MSG : 2048</div><div> MV2_SOCKET_AWARE_ALLREDUCE_MIN_MSG : 1</div><div> MV2_DEFAULT_SRC_PATH_BITS : 0</div><div> MV2_DEFAULT_STATIC_RATE : 0</div><div> MV2_DEFAULT_TIME_OUT : 330772</div><div> MV2_DEFAULT_MTU : 5</div><div> MV2_DEFAULT_PKEY : 0</div><div> MV2_DEFAULT_QKEY : 0</div><div> MV2_DEFAULT_PORT : 1</div><div> MV2_DEFAULT_GID_INDEX : 0</div><div> MV2_DEFAULT_PSN : 0</div><div> MV2_DEFAULT_MAX_RECV_WQE : 128</div><div> MV2_DEFAULT_MAX_SEND_WQE : 64</div><div> MV2_DEFAULT_MAX_SG_LIST : 1</div><div> MV2_DEFAULT_MIN_RNR_TIMER : 12</div><div> MV2_DEFAULT_QP_OUS_RD_ATOM : 272</div><div> MV2_DEFAULT_RETRY_COUNT : 84677639</div><div> MV2_DEFAULT_RNR_RETRY : 202639111</div><div> MV2_DEFAULT_MAX_CQ_SIZE : 40000</div><div> MV2_DEFAULT_MAX_RDMA_DST_OPS : 4</div><div> MV2_INITIAL_PREPOST_DEPTH : 10</div><div> MV2_IWARP_MULTIPLE_CQ_THRESHOLD : 32</div><div> MV2_NUM_HCAS : 1</div><div> MV2_NUM_PORTS : 1</div><div> MV2_NUM_QP_PER_PORT : 1</div><div> MV2_MAX_RDMA_CONNECT_ATTEMPTS : 20</div><div> MV2_ON_DEMAND_UD_INFO_EXCHANGE : 0</div><div> MV2_PREPOST_DEPTH : 64</div><div> MV2_HOMOGENEOUS_CLUSTER : 0</div><div> MV2_NUM_CQES_PER_POLL : 96</div><div>ENDWINDOW_SIZE : 400</div><div><br></div></div><div><div> MV2_UD_VBUF_POOL_SIZE : 8192</div><div> MV2_UD_ZCOPY_RQ_SIZE : 4096</div><div> MV2_UD_ZCOPY_THRESHOLD : 16384</div><div> MV2_UD_ZCOPY_NUM_RETRY : 50000</div><div> MV2_USE_UD_ZCOPY : 1</div><div> MV2_USE_UD_HYBRID : 0</div><div> MV2_USE_ONLY_UD : 0</div><div> MV2_HYBRID_ENABLE_THRESHOLD : 1024</div><div> MV2_HYBRID_MAX_RC_CONN : 32</div><div> MV2_ASYNC_THREAD_STACK_SIZE : 1048576</div><div> MV2_THREAD_YIELD_SPIN_THRESHOLD : 5</div><div> MV2_SUPPORT_DPM : 0</div><div> MV2_USE_HUGEPAGES : 1</div><div>---------------------------------------------------------------------</div><div><br></div><div>Collective Tuning Tables</div><div> Collective Architecture Interconnect</div><div> Allgather MV2_ARCH_AMD_EPYC_7742_128 MV2_HCA_MLX_CX_EDR</div><div> Allreduce MV2_ARCH_AMD_EPYC_7742_128 MV2_HCA_MLX_CX_EDR</div><div> Alltoall MV2_ARCH_AMD_EPYC_7742_128 MV2_HCA_MLX_CX_EDR</div><div> Alltoallv MV2_ARCH_AMD_EPYC_7742_128 MV2_HCA_MLX_CX_EDR</div><div> Broadcast MV2_ARCH_AMD_EPYC_7742_128 MV2_HCA_MLX_CX_EDR</div><div> Gather MV2_ARCH_AMD_EPYC_7742_128 MV2_HCA_MLX_CX_EDR</div><div> Reduce MV2_ARCH_AMD_EPYC_7742_128 MV2_HCA_MLX_CX_EDR</div><div> Scatter MV2_ARCH_AMD_EPYC_7742_128 MV2_HCA_MLX_CX_EDR</div><div><br></div><div>---------------------------------------------------------------------</div><div><br></div><div> MV2_DREG_CACHE_LIMIT : 0</div><div> MV2_IBA_EAGER_THRESHOLD : 16384</div><div> MV2_MAX_INLINE_SIZE : 168</div><div> MV2_MAX_R3_PENDING_DATA : 524288</div><div> MV2_MED_MSG_RAIL_SHARING_POLICY : 0</div><div> MV2_NDREG_ENTRIES : 8704</div><div> MV2_NUM_RDMA_BUFFER : 16</div><div> MV2_NUM_SPINS_BEFORE_LOCK : 2000</div><div> MV2_POLLING_LEVEL : 1</div><div> MV2_POLLING_SET_LIMIT : 64</div><div> MV2_POLLING_SET_THRESHOLD : 256</div><div> MV2_R3_NOCACHE_THRESHOLD : 32768</div><div> MV2_R3_THRESHOLD : 4096</div><div> MV2_RAIL_SHARING_LARGE_MSG_THRESHOLD : 16384</div><div> MV2_RAIL_SHARING_MED_MSG_THRESHOLD : 2048</div><div> MV2_RAIL_SHARING_POLICY : 4</div><div> MV2_RDMA_EAGER_LIMIT : 32</div><div> MV2_RDMA_FAST_PATH_BUF_SIZE : 5120</div><div> MV2_RDMA_NUM_EXTRA_POLLS : 1</div><div> MV2_RNDV_EXT_SENDQ_SIZE : 5</div><div> MV2_RNDV_PROTOCOL : 4</div><div> MV2_SMP_RNDV_PROTOCOL : 4</div><div> MV2_SMALL_MSG_RAIL_SHARING_POLICY : 0</div><div> MV2_SPIN_COUNT : 5000</div><div> MV2_SRQ_LIMIT : 10</div><div> MV2_SRQ_MAX_SIZE : 32767</div><div> MV2_SRQ_SIZE : 80</div><div> MV2_STRIPING_THRESHOLD : 16384</div><div> MV2_USE_COALESCE : 1</div><div> MV2_USE_XRC : 0</div><div> MV2_VBUF_MAX : -1</div><div> MV2_VBUF_POOL_SIZE : 80</div><div> MV2_VBUF_SECONDARY_POOL_SIZE : 16</div><div> MV2_VBUF_TOTAL_SIZE : 16384</div><div> MV2_USE_IWARP_MODE : 0</div><div> MV2_CPU_BINDING_POLICY : hybrid</div><div> MV2_USE_HWLOC_CPU_BINDING : 1</div><div> MV2_ENABLE_AFFINITY : 1</div><div> MV2_ENABLE_LEASTLOAD : 0</div><div> MV2_SMP_BATCH_SIZE : 8</div><div> MV2_SMP_EAGERSIZE : 16385</div><div> MV2_SMP_QUEUE_LENGTH : 262144</div><div> MV2_SMP_NUM_SEND_BUFFER : 32</div><div> MV2_SMP_SEND_BUF_SIZE : 16384</div><div> MV2_USE_SHARED_MEM : 1</div><div> MV2_SMP_CMA_MAX_SIZE : 4194304</div><div> MV2_SMP_LIMIC2_MAX_SIZE : 0</div><div> MV2_SHOW_ENV_INFO : 3</div><div> MV2_DEFAULT_PUT_GET_LIST_SIZE : 200</div><div> MV2_EAGERSIZE_1SC : 8192</div><div> MV2_GET_FALLBACK_THRESHOLD : 0</div><div> MV2_PIN_POOL_SIZE : 2097152</div><div> MV2_PUT_FALLBACK_THRESHOLD : 8192</div><div> MV2_USE_RDMA_CM : 0</div><div> MV2_UD_MAX_ACK_PENDING : 100</div><div> MV2_UD_MAX_RECV_WQE : 4096</div><div> MV2_UD_MAX_RETRY_TIMEOUT : 20000000</div><div> MV2_UD_MAX_SEND_WQE : 2048</div><div> MV2_UD_MTU : 4096</div><div> MV2_UD_NUM_MSG_LIMIT : 512</div><div> MV2_UD_NUM_ZCOPY_RNDV_QPS : 64</div><div> MV2_UD_PROGRESS_SPIN : 1200</div><div> MV2_UD_PROGRESS_TIMEOUT : 48000</div><div> MV2_UD_RECVWINDOW_SIZE : 2501</div><div> MV2_UD_RETRY_COUNT : 1024</div><div> MV2_UD_RETRY_TIMEOUT : 500000</div><div> MV2_UD_SENDWINDOW_SIZE : 400</div><div> MV2_UD_VBUF_POOL_SIZE : 8192</div><div> MV2_UD_ZCOPY_RQ_SIZE : 4096</div><div> MV2_UD_ZCOPY_THRESHOLD : 16384</div><div> MV2_UD_ZCOPY_NUM_RETRY : 50000</div><div> MV2_USE_UD_ZCOPY : 1</div><div> MV2_USE_UD_HYBRID : 0</div><div> MV2_USE_ONLY_UD : 0</div><div> MV2_HYBRID_ENABLE_THRESHOLD : 1024</div><div> MV2_HYBRID_MAX_RC_CONN : 32</div><div> MV2_ASYNC_THREAD_STACK_SIZE : 1048576</div><div> MV2_THREAD_YIELD_SPIN_THRESHOLD : 5</div><div> MV2_SUPPORT_DPM : 0</div><div> MV2_USE_HUGEPAGES : 1</div><div>---------------------------------------------------------------------</div><div><br></div><div>Collective Tuning Tables</div><div> Collective Architecture Interconnect</div><div> Allgather MV2_ARCH_AMD_EPYC_7742_128 MV2_HCA_MLX_CX_EDR</div><div> Allreduce MV2_ARCH_AMD_EPYC_7742_128 MV2_HCA_MLX_CX_EDR</div><div> Alltoall MV2_ARCH_AMD_EPYC_7742_128 MV2_HCA_MLX_CX_EDR</div><div> Alltoallv MV2_ARCH_AMD_EPYC_7742_128 MV2_HCA_MLX_CX_EDR</div><div> Broadcast MV2_ARCH_AMD_EPYC_7742_128 MV2_HCA_MLX_CX_EDR</div><div> Gather MV2_ARCH_AMD_EPYC_7742_128 MV2_HCA_MLX_CX_EDR</div><div> Reduce MV2_ARCH_AMD_EPYC_7742_128 MV2_HCA_MLX_CX_EDR</div><div> Scatter MV2_ARCH_AMD_EPYC_7742_128 MV2_HCA_MLX_CX_EDR</div><div><br></div><div>---------------------------------------------------------------------</div><div><br></div></div><br></div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false"><br></div><div><br></div>
</div><div id="yahoo_quoted_3882804714" class="yahoo_quoted">
<div style="font-family:'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:13px;color:#26282a;">
<div>
On Thursday, 18 February 2021, 20:00:18 CET, vru.inbri--- via Mvapich-discuss <mvapich-discuss@lists.osu.edu> wrote:
</div>
<div><br></div>
<div><br></div>
<div><div id="yiv1947706683"><div><div class="yiv1947706683ydp269a92f6yahoo-style-wrap" style="font-family:Helvetica Neue, Helvetica, Arial, sans-serif;font-size:16px;"><div style="font-size:16px;font-family:Helvetica Neue, Helvetica, Arial, sans-serif;"></div>
<div dir="ltr" style="font-size:16px;font-family:Helvetica Neue, Helvetica, Arial, sans-serif;">When using a different number of processors the error becomes:</div><div dir="ltr" style="font-size:16px;font-family:Helvetica Neue, Helvetica, Arial, sans-serif;"><br clear="none"></div><div dir="ltr" style=""><div style=""><div style=""><font face="courier new, courier, monaco, monospace, sans-serif" style="" size="2">Program received signal SIGSEGV: Segmentation fault - invalid memory reference.</font></div><div style="font-size:16px;font-family:Helvetica Neue, Helvetica, Arial, sans-serif;"><br clear="none"></div></div>Does it help?</div><div style="font-size:16px;font-family:Helvetica Neue, Helvetica, Arial, sans-serif;"><br clear="none"></div>
</div><div class="yiv1947706683yahoo_quoted" id="yiv1947706683yahoo_quoted_3855761257">
<div style="font-family:'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:13px;color:#26282a;">
<div class="yiv1947706683yqt6594856641" id="yiv1947706683yqt63310"><div>
On Thursday, 18 February 2021, 19:38:48 CET, vru.inbri--- via Mvapich-discuss <mvapich-discuss@lists.osu.edu> wrote:
</div>
<div><br clear="none"></div>
<div><br clear="none"></div>
<div><div id="yiv1947706683"><div><div class="yiv1947706683yahoo-style-wrap" style="font-family:Helvetica Neue, Helvetica, Arial, sans-serif;font-size:16px;"><div dir="ltr"><div><div>Hi </div><div><br clear="none"></div><div>I built a Singularity container with Ubuntu, GNU compilers and MVAPICH2 2.3.5 </div><div><br clear="none"></div><div>When trying to run it on our cluster it fails with errors like:</div><div><br clear="none"></div><div><font face="courier new, courier, monaco, monospace, sans-serif" size="2">Fatal error in PMPI_Waitall:</font></div><div><font face="courier new, courier, monaco, monospace, sans-serif" size="2">Other MPI error, error stack:</font></div><div><font face="courier new, courier, monaco, monospace, sans-serif" size="2">PMPI_Waitall(419)..................: MPI_Waitall(count=7, req_array=0x55d7f03d0290, status_array=0x55d7f03b4e50) failed</font></div><div><font face="courier new, courier, monaco, monospace, sans-serif" size="2">MPIR_Waitall_impl(248).............:</font></div><div><font face="courier new, courier, monaco, monospace, sans-serif" size="2">MPIDI_CH3I_Progress(285)...........:</font></div><div><font face="courier new, courier, monaco, monospace, sans-serif" size="2">handle_read(1350)..................:</font></div><div><font face="courier new, courier, monaco, monospace, sans-serif" size="2">handle_read_individual(1408).......:</font></div><div><font face="courier new, courier, monaco, monospace, sans-serif" size="2">MPIDI_CH3I_MRAIL_Parse_header(1502): Control shouldn't reach here in prototype, header %d</font></div><div><font face="courier new, courier, monaco, monospace, sans-serif" size="2"> (errno 71)</font></div><div><br clear="none"></div><div dir="ltr">As a test I also installed the same OS, compilers and libraries in an empty virtual machine (directly, without using singularity) and everything works without problem</div><div><br clear="none"></div><div>Does this make any sense for you?</div><div><br clear="none"></div><div>Vru</div></div><br clear="none"></div></div></div></div>_______________________________________________<br clear="none">Mvapich-discuss mailing list<br clear="none"><a rel="nofollow noopener noreferrer" shape="rect" ymailto="mailto:Mvapich-discuss@lists.osu.edu" target="_blank" href="mailto:Mvapich-discuss@lists.osu.edu">Mvapich-discuss@lists.osu.edu</a><br clear="none"><a rel="nofollow noopener noreferrer" shape="rect" target="_blank" href="https://lists.osu.edu/mailman/listinfo/mvapich-discuss">https://lists.osu.edu/mailman/listinfo/mvapich-discuss</a><br clear="none"></div></div>
</div>
</div></div></div><div class="yqt6594856641" id="yqt08577">_______________________________________________<br clear="none">Mvapich-discuss mailing list<br clear="none"><a shape="rect" ymailto="mailto:Mvapich-discuss@lists.osu.edu" href="mailto:Mvapich-discuss@lists.osu.edu">Mvapich-discuss@lists.osu.edu</a><br clear="none"><a shape="rect" href="https://lists.osu.edu/mailman/listinfo/mvapich-discuss" target="_blank">https://lists.osu.edu/mailman/listinfo/mvapich-discuss</a><br clear="none"></div></div>
</div>
</div></body></html>