[mvapich-discuss] Assertion failed in file ch3_smp_progress.c at line 2818: s_current_bytes[vc->sm

[Chintu]

Hi,

I am running parallel Fortran jobs. Jobs run well for 12 cores but for
more cores like 24, 36, or 48 I am getting the following errors:

/opt/gridengine/default/spool/compute-3-3/active_jobs/48858.1/pe_hostfile
compute-3-3
compute-3-3
compute-3-3
compute-3-3
compute-3-3
compute-3-3
compute-3-3
compute-3-3
compute-3-3
compute-3-3
compute-3-3
compute-3-3
compute-1-31
compute-1-31
compute-1-31
compute-1-31
compute-1-31
compute-1-31
compute-1-31
compute-1-31
compute-1-31
compute-1-31
compute-1-31
compute-1-31
current bytes 840, total bytes 896, remote id 1
Assertion failed in file ch3_smp_progress.c at line 2818: s_current_bytes[vc->sm
p.local_nodes] == 0
internal ABORT - process 0
MPI process (rank: 0) terminated unexpectedly on compute-3-3.local
Exit code -5 signaled from compute-1-31.local
forrtl: error (69): process interrupted (SIGINT)
Image              PC                Routine            Line        Source

libmpich.so.1.2    00002B10379B1B69  Unknown               Unknown  Unknown
libmpich.so.1.2    00002B1037A1521C  Unknown               Unknown  Unknown
libmpich.so.1.2    00002B1037A14975  Unknown               Unknown  Unknown
libmpich.so.1.2    00002B1037A0A73B  Unknown               Unknown  Unknown
libmpich.so.1.2    00002B1037A091CF  Unknown               Unknown  Unknown
libmpich.so.1.2    00002B1037A089FA  Unknown               Unknown  Unknown
pne3_o             000000000043A679  Unknown               Unknown  Unknown
pne3_o             0000000000416459  Unknown               Unknown  Unknown
pne3_o             0000000000410B9F  Unknown               Unknown  Unknown
pne3_o             0000000000427FF2  Unknown               Unknown  Unknown
pne3_o             000000000040539C  Unknown               Unknown  Unknown
libc.so.6          0000003799A1D994  Unknown               Unknown  Unknown
pne3_o             00000000004052A9  Unknown               Unknown  Unknown
forrtl: error (69): process interrupted (SIGINT)
Image              PC                Routine            Line        Source

libmpich.so.1.2    00002B46161F8EBB  Unknown               Unknown  Unknown
libmpich.so.1.2    00002B46161EFB85  Unknown               Unknown  Unknown
libmpich.so.1.2    00002B461625321C  Unknown               Unknown  Unknown
libmpich.so.1.2    00002B4616252975  Unknown               Unknown  Unknown
libmpich.so.1.2    00002B461624873B  Unknown               Unknown  Unknown
libmpich.so.1.2    00002B46162471CF  Unknown               Unknown  Unknown
libmpich.so.1.2    00002B46162469FA  Unknown               Unknown  Unknown
pne3_o             000000000043A679  Unknown               Unknown  Unknown
pne3_o             0000000000416459  Unknown               Unknown  Unknown
pne3_o             0000000000410B9F  Unknown               Unknown  Unknown
pne3_o             0000000000427FF2  Unknown               Unknown  Unknown
pne3_o             000000000040539C  Unknown               Unknown  Unknown
libc.so.6          0000003799A1D994  Unknown               Unknown  Unknown
pne3_o             00000000004052A9  Unknown               Unknown  Unknown
forrtl: error (69): process interrupted (SIGINT)
Image              PC                Routine            Line        Source

libmpich.so.1.2    00002B374E5F37BD  Unknown               Unknown  Unknown
libmpich.so.1.2    00002B374E5EDB4F  Unknown               Unknown  Unknown
libmpich.so.1.2    00002B374E65121C  Unknown               Unknown  Unknown
libmpich.so.1.2    00002B374E650975  Unknown               Unknown  Unknown
libmpich.so.1.2    00002B374E64673B  Unknown               Unknown  Unknown
libmpich.so.1.2    00002B374E6451CF  Unknown               Unknown  Unknown
libmpich.so.1.2    00002B374E6449FA  Unknown               Unknown  Unknown
pne3_o             000000000043A679  Unknown               Unknown  Unknown
pne3_o             0000000000416459  Unknown               Unknown  Unknown
pne3_o             0000000000410B9F  Unknown               Unknown  Unknown
pne3_o             0000000000427FF2  Unknown               Unknown  Unknown
pne3_o             000000000040539C  Unknown               Unknown  Unknown
libc.so.6          0000003799A1D994  Unknown               Unknown  Unknown
pne3_o             00000000004052A9  Unknown               Unknown  Unknown
forrtl: error (69): process interrupted (SIGINT)
Image              PC                Routine            Line        Source

libmpich.so.1.2    00002B9D3D36BF97  Unknown               Unknown  Unknown
libmpich.so.1.2    00002B9D3D308B1C  Unknown               Unknown  Unknown
libmpich.so.1.2    00002B9D3D306C8D  Unknown               Unknown  Unknown
libmpich.so.1.2    00002B9D3D36A21C  Unknown               Unknown  Unknown
libmpich.so.1.2    00002B9D3D369975  Unknown               Unknown  Unknown
libmpich.so.1.2    00002B9D3D35F73B  Unknown               Unknown  Unknown
libmpich.so.1.2    00002B9D3D35E1CF  Unknown               Unknown  Unknown
libmpich.so.1.2    00002B9D3D35D9FA  Unknown               Unknown  Unknown
pne3_o             000000000043A679  Unknown               Unknown  Unknown
pne3_o             0000000000416459  Unknown               Unknown  Unknown
pne3_o             0000000000410B9F  Unknown               Unknown  Unknown
pne3_o             0000000000427FF2  Unknown               Unknown  Unknown
pne3_o             000000000040539C  Unknown               Unknown  Unknown
libc.so.6          000000346021D994  Unknown               Unknown  Unknown
pne3_o             00000000004052A9  Unknown               Unknown  Unknown
rm: cannot remove `/tmp/48858.1.all.q/rsh': No such file or directory
Connection to compute-1-31.local closed by remote host.

Any help or suggestion in this regard will be appreciated.

Thanks,
Praveen