[mvapich-discuss] Assertion failed in file ch3_smp_progress.c at line 2818: s_current_bytes[vc->sm

[Devendar Bureddy]

Hi Praveen

Which version of MVAPICH2 you are using?   The latest version 1.8.1 or
1.9a2 is recommended.   If you are not using latest version, can you
please try with it?

-Devendar

On Tue, Dec 11, 2012 at 11:45 AM, Chintu <chintudcy at gmail.com> wrote:
> Hi,
>
> I am running parallel Fortran jobs. Jobs run well for 12 cores but for
> more cores like 24, 36, or 48 I am getting the following errors:
>
> /opt/gridengine/default/spool/compute-3-3/active_jobs/48858.1/pe_hostfile
> compute-3-3
> compute-3-3
> compute-3-3
> compute-3-3
> compute-3-3
> compute-3-3
> compute-3-3
> compute-3-3
> compute-3-3
> compute-3-3
> compute-3-3
> compute-3-3
> compute-1-31
> compute-1-31
> compute-1-31
> compute-1-31
> compute-1-31
> compute-1-31
> compute-1-31
> compute-1-31
> compute-1-31
> compute-1-31
> compute-1-31
> compute-1-31
> current bytes 840, total bytes 896, remote id 1
> Assertion failed in file ch3_smp_progress.c at line 2818: s_current_bytes[vc->sm
> p.local_nodes] == 0
> internal ABORT - process 0
> MPI process (rank: 0) terminated unexpectedly on compute-3-3.local
> Exit code -5 signaled from compute-1-31.local
> forrtl: error (69): process interrupted (SIGINT)
> Image              PC                Routine            Line        Source
>
> libmpich.so.1.2    00002B10379B1B69  Unknown               Unknown  Unknown
> libmpich.so.1.2    00002B1037A1521C  Unknown               Unknown  Unknown
> libmpich.so.1.2    00002B1037A14975  Unknown               Unknown  Unknown
> libmpich.so.1.2    00002B1037A0A73B  Unknown               Unknown  Unknown
> libmpich.so.1.2    00002B1037A091CF  Unknown               Unknown  Unknown
> libmpich.so.1.2    00002B1037A089FA  Unknown               Unknown  Unknown
> pne3_o             000000000043A679  Unknown               Unknown  Unknown
> pne3_o             0000000000416459  Unknown               Unknown  Unknown
> pne3_o             0000000000410B9F  Unknown               Unknown  Unknown
> pne3_o             0000000000427FF2  Unknown               Unknown  Unknown
> pne3_o             000000000040539C  Unknown               Unknown  Unknown
> libc.so.6          0000003799A1D994  Unknown               Unknown  Unknown
> pne3_o             00000000004052A9  Unknown               Unknown  Unknown
> forrtl: error (69): process interrupted (SIGINT)
> Image              PC                Routine            Line        Source
>
> libmpich.so.1.2    00002B46161F8EBB  Unknown               Unknown  Unknown
> libmpich.so.1.2    00002B46161EFB85  Unknown               Unknown  Unknown
> libmpich.so.1.2    00002B461625321C  Unknown               Unknown  Unknown
> libmpich.so.1.2    00002B4616252975  Unknown               Unknown  Unknown
> libmpich.so.1.2    00002B461624873B  Unknown               Unknown  Unknown
> libmpich.so.1.2    00002B46162471CF  Unknown               Unknown  Unknown
> libmpich.so.1.2    00002B46162469FA  Unknown               Unknown  Unknown
> pne3_o             000000000043A679  Unknown               Unknown  Unknown
> pne3_o             0000000000416459  Unknown               Unknown  Unknown
> pne3_o             0000000000410B9F  Unknown               Unknown  Unknown
> pne3_o             0000000000427FF2  Unknown               Unknown  Unknown
> pne3_o             000000000040539C  Unknown               Unknown  Unknown
> libc.so.6          0000003799A1D994  Unknown               Unknown  Unknown
> pne3_o             00000000004052A9  Unknown               Unknown  Unknown
> forrtl: error (69): process interrupted (SIGINT)
> Image              PC                Routine            Line        Source
>
> libmpich.so.1.2    00002B374E5F37BD  Unknown               Unknown  Unknown
> libmpich.so.1.2    00002B374E5EDB4F  Unknown               Unknown  Unknown
> libmpich.so.1.2    00002B374E65121C  Unknown               Unknown  Unknown
> libmpich.so.1.2    00002B374E650975  Unknown               Unknown  Unknown
> libmpich.so.1.2    00002B374E64673B  Unknown               Unknown  Unknown
> libmpich.so.1.2    00002B374E6451CF  Unknown               Unknown  Unknown
> libmpich.so.1.2    00002B374E6449FA  Unknown               Unknown  Unknown
> pne3_o             000000000043A679  Unknown               Unknown  Unknown
> pne3_o             0000000000416459  Unknown               Unknown  Unknown
> pne3_o             0000000000410B9F  Unknown               Unknown  Unknown
> pne3_o             0000000000427FF2  Unknown               Unknown  Unknown
> pne3_o             000000000040539C  Unknown               Unknown  Unknown
> libc.so.6          0000003799A1D994  Unknown               Unknown  Unknown
> pne3_o             00000000004052A9  Unknown               Unknown  Unknown
> forrtl: error (69): process interrupted (SIGINT)
> Image              PC                Routine            Line        Source
>
> libmpich.so.1.2    00002B9D3D36BF97  Unknown               Unknown  Unknown
> libmpich.so.1.2    00002B9D3D308B1C  Unknown               Unknown  Unknown
> libmpich.so.1.2    00002B9D3D306C8D  Unknown               Unknown  Unknown
> libmpich.so.1.2    00002B9D3D36A21C  Unknown               Unknown  Unknown
> libmpich.so.1.2    00002B9D3D369975  Unknown               Unknown  Unknown
> libmpich.so.1.2    00002B9D3D35F73B  Unknown               Unknown  Unknown
> libmpich.so.1.2    00002B9D3D35E1CF  Unknown               Unknown  Unknown
> libmpich.so.1.2    00002B9D3D35D9FA  Unknown               Unknown  Unknown
> pne3_o             000000000043A679  Unknown               Unknown  Unknown
> pne3_o             0000000000416459  Unknown               Unknown  Unknown
> pne3_o             0000000000410B9F  Unknown               Unknown  Unknown
> pne3_o             0000000000427FF2  Unknown               Unknown  Unknown
> pne3_o             000000000040539C  Unknown               Unknown  Unknown
> libc.so.6          000000346021D994  Unknown               Unknown  Unknown
> pne3_o             00000000004052A9  Unknown               Unknown  Unknown
> rm: cannot remove `/tmp/48858.1.all.q/rsh': No such file or directory
> Connection to compute-1-31.local closed by remote host.
>
> Any help or suggestion in this regard will be appreciated.
>
> Thanks,
> Praveen
> _______________________________________________
> mvapich-discuss mailing list
> mvapich-discuss at cse.ohio-state.edu
> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss



-- 
Devendar