[mvapich-discuss] Intel compilers v16 problems

Jonathan Perkins perkinjo at cse.ohio-state.edu
Sat Apr 23 19:42:02 EDT 2016 

Hi Adam.  Thanks for the investigation.  I didn't see this followup issue
but I'll try the steps as you've described it.  It wasn't explicitly
mentioned in your email but I'm assuming that you ran autogen.sh after
every application of the patch.

On Sat, Apr 23, 2016 at 4:34 PM Moody, Adam T. <moody20 at llnl.gov> wrote:

> Hi Jonathan,
> Good news, strange news.  First, the patch definitely helps, so thanks for
> that!
>
> My first attempt with it succeeded.  This was an in-tree build.  I then
> tried an out-of-tree build, and that worked too.  Then today, I decided to
> try an out-of-tree build w/o the patch and that failed just as before.  So
> far, so good.
>
> Then things got strange.  I went back to the out-of-tree build w/ the
> patch, but this time it failed.  This attempt should have been identical to
> the first try that worked, because it executes commands saved as an rpm
> spec file that is stored as a tagged branch in an svn repo.  However, this
> time it failed with the following error:
>
> ../src/binding/fortran/use_mpi_f08/mpi_c_interface_glue.F90(143): error
> #7013: This module file was not generated by any release of this
> compiler.   [MPI_F08]
>     use :: mpi_f08, only : MPI_ADDRESS_KIND, MPI_Comm,
> MPI_Comm_delete_attr_function
> -----------^
>
> That error suggests the mpi_f08.mod is of the wrong file format.  I double
> checked that it used ifort to compile the module file.  Strange.
>
> Looking back through the make output, it's interesting that it printed
> lines about compiling the mpi_f08.mod file just before it started to
> compile mpi_c_interface_glue.F90.
>
> /bin/sh ./libtool  --tag=FC   --mode=compile ifort
> -Isrc/binding/fortran/use_mpi -Isrc/binding/fortran/use_mpi -g -O3 -g -O2
> -module src/binding/fortran/use_mpi -c
> ../src/binding/fortran/use_mpi_f08/`echo
> src/binding/fortran/use_mpi/mpi_f08.mod | cut -f1 -d '.' | sed -e
> 's+.*/++g'`.F90 -o src/binding/fortran/use_mpi_f08/`echo
> src/binding/fortran/use_mpi/mpi_f08.mod | cut -f1 -d '.' | sed -e
> 's+.*/++g'`.lo
>
> libtool: compile:  ifort -Isrc/binding/fortran/use_mpi
> -Isrc/binding/fortran/use_mpi -g -O3 -g -O2 -module
> src/binding/fortran/use_mpi -c
> ../src/binding/fortran/use_mpi_f08/mpi_f08.F90  -fPIC -o
> src/binding/fortran/use_mpi_f08/.libs/mpi_f08.o
>
> libtool: compile:  ifort -Isrc/binding/fortran/use_mpi
> -Isrc/binding/fortran/use_mpi -g -O3 -g -O2 -module
> src/binding/fortran/use_mpi -c
> ../src/binding/fortran/use_mpi_f08/mpi_f08.F90 -o
> src/binding/fortran/use_mpi_f08/mpi_f08.o >/dev/null 2>&1
>
> /bin/sh ./libtool  --tag=FC   --mode=compile ifort
> -Isrc/binding/fortran/use_mpi -Isrc/binding/fortran/use_mpi -g -O3 -g -O2
> -module src/binding/fortran/use_mpi -c
> ../src/binding/fortran/use_mpi_f08/`echo
> src/binding/fortran/use_mpi/mpi_c_interface_glue.mod | cut -f1 -d '.' | sed
> -e 's+.*/++g'`.F90 -o src/binding/fortran/use_mpi_f08/`echo
> src/binding/fortran/use_mpi/mpi_c_interface_glue.mod | cut -f1 -d '.' | sed
> -e 's+.*/++g'`.lo
>
> I then tried the in-tree build with the patch (again the same that had
> worked once before), but now that failed with an error like the following:
>
> make[2]: ***
> [src/binding/fortran/use_mpi_f08/lib_libmpifort_la-pmpi_f08.lo] Error 1
> make[2]: *** Waiting for unfinished jobs....
> make[2]: ***
> [src/binding/fortran/use_mpi_f08/lib_libmpifort_la-mpi_f08.lo] Error 1
>
> I couldn't find any more detailed error messages to go with that, so it's
> less clear what happened in this case.  Anyway, between these two different
> failures and the fact that seemingly identical build attempts had both
> worked once before, it's starting to feel like there may be a race
> condition in the build process.
>
> All of these were running parallel make with -j8.  I decided to drop the
> -j8 to go back to a serial make.  I tried two different builds with serial
> make, and both attempts completed w/o error.
>
> I'm now batting 2-for-2 with serial make but only hitting 2-for-5 with
> parallel make.
>
> So the short story.  The patch helps for sure.  There's still something
> strange going on in the F90 build, perhaps a race condition when using
> parallel make.
> -Adam
>
>
> ------------------------------
> *From:* Jonathan Perkins [perkinjo at cse.ohio-state.edu]
> *Sent:* Friday, April 22, 2016 11:32 AM
> *To:* Kenneth Raffenetti; Moody, Adam T.
> *Cc:* Troy Baer; mvapich-discuss at cse.ohio-state.edu
> *Subject:* Re: [mvapich-discuss] Intel compilers v16 problems
>
> It appears that whatever I was doing last night may have been checked
> properly because when I run (or did not run) ./autogen.sh.  In the end it
> appears that only the attached patch was necessary.  Adam, can you try this
> out and let us know if it resolves your issue?
>
> Please remember to run autogen.sh and configuere again after applying this
> patch.
>
> On Fri, Apr 22, 2016 at 1:25 PM Jonathan Perkins <
> perkinjo at cse.ohio-state.edu
> <http://redir.aspx?REF=GgyGZrbt7eKt5BzVCYShYK37vfGG02jcjOqrdtVeW2e6YRriq2vTCAFtYWlsdG86cGVya2luam9AY3NlLm9oaW8tc3RhdGUuZWR1>>
> wrote:
>
>> Hi guys.  I tried to backport the first patch Ken pointed out last night
>> but ran into some issues that may or may not be related to his next patch.
>> I'm going to work on this a bit more and let you know whether we have a
>> working patch or not.
>>
>> On Fri, Apr 22, 2016 at 12:21 PM Kenneth Raffenetti <raffenet at mcs.anl.gov
>> <http://redir.aspx?REF=HHbjDcxiXSu9hQbS3gPJMfx1JM4635TzC0oMYF1r1P-6YRriq2vTCAFtYWlsdG86cmFmZmVuZXRAbWNzLmFubC5nb3Y.>>
>> wrote:
>>
>>> There may be other stragglers, but try cherry-picking this one.
>>>
>>> http://git.mpich.org/mpich.git/commitdiff/0750f10fcab0a04b33a86ff12ca95739c7376a27
>>> <http://redir.aspx?REF=oUYkw-BvWEaV9qT3a5VTgi-6cl-Cwqx94Rxuk1eI7vi6YRriq2vTCAFodHRwOi8vc2VjdXJlLXdlYi5jaXNjby5jb20vMXZZUnhZMEx1cm0tN2NnQVQ1LTB6eTByaUhPZXU4dWF5Qnh0bDA4T1NNc2I4dmthMElXMGNJczBoYjVsZWlYcm9WVGNjczFXd0lCMHB2V1QtVVJUQjJZZ3EwQTg1YUMwTy1BZzVULXU2ZFJiTUx4dlhPMGx3NDNFd0s5M0xJb3o0TlhFaFNXelpkWml4QjNLdmNlVHRkOFVsb2xReENpbGVJQ3l2TXFHa0I4eGlKRHlyanBtRHRvMHFzUzV1aVRsdXR0WmRKQVZNem1lVTU2MlRWaG9rNzVySHFoMXpjTGRBSm9UM3NhQTdVV2RJZ3E2TWZvcHFqbEhpNWRzN3JYZ0M2R3l2R3hEYndUNERYYUlXTXhYMzU1S2pGbC1qNkt3WTF1TG9UTzh5OFRkQ0c2Zm40eks3eVJEUVRQSkFmV0xJWU5YY2lJZHRjUzVHeUdNaG5UbGRPclFNbWQ3ME5GZllVWW5USElORGNTVS9odHRwJTNBJTJGJTJGZ2l0Lm1waWNoLm9yZyUyRm1waWNoLmdpdCUyRmNvbW1pdGRpZmYlMkYwNzUwZjEwZmNhYjBhMDRiMzNhODZmZjEyY2E5NTczOWM3Mzc2YTI3>
>>>
>>> Ken
>>>
>>> On 04/21/2016 06:35 PM, Adam T. Moody wrote:
>>> > Thanks everyone.  That definitely improved things.  If I change things
>>> > to do an in-tree build, it gets past that missing file error. Now it
>>> > fails when trying to compile the first F08 wrapper.  I've cut-and-paste
>>> > that below too in case it's familiar to someone.
>>> >
>>> > Jonathan, if you put together a patch, I'll be happy to try it out.
>>> > -Adam
>>> >
>>> >
>>> >
>>> > /bin/sh ./libtool  --tag=FC   --mode=compile ifort -DHAVE_CONFIG_H -I.
>>> > -I./src/include  -I./src/include -I./src/mpi/datatype -I./src/mpi/spawn
>>> > -Isrc/util/logging/common -I./src/util/wrappers -I./src/util/wrappers
>>> > -I./src/binding/cxx
>>> > -I/builddir/build/BUILD/mvapich/src/src/binding/fortran/mpif_h
>>> > -Isrc/binding/fortran/use_mpi
>>> >
>>> -I/builddir/build/BUILD/mvapich/src/src/binding/fortran/use_mpi_f08/wrappers_c
>>> > -I./src/mpid/ch3/include -I./src/mpid/ch3/include
>>> >
>>> -I/builddir/build/BUILD/mvapich/src/src/mpid/ch3/channels/common/include
>>> > -I./src/mpid/ch3/channels/psm/include -I./src/mpid/common/datatype
>>> > -I./src/mpid/common/datatype -I./src/mpid/common/sched
>>> > -I./src/mpid/common/thread     -I./src/pmi/simple
>>> > -Icontrib/hwloc/include/ -I./contrib/hwloc/include
>>> > -I/builddir/build/BUILD/mvapich/src/src/mpid/ch3/channels/psm/include
>>> > -I/builddir/build/BUILD/mvapich/src/src/mpid/ch3/channels/psm/include
>>> > -I/builddir/build/BUILD/mvapich/src/src/util/wrappers
>>> > -I/builddir/build/BUILD/mvapich/src/src/util/wrappers
>>> > -I/builddir/build/BUILD/mvapich/src/src/mpl/include
>>> > -I/builddir/build/BUILD/mvapich/src/src/mpl/include
>>> > -I/builddir/build/BUILD/mvapich/src/src/openpa/src
>>> > -I/builddir/build/BUILD/mvapich/src/src/openpa/src -D_REENTRANT
>>> > -I/builddir/build/BUILD/mvapich/src/src/mpi/romio/include
>>> > -Isrc/binding/fortran/use_mpi -Isrc/binding/fortran/use_mpi -g -O3 -g
>>> > -O2 -c -o
>>> >
>>> src/binding/fortran/use_mpi_f08/wrappers_f/lib_libmpifort_la-abort_f08ts.lo
>>> > `test -f 'src/binding/fortran/use_mpi_f08/wrappers_f/abort_f08ts.F90'
>>> ||
>>> > echo './'`src/binding/fortran/use_mpi_f08/wrappers_f/abort_f08ts.F90
>>> >
>>> >
>>> > libtool: compile:  ifort -DHAVE_CONFIG_H -I. -I./src/include
>>> > -I./src/include -I./src/mpi/datatype -I./src/mpi/spawn
>>> > -Isrc/util/logging/common -I./src/util/wrappers -I./src/util/wrappers
>>> > -I./src/binding/cxx
>>> > -I/builddir/build/BUILD/mvapich/src/src/binding/fortran/mpif_h
>>> > -Isrc/binding/fortran/use_mpi
>>> >
>>> -I/builddir/build/BUILD/mvapich/src/src/binding/fortran/use_mpi_f08/wrappers_c
>>> > -I./src/mpid/ch3/include -I./src/mpid/ch3/include
>>> >
>>> -I/builddir/build/BUILD/mvapich/src/src/mpid/ch3/channels/common/include
>>> > -I./src/mpid/ch3/channels/psm/include -I./src/mpid/common/datatype
>>> > -I./src/mpid/common/datatype -I./src/mpid/common/sched
>>> > -I./src/mpid/common/thread -I./src/pmi/simple -Icontrib/hwloc/include/
>>> > -I./contrib/hwloc/include
>>> > -I/builddir/build/BUILD/mvapich/src/src/mpid/ch3/channels/psm/include
>>> > -I/builddir/build/BUILD/mvapich/src/src/mpid/ch3/channels/psm/include
>>> > -I/builddir/build/BUILD/mvapich/src/src/util/wrappers
>>> > -I/builddir/build/BUILD/mvapich/src/src/util/wrappers
>>> > -I/builddir/build/BUILD/mvapich/src/src/mpl/include
>>> > -I/builddir/build/BUILD/mvapich/src/src/mpl/include
>>> > -I/builddir/build/BUILD/mvapich/src/src/openpa/src
>>> > -I/builddir/build/BUILD/mvapich/src/src/openpa/src -D_REENTRANT
>>> > -I/builddir/build/BUILD/mvapich/src/src/mpi/romio/include
>>> > -Isrc/binding/fortran/use_mpi -Isrc/binding/fortran/use_mpi -g -O3 -g
>>> > -O2 -c src/binding/fortran/use_mpi_f08/wrappers_f/abort_f08ts.F90
>>> -fPIC
>>> > -o
>>> >
>>> src/binding/fortran/use_mpi_f08/wrappers_f/.libs/lib_libmpifort_la-abort_f08ts.o
>>> >
>>> >
>>> >
>>> > src/binding/fortran/use_mpi_f08/wrappers_f/abort_f08ts.F90(8): error
>>> > #7005: Error in reading the compiled module file.   [MPI_F08]
>>> >     use :: mpi_f08, only : MPI_Comm
>>> > -----------^
>>> > src/binding/fortran/use_mpi_f08/wrappers_f/abort_f08ts.F90(14): error
>>> > #6406: Conflicting attributes or multiple declaration of name.
>>>  [MPI_COMM]
>>> >     type(MPI_Comm), intent(in) :: comm
>>> > ---------^
>>> > src/binding/fortran/use_mpi_f08/wrappers_f/abort_f08ts.F90(18): error
>>> > #6683: A kind type parameter must be a compile-time constant.
>>>  [C_COMM]
>>> >     integer(c_Comm) :: comm_c
>>> > ------------^
>>> > src/binding/fortran/use_mpi_f08/wrappers_f/abort_f08ts.F90(23): error
>>> > #6404: This name does not have a type, and must have an explicit type.
>>> > [MPIR_ABORT_C]
>>> >         ierror_c = MPIR_Abort_c(comm%MPI_VAL, errorcode)
>>> > -------------------^
>>> > src/binding/fortran/use_mpi_f08/wrappers_f/abort_f08ts.F90(25): error
>>> > #6404: This name does not have a type, and must have an explicit type.
>>> > [COMM]
>>> >         comm_c = comm%MPI_VAL
>>> > -----------------^
>>> > src/binding/fortran/use_mpi_f08/wrappers_f/abort_f08ts.F90(25): error
>>> > #6460: This is not a field name that is defined in the encompassing
>>> > structure.   [MPI_VAL]
>>> >         comm_c = comm%MPI_VAL
>>> > ----------------------^
>>> > src/binding/fortran/use_mpi_f08/wrappers_f/abort_f08ts.F90(8): error
>>> > #6580: Name in only-list does not exist.   [MPI_COMM]
>>> >     use :: mpi_f08, only : MPI_Comm
>>> > ---------------------------^
>>> > src/binding/fortran/use_mpi_f08/wrappers_f/abort_f08ts.F90(9): error
>>> > #6580: Name in only-list does not exist.   [C_COMM]
>>> >     use :: mpi_c_interface, only : c_Comm
>>> > -----------------------------------^
>>> > src/binding/fortran/use_mpi_f08/wrappers_f/abort_f08ts.F90(10): error
>>> > #6580: Name in only-list does not exist.   [MPIR_ABORT_C]
>>> >     use :: mpi_c_interface, only : MPIR_Abort_c
>>> > -----------------------------------^
>>> > compilation aborted for
>>> > src/binding/fortran/use_mpi_f08/wrappers_f/abort_f08ts.F90 (code 1)
>>> > make[2]: ***
>>> >
>>> [src/binding/fortran/use_mpi_f08/wrappers_f/lib_libmpifort_la-abort_f08ts.lo]
>>> > Error 1
>>> >
>>> >
>>> > It seems that the compiler can't find the mpi_f08.mod file.  From
>>> > another build, I can see that is in
>>> > src/binding/fortran/use_mpi/mpi_f08.mod.
>>> > -Adam
>>> >
>>> >
>>> > Jonathan Perkins wrote:
>>> >
>>> >> Hi Kenneth.  Thanks for pointing out this commit!  It looks like this
>>> >> would
>>> >> be fix the issue that we've seen reported a few times on this list as
>>> >> well.  Once applied cleanly I'll verify that the issue is resolved and
>>> >> include this for our future releases.
>>> >>
>>> >> On Thu, Apr 21, 2016 at 5:37 PM Kenneth Raffenetti <
>>> raffenet at mcs.anl.gov
>>> <http://redir.aspx?REF=HHbjDcxiXSu9hQbS3gPJMfx1JM4635TzC0oMYF1r1P-6YRriq2vTCAFtYWlsdG86cmFmZmVuZXRAbWNzLmFubC5nb3Y.>
>>> >
>>> >> wrote:
>>> >>
>>> >>
>>> >>
>>> >>> On 04/21/2016 04:13 PM, Troy Baer wrote:
>>> >>>
>>> >>>> On 04/21/2016 05:09 PM, Adam T. Moody wrote:
>>> >>>>
>>> >>>>> Did you find an answer to this question?
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>
>>> http://mailman.cse.ohio-state.edu/pipermail/mvapich-discuss/2015-October/005722.html
>>> <http://redir.aspx?REF=QDaKZsEtaR4lcgxS2Qj1LD8zbrEqdEvyfB6mPRcRsAQbwxziq2vTCAFodHRwOi8vbWFpbG1hbi5jc2Uub2hpby1zdGF0ZS5lZHUvcGlwZXJtYWlsL212YXBpY2gtZGlzY3Vzcy8yMDE1LU9jdG9iZXIvMDA1NzIyLmh0bWw.>
>>> >>>
>>> >>>
>>> >>>>> When running my MV2-2.2 build through our buildfarm, I hit this
>>> same
>>> >>>>> problem, but things build fine for me outside of the buildfarm.  It
>>> >>>>> could be due to some missing rpm or other environmental problem,
>>> but I
>>> >>>>> can't login to the buildfarm node, so it's difficult to debug.
>>> I'll
>>> >>>>> keep chasing it, but if you know the answer, that would save me
>>> some
>>> >>>>> time.
>>> >>>>>
>>> >>>> No; in fact, I just tested for it in 2.2rc1 yesterday and found
>>> that it
>>> >>>> is still present.  AFAICT, it is related to out-of-tree builds,
>>> because
>>> >>>> in-tree builds seem to work.
>>> >>>>
>>> >>>>     --Troy
>>> >>>>
>>> >>>>
>>> >>> If it only happens for out-of-tree, it could be related to this fix:
>>> >>>
>>> >>>
>>> http://secure-web.cisco.com/1u62aCZ5KyvZOIIvNMthRsRqcPW4nz6fFiMyT0LK14_7e6D33dbvs5b-HGZf0wSU5acUX7IcpJPCvVwAybXooIoU8v-6PcvtyyP2P1h9JAMNtflERL8OMVtOIjHsCwo_ZiBG1eW41CbjVi3bxaiuDz8hCW13B77xR_fNhJvloi1zG_q_EDruJROL4dwg-ltyVswV7x2N5ydjwxNCTrQRZSYb5JrscmgDjSkqhwFqwhbXzH8wygsh7v9h734bYgOs6QRBqsEXtGqJ4jRzMLobUIzlizQw1BdKIjllmAI6qh2450acS8EIxseBQp6m-rizTf6Q_xx5V2U811LwIFmG-CPaXZh8KjZxXguB0P1W7fO4/http%3A%2F%2Fgit.mpich.org%2Fmpich.git%2Fcommitdiff%2F19908df8262c49544befc288093470a7beda731d
>>> <http://redir.aspx?REF=cXpvrIucEIU-DZRriqexMWflIVNyzokONMD_vJYkwnMbwxziq2vTCAFodHRwOi8vc2VjdXJlLXdlYi5jaXNjby5jb20vMXU2MmFDWjVLeXZaT0lJdk5NdGhSc1JxY1BXNG56NmZGaU15VDBMSzE0XzdlNkQzM2RidnM1Yi1IR1pmMHdTVTVhY1VYN0ljcEpQQ3ZWd0F5Ylhvb0lvVTh2LTZQY3Z0eXlQMlAxaDlKQU1OdGZsRVJMOE9NVnRPSWpIc0N3b19aaUJHMWVXNDFDYmpWaTNieGFpdUR6OGhDVzEzQjc3eFJfZk5oSnZsb2kxekdfcV9FRHJ1SlJPTDRkd2ctbHR5VnN3Vjd4Mk41eWRqd3hOQ1RyUVJaU1liNUpyc2NtZ0RqU2txaHdGcXdoYlh6SDh3eWdzaDd2OWg3MzRiWWdPczZRUkJxc0VYdEdxSjRqUnpNTG9iVUl6bGl6UXcxQmRLSWpsbG1BSTZxaDI0NTBhY1M4RUl4c2VCUXA2bS1yaXpUZjZRX3h4NVYyVTgxMUx3SUZtRy1DUGFYWmg4S2paeFhndUIwUDFXN2ZPNC9odHRwJTNBJTJGJTJGZ2l0Lm1waWNoLm9yZyUyRm1waWNoLmdpdCUyRmNvbW1pdGRpZmYlMkYxOTkwOGRmODI2MmM0OTU0NGJlZmMyODgwOTM0NzBhN2JlZGE3MzFk>
>>> >>>
>>> >>>
>>> >>> Ken
>>> >>> _______________________________________________
>>> >>> mvapich-discuss mailing list
>>> >>> mvapich-discuss at cse.ohio-state.edu
>>> <http://redir.aspx?REF=o9YUNScrWLG4SluUZgqTaInlnmnFWIJFZRhGKBNavvQbwxziq2vTCAFtYWlsdG86bXZhcGljaC1kaXNjdXNzQGNzZS5vaGlvLXN0YXRlLmVkdQ..>
>>> >>> http://mailman.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>>> <http://redir.aspx?REF=fcQIEy-uU12H5u_AxvtHPPDcICcy6MGshfgV99Osxp8bwxziq2vTCAFodHRwOi8vbWFpbG1hbi5jc2Uub2hpby1zdGF0ZS5lZHUvbWFpbG1hbi9saXN0aW5mby9tdmFwaWNoLWRpc2N1c3M.>
>>> >>>
>>> >>>
>>> >>
>>> >>
>>> >>
>>> >>
>>> ------------------------------------------------------------------------
>>> >>
>>> >> _______________________________________________
>>> >> mvapich-discuss mailing list
>>> >> mvapich-discuss at cse.ohio-state.edu
>>> <http://redir.aspx?REF=o9YUNScrWLG4SluUZgqTaInlnmnFWIJFZRhGKBNavvQbwxziq2vTCAFtYWlsdG86bXZhcGljaC1kaXNjdXNzQGNzZS5vaGlvLXN0YXRlLmVkdQ..>
>>> >> http://mailman.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>>> <http://redir.aspx?REF=fcQIEy-uU12H5u_AxvtHPPDcICcy6MGshfgV99Osxp8bwxziq2vTCAFodHRwOi8vbWFpbG1hbi5jc2Uub2hpby1zdGF0ZS5lZHUvbWFpbG1hbi9saXN0aW5mby9tdmFwaWNoLWRpc2N1c3M.>
>>> >>
>>> >>
>>> >
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20160423/25b3d54c/attachment-0001.html>

More information about the mvapich-discuss mailing list