[mvapich-discuss] MVAPICH2 1.9b build is broken with Clang

[Dmitri Gribenko]

On Thu, Apr 11, 2013 at 2:28 AM, Jonathan Perkins
<perkinjo at cse.ohio-state.edu> wrote:
> This is error should be resolved by the attached patch.

Hello Jonathan,

The patch fixed all compilation issues for me!  Thank you!

But now I have issues with linking ScaLAPACK when MVAPICH2 is
configured without --enable-shared* options.  I can reproduce the same
issue with this short program:

$ cat hh.c
#include <mpi.h>

void *foo = MPI_Comm_create;
void *bar = PMPI_Comm_create;

int main() {}

$ mpicc ~/hh.c
/opt/eb/software/MVAPICH2/1.9b-ClangGCC-1.1.2/lib/libmpich.a(comm_create.o):(.data+0x0):
multiple definition of `split_comm'
/opt/eb/software/MVAPICH2/1.9b-ClangGCC-1.1.2/lib/libpmpich.a(lib_libpmpich_la-comm_create.o):(.data+0x0):
first defined here
clang-3: error: linker command failed with exit code 1 (use -v to see
invocation)

Here's some more information to help debugging:

$ mpicc -show ~/hh.c
clang -O2 -march=native -L/lib -L/lib
-L/opt/eb/software/GCC/4.7.2/lib64 -L/lib -Wl,-rpath,/lib -L/lib
-Wl,-rpath,/lib -L/lib -L/lib /home/easybuild/hh.c
-I/opt/eb/software/MVAPICH2/1.9b-ClangGCC-1.1.2/include
-L/opt/eb/software/MVAPICH2/1.9b-ClangGCC-1.1.2/lib -lpmpich -lmpich
-lopa -lmpl -libmad -libumad -libverbs -ldl -lrt -lm -lpthread

'split_comm' variable is indeed defined twice:

$ nm /opt/eb/software/MVAPICH2/1.9b-ClangGCC-1.1.2/lib/libmpich.a
...
comm_create.o:
                 U MPIC_Sendrecv
                 U MPID_Comm_builtin
                 U MPID_Comm_direct
                 U MPID_Comm_mem
                 U MPID_Group_builtin
                 U MPID_Group_direct
                 U MPID_Group_mem
                 U MPID_VCRT_Create
                 U MPID_VCRT_Get_ptr
                 U MPID_VCR_Dup
                 U MPID_VCR_Get_lpid
                 U MPIR_Bcast_impl
                 U MPIR_Comm_commit
                 U MPIR_Comm_create
0000000000000000 T MPIR_Comm_create_calculate_mapping
00000000000001c0 T MPIR_Comm_create_create_and_map_vcrt
0000000000000410 T MPIR_Comm_create_inter
0000000000000240 T MPIR_Comm_create_intra
                 U MPIR_Comm_delete_internal
                 U MPIR_Err_return_comm
                 U MPIR_Free_contextid
                 U MPIR_Get_contextid
                 U MPIR_Group_setup_lpid_list
                 U MPIR_Process
                 U MPIR_Setup_intercomm_localcomm
                 U MPIR_Thread
                 U MPIR_ThreadInfo
                 U MPIU_Handle_get_ptr_indirect
0000000000000950 T PMPI_Comm_create
                 U _GLOBAL_OFFSET_TABLE_
                 U free
                 U malloc
                 U pthread_mutex_lock
                 U pthread_mutex_unlock
0000000000000008 C reduce_fn
0000000000000000 D split_comm  // here
...

$ nm /opt/eb/software/MVAPICH2/1.9b-ClangGCC-1.1.2/lib/libpmpich.a
...
lib_libpmpich_la-comm_create.o:
                 U MPID_Comm_builtin
                 U MPID_Comm_direct
                 U MPID_Comm_mem
                 U MPID_Group_builtin
                 U MPID_Group_direct
                 U MPID_Group_mem
                 U MPIR_Comm_create_inter
                 U MPIR_Comm_create_intra
                 U MPIR_Err_return_comm
                 U MPIR_Thread
                 U MPIR_ThreadInfo
                 U MPIU_Handle_get_ptr_indirect
0000000000000000 T MPI_Comm_create
                 U _GLOBAL_OFFSET_TABLE_
                 U pthread_mutex_lock
                 U pthread_mutex_unlock
0000000000000008 C reduce_fn
0000000000000000 D split_comm // and here
...

With dynamic libraries (--enable-shared --enable-sharedlibs=gcc)
multiple definitions are still there:
$ nm /opt/eb/software/MVAPICH2/1.9b-ClangGCC-1.1.2/lib/libpmpich.so |
grep "D split_comm"
00000000002df448 D split_comm

$ nm /opt/eb/software/MVAPICH2/1.9b-ClangGCC-1.1.2/lib/libmpich.so |
grep "D split_comm"
0000000000402688 D split_comm

But dynamic linking seems to work around this somehow, and there are
no errors while compiling and linking the above 'hh.c' program.
ScaLAPACK builds fine with dynamic libraries, too.

Anyway, multiple definitions are scary, and ignoring the issue is not
the best option, so I decided to write an email about this.

Dmitri

--
main(i,j){for(i=2;;i++){for(j=2;j<i;j++){if(!(i%j)){j=0;break;}}if
(j){printf("%d\n",i);}}} /*Dmitri Gribenko <gribozavr at gmail.com>*/