[mvapich-discuss] MVAPICH2-GDR/2.2-2 problems on RHEL 7.3 system

Thu Apr 20 16:46:58 EDT 2017

Hi all,

We are in the process of upgrading our system to RHEL 7.3, and are having
problems trying to compile a simple program with the mvapich2-gdr/2.2-2
library.

Our production system is running the following OS:

-          Red Hat Enterprise Linux Server release 6.8 (Santiago)

And we've been running mvapich2-gdr/2.2.1 on that system for some time now.

However, now we're in the process of upgrading to RHEL 7.3 and have that
installed on a smaller test system:

System information for the test system:

-          Each node consists of 2 Haswell processors with 10 cores each.

-          Each node has 8 Tesla P100 (Pascal) GPUs.

-          The interconnect is using 1 Mellanox Connect-X 3 IB card
connected to socket 1.

-          Red Hat Enterprise Linux Server release 7.3 (Maipo)

Initially I tried to use the existing mvapich2-gdr/2.2.1 and that one failed
to compile a simple program.  So I went ahead and downloaded the from
RHEL/CENTOS 7 secction of the mvapich2-gdr 2.2 library.  Since there was
only one option available that was suitable for us with the following
combination:

-          Intel 16.0.2 without SLURM

-          CUDA 8.0

I downloaded the version from the MLNX-OFED 3.2 row in that table.

The error I am getting when trying to compile it is the following:

sg001% module purge

sg001% module load intel/16.1.150 cuda/8.0 mvapich2-gdr/2.2-2-cuda-8.0-intel

sg001% 

sg001% mpicc -g -o osu_bibw osu_bibw.c

/apps/mvapich2-gdr-cuda8.0-intel/2.2-2/lib64/libmpi.so: undefined reference
to `ibv_modify_xrc_rcv_qp at IBVERBS_1.1'

/apps/mvapich2-gdr-cuda8.0-intel/2.2-2/lib64/libmpi.so: undefined reference
to `ibv_unreg_xrc_rcv_qp at IBVERBS_1.1'

/apps/mvapich2-gdr-cuda8.0-intel/2.2-2/lib64/libmpi.so: undefined reference
to `ibv_open_xrc_domain at IBVERBS_1.1'

/apps/mvapich2-gdr-cuda8.0-intel/2.2-2/lib64/libmpi.so: undefined reference
to `ibv_create_xrc_srq at IBVERBS_1.1'

/apps/mvapich2-gdr-cuda8.0-intel/2.2-2/lib64/libmpi.so: undefined reference
to `ibv_close_xrc_domain at IBVERBS_1.1'

/apps/mvapich2-gdr-cuda8.0-intel/2.2-2/lib64/libmpi.so: undefined reference
to `ibv_reg_xrc_rcv_qp at IBVERBS_1.1'

/apps/mvapich2-gdr-cuda8.0-intel/2.2-2/lib64/libmpi.so: undefined reference
to `ibv_create_xrc_rcv_qp at IBVERBS_1.1'

sg001%

-          One question is why is this specifically trying to link to
ibverbs_1.1?

-          Are we missing some configuration step?

-          Are we missing any of the RPMs?

Any advice his very much appreciated!

I'm including the output from the verbose option below.

Thanks,

Raghu

=================== verbose output ==================

mpicc for MVAPICH2 version 2.2

icc version 16.0.1 (gcc version 4.8.5 compatibility)

/apps/intel/compilers_and_libraries_2016.1.150/linux/bin/intel64/mcpcom
--target_efi2 --lang=c -_g -mP3OPT_inline_alloca -D__ICC=1600 -D__

INTEL_COMPILER=1600 -D__INTEL_COMPILER_UPDATE=1 -D__PTRDIFF_TYPE__=long
"-D__SIZE_TYPE__=unsigned long" -D__WCHAR_TYPE__=int "-D__WINT_TYPE__

=unsigned int" "-D__INTMAX_TYPE__=long int" "-D__UINTMAX_TYPE__=long
unsigned int" -D__LONG_MAX__=9223372036854775807L -D__QMSPP_ -D__OPTIMIZ

E__ -D__NO_MATH_INLINES -D__NO_STRING_INLINES -D__GNUC_GNU_INLINE__
-D__GNUC__=4 -D__GNUC_MINOR__=8 -D__GNUC_PATCHLEVEL__=5 -D__LP64__ -D_LP6

4 -D__GXX_ABI_VERSION=1002 "-D__USER_LABEL_PREFIX__= "
-D__REGISTER_PREFIX__= -D__INTEL_RTTI__ -D__EXCEPTIONS=1 -D__unix__ -D__unix
-D__linux

__ -D__linux -D__gnu_linux__ -B -Dunix -Dlinux "-_Asystem(unix)" -D__ELF__
-D__x86_64 -D__x86_64__ -D__amd64 -D__amd64__ "-_Acpu(x86_64)" "-_

Amachine(x86_64)" -D__INTEL_COMPILER_BUILD_DATE=20151021 -D__INTEL_OFFLOAD
-D__i686 -D__i686__ -D__pentiumpro -D__pentiumpro__ -D__pentium4 -

D__pentium4__ -D__tune_pentium4__ -D__SSE2__ -D__SSE2_MATH__ -D__SSE__
-D__SSE_MATH__ -D__MMX__ -_k -_8 -_l --has_new_stdarg_support -_a -_b 

--gnu_version=40805 -_W5 --gcc-extern-inline --c_exceptions
--multibyte_chars -mGLOB_diag_suppress_sys -I/usr/local/cuda-8.0/include
-I/apps/

mvapich2-gdr-cuda8.0-intel/2.2-2/include --array_section --simd --simd_func
--offload_mode=1 --offload_target_names=gfx,GFX,mic,MIC --offload

_unique_string=icc010330396754DpBdBk -D_FORTIFY_SOURCE=2 -mGLOB_em64t=TRUE
-mP1OPT_version=16.0-intel64 -mGLOB_diag_enable_disable=E:level1 -

mGLOB_diag_file=/tmp/iccAPK0ST.diag -mP1OPT_print_version=FALSE
-mCG_use_gas_got_workaround=F -mP2OPT_align_option_used=TRUE
-mGLOB_gcc_versi

on=485 "-mGLOB_options_string=-I/usr/local/cuda-8.0/include
-I/apps/mvapich2-gdr-cuda8.0-intel/2.2-2/include -O2 -g -pipe -Wall
-Wp,-D_FORTIF

Y_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4
-grecord-gcc-switches -m64 -mtune=generic -v -g -o osu_bibw -L/app

s/mvapich2-gdr-cuda8.0-intel/2.2-2/lib64 -Wl,-rpath
-Wl,/apps/mvapich2-gdr-cuda8.0-intel/2.2-2/lib64 -Wl,--enable-new-dtags
-lmpi" -mGLOB_cxx

_limited_range=FALSE -mCG_extend_parms=FALSE
-mGLOB_compiler_bin_directory=/apps/intel/compilers_and_libraries_2016.1.150
/linux/bin/intel64 -

mP3OPT_emit_line_numbers -mGLOB_debug_target=GLOB_DEBUG_TARGET_ALL
-mDEBUG_record_switches -mDEBUG_info_level=2 -mDEBUG_use_indirect_strings=

TRUE -mIPOPT_ninl_debug_info=TRUE -mDEBUG_emit_dwarf_inline_info=TRUE
-mDEBUG_debug_ranges=TRUE -mGLOB_debug_format=GLOB_DEBUG_FORMAT_DWARF30

-mGLOB_as_output_backup_file_name=/tmp/iccyBC07bas_.s
-mGLOB_dashboard_use_source_name -mIPOPT_activate -mIPOPT_lite
-mGLOB_instruction_tuni

ng=0x0 -mGLOB_product_id_code=0x22006d91 -mCG_bnl_movbe=T
-mGLOB_extended_instructions=0x8 -mP3OPT_use_mspp_call_convention
-mP2OPT_subs_out_

of_bound=FALSE -mP2OPT_disam_type_based_disam=2 -mP2OPT_disam_assume_ansi_c
-mP2OPT_checked_disam_ansi_alias=TRUE -mGLOB_ansi_alias -mPGOPTI_

value_profile_use=T -mGLOB_opt_report_use_source_name
-mCG_stack_security_check=0x75 -mP2OPT_il0_array_sections=TRUE
-mGLOB_offload_mode=1 -m

P2OPT_offload_unique_var_string=icc010330396754DpBdBk -mGLOB_opt_level=2
-mP2OPT_hlo_level=2 -mP2OPT_hlo -mP2OPT_hpo_rtt_control=0 -mIPOPT_ar

gs_in_regs=0 -mP2OPT_disam_assume_nonstd_intent_in=FALSE
-mGLOB_imf_mapping_library=/apps/intel/compilers_and_libraries_2016.1.150/li
nux/bin/

intel64/libiml_attr.so -mIPOPT_single_file_compile_and_link=TRUE
-mP2OPT_hlo_embed_loopinfo -mPGOPTI_gen_threadsafe_level=0 -mIPOPT_lto_objec

t_enabled -mIPOPT_lto_object_value=1
-mIPOPT_obj_output_file_name=/tmp/iccAPK0ST.o
-mIPOPT_whole_archive_fixup_file_name=/tmp/iccwarch9YWwnu 

-mGLOB_linker_version=2.25.1 -mGLOB_long_size_64
-mGLOB_routine_pointer_size_64
-mGLOB_driver_tempfile_name=/tmp/icctempfileLHGnvD -mP3OPT_as

m_target=P3OPT_ASM_TARGET_GAS -mGLOB_async_unwind_tables=TRUE
-mGLOB_obj_output_file=/tmp/iccAPK0ST.o
-mGLOB_source_dialect=GLOB_SOURCE_DIALE

CT_C -mP1OPT_source_file_name=osu_bibw.c -mGLOB_eh_c_linux osu_bibw.c

#include "..." search starts here:

#include <...> search starts here:

/apps/mvapich2-gdr-cuda8.0-intel/2.2-2/include

/apps/mvapich2-gdr-cuda8.0-intel/2.2-2/include

/apps/cuda/cuda-8.0/include

/apps/intel/compilers_and_libraries_2016.1.150/linux/ipp/include

/apps/intel/compilers_and_libraries_2016.1.150/linux/tbb/include

/apps/intel/compilers_and_libraries_2016.1.150/linux/compiler/include/intel6
4

/apps/intel/compilers_and_libraries_2016.1.150/linux/compiler/include

/usr/local/include

/usr/lib/gcc/x86_64-redhat-linux/4.8.5/include

/usr/include/

/usr/include

End of search list.

ld    /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64/crt1.o
/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64/crti.o /usr/li

b/gcc/x86_64-redhat-linux/4.8.5/crtbegin.o --eh-frame-hdr --build-id
-dynamic-linker /lib64/ld-linux-x86-64.so.2 -m elf_x86_64 -L/apps/mvapic

h2-gdr-cuda8.0-intel/2.2-2/lib64 -o osu_bibw
-L/apps/intel/compilers_and_libraries_2016.1.150/linux/compiler/lib/intel64
-L/apps/intel/compil

ers_and_libraries_2016.1.150/linux/ipp/intel64
-L/apps/intel/compilers_and_libraries_2016.1.150/linux/mkl/lib/intel64
-L/apps/intel/compilers

_and_libraries_2016.1.150/linux/tbb/lib/intel64/gcc4.4
-L/apps/intel/compilers_and_libraries_2016.1.150/linux/compiler/lib/intel64_
lin -L/usr

/lib/gcc/x86_64-redhat-linux/4.8.5/
-L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64
-L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../..

/../../lib64/ -L/lib/../lib64 -L/lib/../lib64/ -L/usr/lib/../lib64
-L/usr/lib/../lib64/ -L/apps/intel/compilers_and_libraries_2016.1.150/linu

x/compiler/lib/intel64/
-L/apps/intel/compilers_and_libraries_2016.1.150/linux/mkl/lib/intel64/
-L/apps/intel/compilers_and_libraries_2016.1.

150/linux/tbb/lib/intel64/gcc4.4/
-L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../ -L/lib64 -L/lib/
-L/usr/lib64 -L/usr/lib /tmp/iccAPK0ST.

o -rpath /apps/mvapich2-gdr-cuda8.0-intel/2.2-2/lib64 --enable-new-dtags
-lmpi -Bdynamic -Bstatic -limf -lsvml -lirng -Bdynamic -lm -Bstatic 

-lipgo -ldecimal --as-needed -Bdynamic -lcilkrts -lstdc++ --no-as-needed
-lgcc -lgcc_s -Bstatic -lirc -lsvml -Bdynamic -lc -lgcc -lgcc_s -Bst

atic -lirc_s -Bdynamic -ldl -lc
/usr/lib/gcc/x86_64-redhat-linux/4.8.5/crtend.o
/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64/crtn

.o

/apps/mvapich2-gdr-cuda8.0-intel/2.2-2/lib64/libmpi.so: undefined reference
to `ibv_modify_xrc_rcv_qp at IBVERBS_1.1'

/apps/mvapich2-gdr-cuda8.0-intel/2.2-2/lib64/libmpi.so: undefined reference
to `ibv_unreg_xrc_rcv_qp at IBVERBS_1.1'

/apps/mvapich2-gdr-cuda8.0-intel/2.2-2/lib64/libmpi.so: undefined reference
to `ibv_open_xrc_domain at IBVERBS_1.1'

/apps/mvapich2-gdr-cuda8.0-intel/2.2-2/lib64/libmpi.so: undefined reference
to `ibv_create_xrc_srq at IBVERBS_1.1'

/apps/mvapich2-gdr-cuda8.0-intel/2.2-2/lib64/libmpi.so: undefined reference
to `ibv_close_xrc_domain at IBVERBS_1.1'

/apps/mvapich2-gdr-cuda8.0-intel/2.2-2/lib64/libmpi.so: undefined reference
to `ibv_reg_xrc_rcv_qp at IBVERBS_1.1'

/apps/mvapich2-gdr-cuda8.0-intel/2.2-2/lib64/libmpi.so: undefined reference
to `ibv_create_xrc_rcv_qp at IBVERBS_1.1'

sg001%

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20170420/95989b43/attachment-0001.html>