[mvapich-discuss] MVAPICH2-GDR/2.2-2 problems on RHEL 7.3 system

Fri Apr 21 08:30:04 EDT 2017

Hi Hari,

Carl Ponder found the following link that seems similar to what I am seeing now, but this was some time ago and was with RHEL 6:

http://mailman.cse.ohio-state.edu/pipermail/mvapich-discuss/2014-February/004810.html

I was wondering if that may give you some clues?

Thanks,

Raghu

From: Raghu Reddy [mailto:raghu.reddy at noaa.gov] 
Sent: Thursday, April 20, 2017 7:28 PM
To: 'Hari Subramoni'
Cc: mvapich-discuss at cse.ohio-state.edu; 'Raghu Reddy'
Subject: RE: [mvapich-discuss] MVAPICH2-GDR/2.2-2 problems on RHEL 7.3 system

Hi Hari,

We are using the open source OFED included with RHEL 7.3.

Thanks,

Raghu

From: hari.subramoni at gmail.com [mailto:hari.subramoni at gmail.com] On Behalf Of Hari Subramoni
Sent: Thursday, April 20, 2017 5:22 PM
To: Raghu Reddy <raghu.reddy at noaa.gov>
Cc: mvapich-discuss at cse.ohio-state.edu
Subject: Re: [mvapich-discuss] MVAPICH2-GDR/2.2-2 problems on RHEL 7.3 system

Hi Dr. Reddy,

It looks like the version of OFED you have installed on the system does not support XRC. Could you please let us know what version OFED you've installed on the system? Is it MOFED 3.2?

Best Regards,

Hari.

On Thu, Apr 20, 2017 at 4:46 PM, Raghu Reddy <raghu.reddy at noaa.gov> wrote:

Hi all,

We are in the process of upgrading our system to RHEL 7.3, and are having problems trying to compile a simple program with the mvapich2-gdr/2.2-2 library.

Our production system is running the following OS:

-          Red Hat Enterprise Linux Server release 6.8 (Santiago)

And we’ve been running mvapich2-gdr/2.2.1 on that system for some time now.

However, now we’re in the process of upgrading to RHEL 7.3 and have that installed on a smaller test system:

System information for the test system:

-          Each node consists of 2 Haswell processors with 10 cores each.

-          Each node has 8 Tesla P100 (Pascal) GPUs.

-          The interconnect is using 1 Mellanox Connect-X 3 IB card connected to socket 1.

-          Red Hat Enterprise Linux Server release 7.3 (Maipo)

Initially I tried to use the existing mvapich2-gdr/2.2.1 and that one failed to compile a simple program.  So I went ahead and downloaded the from RHEL/CENTOS 7 secction of the mvapich2-gdr 2.2 library.  Since there was only one option available that was suitable for us with the following combination:

-          Intel 16.0.2 without SLURM

-          CUDA 8.0

I downloaded the version from the MLNX-OFED 3.2 row in that table.

The error I am getting when trying to compile it is the following:

sg001% module purge

sg001% module load intel/16.1.150 cuda/8.0 mvapich2-gdr/2.2-2-cuda-8.0-intel

sg001% 

sg001% mpicc -g -o osu_bibw osu_bibw.c

/apps/mvapich2-gdr-cuda8.0-intel/2.2-2/lib64/libmpi.so: undefined reference to `ibv_modify_xrc_rcv_qp at IBVERBS_1.1'

/apps/mvapich2-gdr-cuda8.0-intel/2.2-2/lib64/libmpi.so: undefined reference to `ibv_unreg_xrc_rcv_qp at IBVERBS_1.1'

/apps/mvapich2-gdr-cuda8.0-intel/2.2-2/lib64/libmpi.so: undefined reference to `ibv_open_xrc_domain at IBVERBS_1.1'

/apps/mvapich2-gdr-cuda8.0-intel/2.2-2/lib64/libmpi.so: undefined reference to `ibv_create_xrc_srq at IBVERBS_1.1'

/apps/mvapich2-gdr-cuda8.0-intel/2.2-2/lib64/libmpi.so: undefined reference to `ibv_close_xrc_domain at IBVERBS_1.1'

/apps/mvapich2-gdr-cuda8.0-intel/2.2-2/lib64/libmpi.so: undefined reference to `ibv_reg_xrc_rcv_qp at IBVERBS_1.1'

/apps/mvapich2-gdr-cuda8.0-intel/2.2-2/lib64/libmpi.so: undefined reference to `ibv_create_xrc_rcv_qp at IBVERBS_1.1'

sg001%

-          One question is why is this specifically trying to link to ibverbs_1.1?

-          Are we missing some configuration step?

-          Are we missing any of the RPMs?

Any advice his very much appreciated!

I’m including the output from the verbose option below.

Thanks,

Raghu

=================== verbose output ==================

mpicc for MVAPICH2 version 2.2

icc version 16.0.1 (gcc version 4.8.5 compatibility)

/apps/intel/compilers_and_libraries_2016.1.150/linux/bin/intel64/mcpcom    --target_efi2 --lang=c -_g -mP3OPT_inline_alloca -D__ICC=1600 -D__

INTEL_COMPILER=1600 -D__INTEL_COMPILER_UPDATE=1 -D__PTRDIFF_TYPE__=long "-D__SIZE_TYPE__=unsigned long" -D__WCHAR_TYPE__=int "-D__WINT_TYPE__

=unsigned int" "-D__INTMAX_TYPE__=long int" "-D__UINTMAX_TYPE__=long unsigned int" -D__LONG_MAX__=9223372036854775807L -D__QMSPP_ -D__OPTIMIZ

E__ -D__NO_MATH_INLINES -D__NO_STRING_INLINES -D__GNUC_GNU_INLINE__ -D__GNUC__=4 -D__GNUC_MINOR__=8 -D__GNUC_PATCHLEVEL__=5 -D__LP64__ -D_LP6

4 -D__GXX_ABI_VERSION=1002 "-D__USER_LABEL_PREFIX__= " -D__REGISTER_PREFIX__= -D__INTEL_RTTI__ -D__EXCEPTIONS=1 -D__unix__ -D__unix -D__linux

__ -D__linux -D__gnu_linux__ -B -Dunix -Dlinux "-_Asystem(unix)" -D__ELF__ -D__x86_64 -D__x86_64__ -D__amd64 -D__amd64__ "-_Acpu(x86_64)" "-_

Amachine(x86_64)" -D__INTEL_COMPILER_BUILD_DATE=20151021 -D__INTEL_OFFLOAD -D__i686 -D__i686__ -D__pentiumpro -D__pentiumpro__ -D__pentium4 -

D__pentium4__ -D__tune_pentium4__ -D__SSE2__ -D__SSE2_MATH__ -D__SSE__ -D__SSE_MATH__ -D__MMX__ -_k -_8 -_l --has_new_stdarg_support -_a -_b 

--gnu_version=40805 -_W5 --gcc-extern-inline --c_exceptions --multibyte_chars -mGLOB_diag_suppress_sys -I/usr/local/cuda-8.0/include -I/apps/

mvapich2-gdr-cuda8.0-intel/2.2-2/include --array_section --simd --simd_func --offload_mode=1 --offload_target_names=gfx,GFX,mic,MIC --offload

_unique_string=icc010330396754DpBdBk -D_FORTIFY_SOURCE=2 -mGLOB_em64t=TRUE -mP1OPT_version=16.0-intel64 -mGLOB_diag_enable_disable=E:level1 -

mGLOB_diag_file=/tmp/iccAPK0ST.diag -mP1OPT_print_version=FALSE -mCG_use_gas_got_workaround=F -mP2OPT_align_option_used=TRUE -mGLOB_gcc_versi

on=485 "-mGLOB_options_string=-I/usr/local/cuda-8.0/include -I/apps/mvapich2-gdr-cuda8.0-intel/2.2-2/include -O2 -g -pipe -Wall -Wp,-D_FORTIF

Y_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -v -g -o osu_bibw -L/app

s/mvapich2-gdr-cuda8.0-intel/2.2-2/lib64 -Wl,-rpath -Wl,/apps/mvapich2-gdr-cuda8.0-intel/2.2-2/lib64 -Wl,--enable-new-dtags -lmpi" -mGLOB_cxx

_limited_range=FALSE -mCG_extend_parms=FALSE -mGLOB_compiler_bin_directory=/apps/intel/compilers_and_libraries_2016.1.150/linux/bin/intel64 -

mP3OPT_emit_line_numbers -mGLOB_debug_target=GLOB_DEBUG_TARGET_ALL -mDEBUG_record_switches -mDEBUG_info_level=2 -mDEBUG_use_indirect_strings=

TRUE -mIPOPT_ninl_debug_info=TRUE -mDEBUG_emit_dwarf_inline_info=TRUE -mDEBUG_debug_ranges=TRUE -mGLOB_debug_format=GLOB_DEBUG_FORMAT_DWARF30

-mGLOB_as_output_backup_file_name=/tmp/iccyBC07bas_.s -mGLOB_dashboard_use_source_name -mIPOPT_activate -mIPOPT_lite -mGLOB_instruction_tuni

ng=0x0 -mGLOB_product_id_code=0x22006d91 -mCG_bnl_movbe=T -mGLOB_extended_instructions=0x8 -mP3OPT_use_mspp_call_convention -mP2OPT_subs_out_

of_bound=FALSE -mP2OPT_disam_type_based_disam=2 -mP2OPT_disam_assume_ansi_c -mP2OPT_checked_disam_ansi_alias=TRUE -mGLOB_ansi_alias -mPGOPTI_

value_profile_use=T -mGLOB_opt_report_use_source_name -mCG_stack_security_check=0x75 -mP2OPT_il0_array_sections=TRUE -mGLOB_offload_mode=1 -m

P2OPT_offload_unique_var_string=icc010330396754DpBdBk -mGLOB_opt_level=2 -mP2OPT_hlo_level=2 -mP2OPT_hlo -mP2OPT_hpo_rtt_control=0 -mIPOPT_ar

gs_in_regs=0 -mP2OPT_disam_assume_nonstd_intent_in=FALSE -mGLOB_imf_mapping_library=/apps/intel/compilers_and_libraries_2016.1.150/linux/bin/

intel64/libiml_attr.so -mIPOPT_single_file_compile_and_link=TRUE -mP2OPT_hlo_embed_loopinfo -mPGOPTI_gen_threadsafe_level=0 -mIPOPT_lto_objec

t_enabled -mIPOPT_lto_object_value=1 -mIPOPT_obj_output_file_name=/tmp/iccAPK0ST.o -mIPOPT_whole_archive_fixup_file_name=/tmp/iccwarch9YWwnu 

-mGLOB_linker_version=2.25.1 -mGLOB_long_size_64 -mGLOB_routine_pointer_size_64 -mGLOB_driver_tempfile_name=/tmp/icctempfileLHGnvD -mP3OPT_as

m_target=P3OPT_ASM_TARGET_GAS -mGLOB_async_unwind_tables=TRUE -mGLOB_obj_output_file=/tmp/iccAPK0ST.o -mGLOB_source_dialect=GLOB_SOURCE_DIALE

CT_C -mP1OPT_source_file_name=osu_bibw.c -mGLOB_eh_c_linux osu_bibw.c

#include "..." search starts here:

#include <...> search starts here:

/apps/mvapich2-gdr-cuda8.0-intel/2.2-2/include

/apps/mvapich2-gdr-cuda8.0-intel/2.2-2/include

/apps/cuda/cuda-8.0/include

/apps/intel/compilers_and_libraries_2016.1.150/linux/ipp/include

/apps/intel/compilers_and_libraries_2016.1.150/linux/tbb/include

/apps/intel/compilers_and_libraries_2016.1.150/linux/compiler/include/intel64

/apps/intel/compilers_and_libraries_2016.1.150/linux/compiler/include

/usr/local/include

/usr/lib/gcc/x86_64-redhat-linux/4.8.5/include

/usr/include/

/usr/include

End of search list.

ld    /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64/crt1.o /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64/crti.o /usr/li

b/gcc/x86_64-redhat-linux/4.8.5/crtbegin.o --eh-frame-hdr --build-id -dynamic-linker /lib64/ld-linux-x86-64.so.2 -m elf_x86_64 -L/apps/mvapic

h2-gdr-cuda8.0-intel/2.2-2/lib64 -o osu_bibw -L/apps/intel/compilers_and_libraries_2016.1.150/linux/compiler/lib/intel64 -L/apps/intel/compil

ers_and_libraries_2016.1.150/linux/ipp/intel64 -L/apps/intel/compilers_and_libraries_2016.1.150/linux/mkl/lib/intel64 -L/apps/intel/compilers

_and_libraries_2016.1.150/linux/tbb/lib/intel64/gcc4.4 -L/apps/intel/compilers_and_libraries_2016.1.150/linux/compiler/lib/intel64_lin -L/usr

/lib/gcc/x86_64-redhat-linux/4.8.5/ -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../..

/../../lib64/ -L/lib/../lib64 -L/lib/../lib64/ -L/usr/lib/../lib64 -L/usr/lib/../lib64/ -L/apps/intel/compilers_and_libraries_2016.1.150/linu

x/compiler/lib/intel64/ -L/apps/intel/compilers_and_libraries_2016.1.150/linux/mkl/lib/intel64/ -L/apps/intel/compilers_and_libraries_2016.1.

150/linux/tbb/lib/intel64/gcc4.4/ -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../ -L/lib64 -L/lib/ -L/usr/lib64 -L/usr/lib /tmp/iccAPK0ST.

o -rpath /apps/mvapich2-gdr-cuda8.0-intel/2.2-2/lib64 --enable-new-dtags -lmpi -Bdynamic -Bstatic -limf -lsvml -lirng -Bdynamic -lm -Bstatic 

-lipgo -ldecimal --as-needed -Bdynamic -lcilkrts -lstdc++ --no-as-needed -lgcc -lgcc_s -Bstatic -lirc -lsvml -Bdynamic -lc -lgcc -lgcc_s -Bst

atic -lirc_s -Bdynamic -ldl -lc /usr/lib/gcc/x86_64-redhat-linux/4.8.5/crtend.o /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64/crtn

.o

/apps/mvapich2-gdr-cuda8.0-intel/2.2-2/lib64/libmpi.so: undefined reference to `ibv_modify_xrc_rcv_qp at IBVERBS_1.1'

/apps/mvapich2-gdr-cuda8.0-intel/2.2-2/lib64/libmpi.so: undefined reference to `ibv_unreg_xrc_rcv_qp at IBVERBS_1.1'

/apps/mvapich2-gdr-cuda8.0-intel/2.2-2/lib64/libmpi.so: undefined reference to `ibv_open_xrc_domain at IBVERBS_1.1'

/apps/mvapich2-gdr-cuda8.0-intel/2.2-2/lib64/libmpi.so: undefined reference to `ibv_create_xrc_srq at IBVERBS_1.1'

/apps/mvapich2-gdr-cuda8.0-intel/2.2-2/lib64/libmpi.so: undefined reference to `ibv_close_xrc_domain at IBVERBS_1.1'

/apps/mvapich2-gdr-cuda8.0-intel/2.2-2/lib64/libmpi.so: undefined reference to `ibv_reg_xrc_rcv_qp at IBVERBS_1.1'

/apps/mvapich2-gdr-cuda8.0-intel/2.2-2/lib64/libmpi.so: undefined reference to `ibv_create_xrc_rcv_qp at IBVERBS_1.1'

sg001%

_______________________________________________
mvapich-discuss mailing list
mvapich-discuss at cse.ohio-state.edu
http://mailman.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20170421/a8a1bac8/attachment-0001.html>