[mvapich-discuss] Patch to mvapich spec file to improve performance when using the QLogic PSM interface

[Jonathan Perkins]

Thank you for the patch.  OFED is reaching the final stage before
their next release so I do not think this can be put there.  If there
is a further release, we'll apply these changes there.

On Wed, Nov 30, 2011 at 12:45 PM, Mike Heinz <michael.heinz at qlogic.com> wrote:
> Hey,
>
> This patch to the spec file changes the compile options when building against the QLogic PSM library instead of verbs, I hope you find it useful:
>
> Index: mvapich.spec
> ===================================================================
> RCS file: /cvs/vendor/OFA/Ofed/mvapich/mvapich.spec,v
> retrieving revision 1.1.1.9
> retrieving revision 1.34
> diff -w -u -d -r1.1.1.9 -r1.34
> --- mvapich.spec        29 Apr 2010 21:11:46 -0000      1.1.1.9
> +++ mvapich.spec        15 Nov 2011 17:40:17 -0000      1.34
> @@ -245,7 +245,11 @@
>  IB_INCLUDE=%{openib_prefix}/include
>  IB_LIB=%{openib_prefix}/lib${BIT}
>
> +if [ "%{mpi_device}" == "ch_psm" ] ; then
> +export CFLAGS="$CFLAGS $OPTIMIZATION_FLAG -g -D_GNU_SOURCE -DCH_PSM -D_AFFINITY_ -D_SMP_ -DXRC $ARCH_NAME -I$IB_INCLUDE"
> +else
>  export CFLAGS="$CFLAGS $OPTIMIZATION_FLAG -g -D_GNU_SOURCE -DCH_GEN2 -D_AFFINITY_ -DMEMORY_SCALE -D_SMP_ -D_SMP_RNDV_ -DVIADEV_RPUT_SUPPORT -DEARLY_SEND_COMPLETION -DXRC $ARCH_NAME -I$IB_INCLUDE"
> +fi
>
>  # Mvapich configure is not functional, so I must to add "manual check" for LINK LAYER defenition
>  if ( grep IBV_LINK_LAYER_ETHERNET %{openib_prefix}/include/infiniband/verbs.h &> /dev/null ) ; then
> @@ -257,11 +261,19 @@
>  export MPE_CFLAGS
>  export LDFLAGS
>  export CXXFLAGS="$CXXFLAGS"
> +if [ "%{mpi_device}" == "ch_psm" ] ; then
> +export FFLAGS="$FFLAGS $EXTRA_CFLAG"
> +else
>  export FFLAGS="$FFLAGS -L$IB_LIB $EXTRA_CFLAG"
> +fi
>  export F90FLAGS="$F90FLAGS $EXTRA_CFLAG"
>  export CONFIG_FLAGS
>  export MPIRUN_CFLAGS="$MPIRUN_CFLAGS -DPARAM_GLOBAL=\\\"%{_prefix}/etc/$conffile\\\" -DLD_LIBRARY_PATH_MPI=\\\"%{_prefix}/lib/shared\\\" -DMPI_PREFIX=\\\"%{_prefix}/\\\" $DEF_BUILDID"
> +if [ "%{mpi_device}" == "ch_psm" ] ; then
> +./configure --enable-sharedlib --with-device=%{mpi_device} --with-arch=LINUX --prefix=$RPM_BUILD_ROOT%{_prefix} $CONFIG_ENABLE_F77 $CONFIG_ENABLE_F90 $COMPILER_CONFIG -lib="-L$IB_LIB -lpsm_infinipath -libverbs -libumad -lpthread $EXTRA_CFLAG" $MPE_FLAGS $CONFIG_FLAGS
> +else
>  ./configure --enable-sharedlib --with-device=%{mpi_device} --with-arch=LINUX --prefix=$RPM_BUILD_ROOT%{_prefix} $CONFIG_ENABLE_F77 $CONFIG_ENABLE_F90 $COMPILER_CONFIG -lib="-L$IB_LIB -libverbs -libumad -lpthread $EXTRA_CFLAG" $MPE_FLAGS $CONFIG_FLAGS
> +fi
>  %{__make}
>
>  #############################################################################
>
> This message and any attached documents contain information from QLogic Corporation or its wholly-owned subsidiaries that may be confidential. If you are not the intended recipient, you may not read, copy, distribute, or use this information. If you have received this transmission in error, please notify the sender immediately by reply e-mail and then delete this message.
>
>
> _______________________________________________
> mvapich-discuss mailing list
> mvapich-discuss at cse.ohio-state.edu
> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>



-- 
Jonathan Perkins
http://www.cse.ohio-state.edu/~perkinjo