[mvapich-discuss] MVAPICH2, Open MPI, PGI, f2py and behavior of -V/--version

[Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC]]

MVAPICH List,

This is a bit of an odd issue, but I'm hoping someone here can help. In 
the code I work with, we use f2py. Due to the fact that some 
sub-libraries (HDF5 and netCDF) were compiled with parallel support, 
that means the compiler they were technically built with was "mpif90" 
(and/or "mpif77"). So, when we run f2py, we have to run:

    $ f2py --f77exec=mpif77 --f90exec=mpif90 -c ...

so that it can pick up the MPI libraries when it compiles.

The compilers I use are, mainly, Intel and PGI, and since I maintain 
some GPU code, PGI is often my focus. That said, I'm having an issue 
trying to use PGI and MVAPICH2 in combination that prevents me from 
exploring the GPU enhancements of MVAPICH2.

I think the issue boils down to behavior upon returning the version for 
the compiler through the MPI wrapper. When I traced through f2py's 
automatic compiler detection, it seems to depend on a successful return 
code from a version check to say "Yes, the --f90exec is installed and 
valid." (I'm pretty sure of this, but it took a while to trace through.)

To wit, when I just use PGI 14.1:

> $ pgfortran -V
>
> pgfortran 14.1-0 64-bit target on x86-64 Linux -tp nehalem
> The Portland Group - PGI Compilers and Tools
> Copyright (c) 2014, NVIDIA CORPORATION.  All rights reserved.
> $ echo $?
> $ pgfortran --version
>
> pgfortran 14.1-0 64-bit target on x86-64 Linux -tp nehalem
> The Portland Group - PGI Compilers and Tools
> Copyright (c) 2014, NVIDIA CORPORATION.  All rights reserved.
> $ echo $?
> 0

So, if you use -V or --version, you get the usual spiel and a return 
code of 0. Now, when I use PGI 14.1 and Open MPI 1.7.3 and look for the 
version under mpif90, I get this behavior:

> $ mpif90 -V
>
> pgf90 14.1-0 64-bit target on x86-64 Linux -tp nehalem
> The Portland Group - PGI Compilers and Tools
> Copyright (c) 2014, NVIDIA CORPORATION.  All rights reserved.
> (1427) $ echo $?
> 0
> $ mpif90 --version
>
> pgf90 14.1-0 64-bit target on x86-64 Linux -tp nehalem
> The Portland Group - PGI Compilers and Tools
> Copyright (c) 2014, NVIDIA CORPORATION.  All rights reserved.
> $ echo $?
> 0

Again, I get the same behavior as the bare compiler with return codes of 0.

Now, PGI 14.1 and MVAPICH2 2.0b:

> $ mpif90 -V
>
> pgf90 14.1-0 64-bit target on x86-64 Linux -tp sandybridge
> The Portland Group - PGI Compilers and Tools
> Copyright (c) 2014, NVIDIA CORPORATION.  All rights reserved.
> /usr/local/sles11/pgi/linux86-64/14.1/libso/f90main.o: In function `main':
> f90main.c:(.text+0x40): undefined reference to `MAIN_'
> $ echo $?
> 2
> $ mpif90 --version
>
> pgf90 14.1-0 64-bit target on x86-64 Linux -tp sandybridge
> The Portland Group - PGI Compilers and Tools
> Copyright (c) 2014, NVIDIA CORPORATION.  All rights reserved.
> /usr/local/sles11/pgi/linux86-64/14.1/libso/f90main.o: In function `main':
> f90main.c:(.text+0x40): undefined reference to `MAIN_'
> $ echo $?
> 2

And herein lies the reason, I think, I can't use MVAPICH2 and PGI with 
f2py, when you run mpif90 --version/-V with MVAPICH2, it returns an 
error code of 2, not 0.

Is there a reason for this? I'm especially surprised at the --version 
behavior. Are there compilers/situations that when you do "$FC 
--version" one should expect a "main" program to link?

The one simple workaround I can think of is to change the mpif90 script 
itself:

> --- mpif90_withchange 	2014-03-13 09:25:24.496774411 -0400
> +++ /usr/local/other/SLES11.1/mvapich2/2.0b/pgi-14.1.0/bin/mpif90	2014-01-31 13:32:39.033436000 -0500
> @@ -151,7 +151,7 @@
>      case "$arg" in
>   	# ----------------------------------------------------------------
>  	# Compiler options that affect whether we are linking or no
> -    -c|-S|-E|-M|-MM|-V|--version)
> +    -c|-S|-E|-M|-MM)
>      # The compiler links by default
>      linking=no
>      ;;

but as that script (and mpif77) is fairly fundamental to MVAPICH2, well, 
I'm a bit hesitant to have my sysadmins patch it every time they install 
for fear of breaking...everything! (Of course, I might have them try it 
to see if it helps.)

Would you have an objection (if I find it works) to adding these as 
"linking=no" flags for mpif90/f77/cc/cxx? I think this does preserve the 
PGI ability to change compilers on the fly with -V<ver> since, in bash, 
"-V14.3" does not equal "-V".

Thanks,
Matt

-- 
Matt Thompson, PhD     SSAI, Sr Software Test Engr
NASA GSFC, Global Modeling and Assimilation Office
Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771
Phone: 301-614-6712              Fax: 301-614-6246