[Mvapich-discuss] Compiling / Instalation without osu-benchmark - ch3:nemesis:ib, tcp

Wed Jan 26 03:42:40 EST 2022

Hi,
I am aware of the fact, that there is a possibility to use the aws-version, however we do just provide the container and the customers choose the cloud. To minimize the effort we want to use the normal version. In this case the use of tcp for aws would be enough (at the moment). However the question (compare to 4.13 Configuring a build for OFA-IB-Nemesis and TCP/IP Nemesis (unified binary))
Why

./configure  --with-device=ch3:nemesis:ib,tcp
does not compile (note nemesis:ib is depricated):

error: 'MPIDI_CH3I_CH_comm_t' {aka 'struct <anonymous>'} has no member named 'rank_list'
.....

remains. 
Am I missing any flags, prerequests?

Cheers
Joachim 

-----Ursprüngliche Nachricht-----
Von: Mvapich-discuss <mvapich-discuss-bounces at lists.osu.edu> Im Auftrag von mvapich-discuss-request at lists.osu.edu
Gesendet: Freitag, 21. Januar 2022 15:45
An: mvapich-discuss at lists.osu.edu
Betreff: Mvapich-discuss Digest, Vol 14, Issue 8

Send Mvapich-discuss mailing list submissions to
	mvapich-discuss at lists.osu.edu

To subscribe or unsubscribe via the World Wide Web, visit
	https://urldefense.com/v3/__https://ofcsg2dvf1.dwd.de/fmlurlsvc/?fewReq=:B:JVMxOj48MS19NjklOy1ibzY7OjE7Oi14Ymxlan9*eW42OztuP245Mz85ajhoM2hqb2k5Mjw8M288PD44Om86aDk4b28zaDxuaC1*Njo9Pzk8PD04OzstemJvNjk7R05hO0o9Ozs5PDI8Jjk7R05hO0o8Ozs5PDI8LXloe382QWRqaGNiZiVfeGhjbn54aGNlbnlLb3xvJW9uLWg2ODstY29nNjs=&url=https*3a*2f*2flists.osu.edu*2fmailman*2flistinfo*2fmvapich-discuss__;Ky8lJSUlJSU!!KGKeukY!m0J72VA1dX6qtl644J8PztmG1c3Tw55XacP4zrawbPpxohkN10OJ1P_6Tn5bUAbQ8xznDRXV8g$  
or, via email, send a message with subject or body 'help' to
	mvapich-discuss-request at lists.osu.edu

You can reach the person managing the list at
	mvapich-discuss-owner at lists.osu.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Mvapich-discuss digest..."

Today's Topics:

   1. Re: Compiling / Instalation without osu-benchmark
      (Tscheuschner Joachim)
   2. Re: Compiling / Instalation without osu-benchmark
      (Panda, Dhabaleswar)

----------------------------------------------------------------------

Message: 1
Date: Fri, 21 Jan 2022 14:27:51 +0000
From: Tscheuschner Joachim <Joachim.Tscheuschner at dwd.de>
To: "'Shineman, Nat'" <shineman.5 at osu.edu>, Tscheuschner Joachim
	<Joachim.Tscheuschner at dwd.de>
Cc: "Panda, Dhabaleswar" <panda at cse.ohio-state.edu>,
	"mvapich-discuss at lists.osu.edu" <mvapich-discuss at lists.osu.edu>
Subject: Re: [Mvapich-discuss] Compiling / Instalation without
	osu-benchmark
Message-ID: <7bc0af678f3d41dc84b7d20d206320c9 at exch.dwd.de>
Content-Type: text/plain; charset="utf-8"

Hello Nat,

we are building a portable container for different cloud-provider. While everything works in our system, testing with AWS-cloud shows a problem.  
If we compile mvapich2-2.3.6 with 

./configure  --enable-g=none 

the communication does not work (aws)
[rdma_open_hca] Unknown HCA type: this build of MVAPICH2 does notfully support the HCA found on the system (try with other build options)
[cli_25]: aborting job:
Fatal error in PMPI_Init_thread:
Other MPI error, error stack:
MPIR_Init_thread(493)............: 
MPID_Init(419)...................: channel initialization failed
MPIDI_CH3_Init(471)..............: rdma_get_control_parameters
rdma_get_control_parameters(1926): rdma_open_hca
rdma_open_hca(1046)..............: No IB device found
While with 

./configure --enable-g=all 
and
singularity exec ../base.sif mpirun -n 2 -v -genv FI_IFACE=tcp ~/mvapich/libexec/osu-micro-benchmarks/mpi/pt2pt/osu_bw

CPU Affinity is undefined 
[cli_1]: aborting job:
Fatal error in MPI_Init:
Other MPI error, error stack:
MPIR_Init_thread(493)........: 
MPID_Init(400)...............: 
MPIDI_CH3I_set_affinity(3474): 
smpi_setaffinity(2719).......: CPU Affinity is undefined.

And

./configure  --with-device=ch3:nemesis:tcp

MPI works again on AWS, BUT as a portable container I will lose ib for other cloud-systems. And, as mentioned in the userguide,

./configure  --with-device=ch3:nemesis:ib,tcp

does not compile:

error: 'MPIDI_CH3I_CH_comm_t' {aka 'struct <anonymous>'} has no member named 'rank_list'

The document contains the script to install mvapich2 in a docker-container with Debian:bullseye 10 or 11.
Am I missing any flags or prerequests?

Cheers
Joachim

-----Urspr?ngliche Nachricht-----
Von: Shineman, Nat <shineman.5 at osu.edu> 
Gesendet: Mittwoch, 1. Dezember 2021 19:28
An: Tscheuschner Joachim <Joachim.Tscheuschner at dwd.de>
Cc: mvapich-discuss at lists.osu.edu; Panda, Dhabaleswar <panda at cse.ohio-state.edu>; Subramoni, Hari <subramoni.1 at osu.edu>
Betreff: Re: Compiling / Instalation without osu-benchmark

Joachim, 

Thanks for reaching out to us. First off, we would recommend mvapich2-2.3.6 for your use case. mv2-virt is very old and has not been updated for some time. Using a debian based container should be fine. Everything will compile normally from within the container, just ensure that you have GNU Autotools installed.  

Unfortunately, at this time we do not have a configure time flag for disabling the OMB suite. We will look into adding support for this in our upcoming 2.3.7 release. For the time being we can suggest a workaround. When you run make install? though you should see all of the OMB binaries in a directory called libexec? in your mvapich2 installation directory. If you are installing it in the default location (/usr) you will see a directory /usr/libexec/osu-micro-benchmarks?, in my experience it is typically around 3MB. You can delete this directory to remove the installed binaries from your system. This will have no impact on the rest of the library's functionality.

Regarding the other disable options: running ./configure --help? will list all of the configuration options available to you. You are correct that --disable-fortran? is one way to reduce the installation size. Likewise you can use --disable-cxx? to disable C++ bindings if you only wish to have C libraries installed. However, there is no option for "disable all" as many of the enable/disable flags are used to determine feature sets within the library and not actual binaries. Please ensure that you set --enable-g=none? (which should be default) to remove all debugging symbols and reduce size. Other than that, just avoiding enabling additional features should yield the smallest libraries possible. 

Please let us know if you have any trouble with the installation or if the compiled libraries are still too large and we can see what can be done. 

Thanks, 

Nat

________________________________

From: Mvapich-discuss <mvapich-discuss-bounces+shineman.5=osu.edu at lists.osu.edu> on behalf of Tscheuschner Joachim via Mvapich-discuss <mvapich-discuss at lists.osu.edu>
Sent: Tuesday, November 30, 2021 08:34
To: 'mvapich-discuss at lists.osu.edu' <mvapich-discuss at lists.osu.edu>
Subject: [Mvapich-discuss] Compiling / Instalation without osu-benchmark 

Hi MVAPICH-Team,

we want to use containers for a cloud-project. To minimize the container-size &ability, we would like to disable for some, the benchmark-test and compiling ability. Could you give us a recommendation to do so?

We would like to use a debian-version(instead of centos) in the container.

As far as I understood with <--disable-x> the compiling ability can be reduced, e.g. <--disable-fortran> for fortran. Is there an option to disable all? What is the option to disable the osu-benchmarks?
Would you recommend the package <MV2-virt-2.2> or <MV2-2.3.6> for any cloud? 

Sincerely
Joachim Tscheuschner
_______________________________________________
Mvapich-discuss mailing list
Mvapich-discuss at lists.osu.edu
https://urldefense.com/v3/__https://ofcsg2dvf1.dwd.de/fmlurlsvc/?fewReq=:B:JVMxOj48MS19NjklOy1ibzY7OjE7Oi14Ymxlan9*eW42OztuP245Mz85ajhoM2hqb2k5Mjw8M288PD44Om86aDk4b28zaDxuaC1*Njo9Pzk8PD04OzstemJvNjk7R05hO0o9Ozs5PDI8Jjk7R05hO0o8Ozs5PDI8LXloe382QWRqaGNiZiVfeGhjbn54aGNlbnlLb3xvJW9uLWg2ODstY29nNjs=&url=https*3a*2f*2flists.osu.edu*2fmailman*2flistinfo*2fmvapich-discuss__;Ky8lJSUlJSU!!KGKeukY!m0J72VA1dX6qtl644J8PztmG1c3Tw55XacP4zrawbPpxohkN10OJ1P_6Tn5bUAbQ8xznDRXV8g$   <https://urldefense.com/v3/__https://ofcsg2dvf1.dwd.de/fmlurlsvc/?fewReq=:B:JVQwOz89MCx8NzgkOixjbjc6OzA6Oyx5Y21ka35*eG83Mz06bDxvOD07azs8OT0*Pjo*PGhoOT4*bjhpP2xpbz07PW9uMzo*Oix*Nzs8OTI5Mjw*OTgse2NuNztIO0BHSWw*Ojs*Ojw5JztIO0BHSWw*Ojs*Ojw5LHhpen43QGVraWJjZyReeWlib395aWJkb3hKbn1uJG5vLGk3OTosYm5mNzo=&url=https*3a*2f*2flists.osu.edu*2fmailman*2flistinfo*2fmvapich-discuss__;LysrLysrLysrLyslJSUlJSU!!KGKeukY!hsDToDgLWMXRc4QwF-MVI2nZLbAn04KaCLigj1wCNxrCILQY6qOs627VlGUQd9K5sSJzggrvVA$ > 

----------
Hinweis (DWD): Mindestens eine URL in dieser E-Mail wurde vom Schadsoftware-Erkennungssystem als potentiell gef?hrlich eingestuft und so umgeschrieben, dass ein Klick darauf diese zun?chst auf gef?hrliche Downloads hin untersucht, bevor Inhalte angezeigt werden. Die Analyse erfolgt ohne menschliches Zutun voll automatisiert. Der Datenschutz ist gew?hrleistet. Es kann einige Sekunden dauern, bis die Inhalte angezeigt werden.
Der Originalabsender dieser E-Mail ist: shineman.5 at osu.edu. Wenn Ihnen dieser unbekannt ist, so sollten Sie die enthaltenen Links nicht anklicken.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: install.mvapich.sh
Type: application/octet-stream
Size: 2324 bytes
Desc: install.mvapich.sh
URL: <https://urldefense.com/v3/__https://ofcsg2dvf1.dwd.de/fmlurlsvc/?fewReq=:B:JVMxOj48MS19NjklOy1ibzY7OjE7Oi14Ymxlan9*eW42Oz09b2o5bjs*bTk*M204am8*bjM*bj9oPm4*bz45aWk6Ozk5Oj5taC1*Njo9Pzk8PD04OzstemJvNjk7R05hO0o9Ozs5PDI8Jjk7R05hO0o8Ozs5PDI8LXloe382QWRqaGNiZiVfeGhjbn54aGNlbnlLb3xvJW9uLWg2ODstY29nNjs=&url=https*3a*2f*2flists.osu.edu*2fmailman*2fprivate*2fmvapich-discuss*2fattachments*2f20220121*2f9964a732*2fattachment-0001.obj__;KysrLy8vLyUlJSUlJSUlJSU!!KGKeukY!m0J72VA1dX6qtl644J8PztmG1c3Tw55XacP4zrawbPpxohkN10OJ1P_6Tn5bUAbQ8xyZPQ5rmA$  >

------------------------------

Message: 2
Date: Fri, 21 Jan 2022 14:44:53 +0000
From: "Panda, Dhabaleswar" <panda at cse.ohio-state.edu>
To: Tscheuschner Joachim <Joachim.Tscheuschner at dwd.de>, "Shineman,
	Nat"	<shineman.5 at osu.edu>
Cc: "mvapich-discuss at lists.osu.edu" <mvapich-discuss at lists.osu.edu>
Subject: Re: [Mvapich-discuss] Compiling / Instalation without
	osu-benchmark
Message-ID:
	<SA0PR01MB6153146A4AB8FD01F1990805DA5B9 at SA0PR01MB6153.prod.exchangelabs.com>

Content-Type: text/plain; charset="utf-8"

Hi Joachim, 

Since AWS HPC instances use EFA adapters (not IB adapters), we have a separate version of MVAPICH2 named `MVAPICH2-X-AWS'. This version is optimized for the AWS EFA adapters. Please use this version (you can download it from the MVAPICH2 download site) and follow the steps mentioned in the associated user guide. 

Let us know if you encounter any issues with this version. 

Thanks, 

DK

-----Original Message-----
From: Tscheuschner Joachim <Joachim.Tscheuschner at dwd.de> 
Sent: Friday, January 21, 2022 9:28 AM
To: Shineman, Nat <shineman.5 at osu.edu>; Tscheuschner Joachim <Joachim.Tscheuschner at dwd.de>
Cc: mvapich-discuss at lists.osu.edu; Panda, Dhabaleswar <panda at cse.ohio-state.edu>; Subramoni, Hari <subramoni.1 at osu.edu>
Subject: AW: Compiling / Instalation without osu-benchmark

Hello Nat,

we are building a portable container for different cloud-provider. While everything works in our system, testing with AWS-cloud shows a problem.  
If we compile mvapich2-2.3.6 with 

./configure  --enable-g=none 

the communication does not work (aws)
[rdma_open_hca] Unknown HCA type: this build of MVAPICH2 does notfully support the HCA found on the system (try with other build options)
[cli_25]: aborting job:
Fatal error in PMPI_Init_thread:
Other MPI error, error stack:
MPIR_Init_thread(493)............: 
MPID_Init(419)...................: channel initialization failed
MPIDI_CH3_Init(471)..............: rdma_get_control_parameters
rdma_get_control_parameters(1926): rdma_open_hca
rdma_open_hca(1046)..............: No IB device found While with 

./configure --enable-g=all
and
singularity exec ../base.sif mpirun -n 2 -v -genv FI_IFACE=tcp ~/mvapich/libexec/osu-micro-benchmarks/mpi/pt2pt/osu_bw

CPU Affinity is undefined
[cli_1]: aborting job:
Fatal error in MPI_Init:
Other MPI error, error stack:
MPIR_Init_thread(493)........: 
MPID_Init(400)...............: 
MPIDI_CH3I_set_affinity(3474): 
smpi_setaffinity(2719).......: CPU Affinity is undefined.

And

./configure  --with-device=ch3:nemesis:tcp

MPI works again on AWS, BUT as a portable container I will lose ib for other cloud-systems. And, as mentioned in the userguide,

./configure  --with-device=ch3:nemesis:ib,tcp

does not compile:

error: 'MPIDI_CH3I_CH_comm_t' {aka 'struct <anonymous>'} has no member named 'rank_list'

The document contains the script to install mvapich2 in a docker-container with Debian:bullseye 10 or 11.
Am I missing any flags or prerequests?

Cheers
Joachim

-----Urspr?ngliche Nachricht-----
Von: Shineman, Nat <shineman.5 at osu.edu>
Gesendet: Mittwoch, 1. Dezember 2021 19:28
An: Tscheuschner Joachim <Joachim.Tscheuschner at dwd.de>
Cc: mvapich-discuss at lists.osu.edu; Panda, Dhabaleswar <panda at cse.ohio-state.edu>; Subramoni, Hari <subramoni.1 at osu.edu>
Betreff: Re: Compiling / Instalation without osu-benchmark

Joachim, 

Thanks for reaching out to us. First off, we would recommend mvapich2-2.3.6 for your use case. mv2-virt is very old and has not been updated for some time. Using a debian based container should be fine. Everything will compile normally from within the container, just ensure that you have GNU Autotools installed.  

Unfortunately, at this time we do not have a configure time flag for disabling the OMB suite. We will look into adding support for this in our upcoming 2.3.7 release. For the time being we can suggest a workaround. When you run make install? though you should see all of the OMB binaries in a directory called libexec? in your mvapich2 installation directory. If you are installing it in the default location (/usr) you will see a directory /usr/libexec/osu-micro-benchmarks?, in my experience it is typically around 3MB. You can delete this directory to remove the installed binaries from your system. This will have no impact on the rest of the library's functionality.

Regarding the other disable options: running ./configure --help? will list all of the configuration options available to you. You are correct that --disable-fortran? is one way to reduce the installation size. Likewise you can use --disable-cxx? to disable C++ bindings if you only wish to have C libraries installed. However, there is no option for "disable all" as many of the enable/disable flags are used to determine feature sets within the library and not actual binaries. Please ensure that you set --enable-g=none? (which should be default) to remove all debugging symbols and reduce size. Other than that, just avoiding enabling additional features should yield the smallest libraries possible. 

Please let us know if you have any trouble with the installation or if the compiled libraries are still too large and we can see what can be done. 

Thanks, 

Nat

________________________________

From: Mvapich-discuss <mvapich-discuss-bounces+shineman.5=osu.edu at lists.osu.edu> on behalf of Tscheuschner Joachim via Mvapich-discuss <mvapich-discuss at lists.osu.edu>
Sent: Tuesday, November 30, 2021 08:34
To: 'mvapich-discuss at lists.osu.edu' <mvapich-discuss at lists.osu.edu>
Subject: [Mvapich-discuss] Compiling / Instalation without osu-benchmark 

Hi MVAPICH-Team,

we want to use containers for a cloud-project. To minimize the container-size &ability, we would like to disable for some, the benchmark-test and compiling ability. Could you give us a recommendation to do so?

We would like to use a debian-version(instead of centos) in the container.

As far as I understood with <--disable-x> the compiling ability can be reduced, e.g. <--disable-fortran> for fortran. Is there an option to disable all? What is the option to disable the osu-benchmarks?
Would you recommend the package <MV2-virt-2.2> or <MV2-2.3.6> for any cloud? 

Sincerely
Joachim Tscheuschner
_______________________________________________
Mvapich-discuss mailing list
Mvapich-discuss at lists.osu.edu
https://urldefense.com/v3/__https://ofcsg2dvf1.dwd.de/fmlurlsvc/?fewReq=:B:JVMxOj48MS19NjklOy1ibzY7OjE7Oi14Ymxlan9*eW42OztuP245Mz85ajhoM2hqb2k5Mjw8M288PD44Om86aDk4b28zaDxuaC1*Njo9Pzk8PD04OzstemJvNjk7R05hO0o9Ozs5PDI8Jjk7R05hO0o8Ozs5PDI8LXloe382QWRqaGNiZiVfeGhjbn54aGNlbnlLb3xvJW9uLWg2ODstY29nNjs=&url=https*3a*2f*2flists.osu.edu*2fmailman*2flistinfo*2fmvapich-discuss__;Ky8lJSUlJSU!!KGKeukY!m0J72VA1dX6qtl644J8PztmG1c3Tw55XacP4zrawbPpxohkN10OJ1P_6Tn5bUAbQ8xznDRXV8g$   <https://urldefense.com/v3/__https://ofcsg2dvf1.dwd.de/fmlurlsvc/?fewReq=:B:JVQwOz89MCx8NzgkOixjbjc6OzA6Oyx5Y21ka35*eG83Mz06bDxvOD07azs8OT0*Pjo*PGhoOT4*bjhpP2xpbz07PW9uMzo*Oix*Nzs8OTI5Mjw*OTgse2NuNztIO0BHSWw*Ojs*Ojw5JztIO0BHSWw*Ojs*Ojw5LHhpen43QGVraWJjZyReeWlib395aWJkb3hKbn1uJG5vLGk3OTosYm5mNzo=&url=https*3a*2f*2flists.osu.edu*2fmailman*2flistinfo*2fmvapich-discuss__;LysrLysrLysrLyslJSUlJSU!!KGKeukY!mvoyh0vhSNgwdontK5koCGcPuOg5i-SDd194Ax7gIRqd1dKvxNa0pGqe-I4prLFsL4IPkw$ > 

----------
Hinweis (DWD): Mindestens eine URL in dieser E-Mail wurde vom Schadsoftware-Erkennungssystem als potentiell gef?hrlich eingestuft und so umgeschrieben, dass ein Klick darauf diese zun?chst auf gef?hrliche Downloads hin untersucht, bevor Inhalte angezeigt werden. Die Analyse erfolgt ohne menschliches Zutun voll automatisiert. Der Datenschutz ist gew?hrleistet. Es kann einige Sekunden dauern, bis die Inhalte angezeigt werden.
Der Originalabsender dieser E-Mail ist: shineman.5 at osu.edu. Wenn Ihnen dieser unbekannt ist, so sollten Sie die enthaltenen Links nicht anklicken.

------------------------------

Subject: Digest Footer

_______________________________________________
Mvapich-discuss mailing list
Mvapich-discuss at lists.osu.edu
https://urldefense.com/v3/__https://ofcsg2dvf1.dwd.de/fmlurlsvc/?fewReq=:B:JVMxOj48MS19NjklOy1ibzY7OjE7Oi14Ymxlan9*eW42OztuP245Mz85ajhoM2hqb2k5Mjw8M288PD44Om86aDk4b28zaDxuaC1*Njo9Pzk8PD04OzstemJvNjk7R05hO0o9Ozs5PDI8Jjk7R05hO0o8Ozs5PDI8LXloe382QWRqaGNiZiVfeGhjbn54aGNlbnlLb3xvJW9uLWg2ODstY29nNjs=&url=https*3a*2f*2flists.osu.edu*2fmailman*2flistinfo*2fmvapich-discuss__;Ky8lJSUlJSU!!KGKeukY!m0J72VA1dX6qtl644J8PztmG1c3Tw55XacP4zrawbPpxohkN10OJ1P_6Tn5bUAbQ8xznDRXV8g$  

------------------------------

End of Mvapich-discuss Digest, Vol 14, Issue 8
**********************************************

----------
Hinweis (DWD): Mindestens eine URL in dieser E-Mail wurde vom Schadsoftware-Erkennungssystem als potentiell gefährlich eingestuft und so umgeschrieben, dass ein Klick darauf diese zunächst auf gefährliche Downloads hin untersucht, bevor Inhalte angezeigt werden. Die Analyse erfolgt ohne menschliches Zutun voll automatisiert. Der Datenschutz ist gewährleistet. Es kann einige Sekunden dauern, bis die Inhalte angezeigt werden.
Der Originalabsender dieser E-Mail ist: mvapich-discuss-bounces at lists.osu.edu. Wenn Ihnen dieser unbekannt ist, so sollten Sie die enthaltenen Links nicht anklicken.