[mvapich-discuss] problems with getting both shared memory and tcp
communication
Zulauf, Michael
Michael.Zulauf at iberdrolaren.com
Fri Mar 30 17:21:55 EDT 2012
Hello all,
At my place of work for the last several months, we've been using
mvapich2-1.7rc1 on hardware with an Infiniband interconnect, and it's
been working extremely well. Recently we got some new hardware, but
this came with 10gigE (not my choice), and I've been trying to get
things running on it. In case it matters, I believe the network
controllers are Intel 82598EB. Our nodes have dual Intel Xeon X5672 (so
8 cores per node). We're primarily using the PGI 7.2-5 compilers.
We've also got PGI 10.6 (I believe), but we don't use it as much.
>From the documentation for 1.7rc1, it's not entirely clear to me how to
configure and build things so that we can run jobs that will utilize
both shared memory communication within nodes, and tcp communication
across nodes. We thought our best bet would be some flavor of Nemesis,
but the mvapich2 documentation confused me a bit. It did refer us to
the mpich2 docs, which made it sound like we could configure for
ch3:nemesis.
ch3:nemesis This method is our new, high performance method. It
has
been made the default communication channel starting the 1.1 release
of MPICH2. It uses shared-memory to send messages between processes
on the same node and the network for processes between nodes.
The configure/build steps appeared to go well (I used ./configure
--with-device=ch3:nemesis). But when I try running the OSU benchmark
tests, they will only work if I attempt to run a single process on each
node (specified using the -hosts option). Interestingly, if I use our
original mvapich2 installation (which was built for our Infiniband
hardware), it works for shared memory tests, but not for tests across
nodes - which is not surprising.
Here's an example of a failed test (new build, within a single node):
------------------------------------------------------------------------
---------------------------------------------------------
%
/apps/new_cluster/mvapich2-1.7rc1_PGI7.2-5-nemesis/install_dir/bin/mpiex
ec -hosts compute-1-15,compute-1-15 -n 2
/apps/new_cluster/mvapich2-1.7rc1_PGI7.2-5-nemesis/osu_benchmarks/osu_bi
bw
# OSU MPI Bi-Directional Bandwidth Test v3.3
# Size Bi-Bandwidth (MB/s)
[compute-1-15.local:mpi_rank_0][error_sighandler] Caught error:
Segmentation fault (signal 11)
========================================================================
=============
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= EXIT CODE: 11
= CLEANING UP REMAINING PROCESSES
= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
========================================================================
=============
APPLICATION TERMINATED WITH THE EXIT STRING: Segmentation fault (signal
11)
------------------------------------------------------------------------
---------------------------------------------------------
But if I run it between different nodes, it works:
------------------------------------------------------------------------
---------------------------------------------------------
%
/apps/new_cluster/mvapich2-1.7rc1_PGI7.2-5-nemesis/install_dir/bin/mpiex
ec -hosts compute-1-14,compute-1-15 -n 2
/apps/new_cluster/mvapich2-1.7rc1_PGI7.2-5-nemesis/osu_benchmarks/osu_bi
bw
# OSU MPI Bi-Directional Bandwidth Test v3.3
# Size Bi-Bandwidth (MB/s)
1 0.28
2 0.57
4 1.31
8 3.37
16 6.58
32 13.14
64 24.21
128 48.28
256 84.43
512 122.64
1024 180.40
2048 331.52
4096 562.93
8192 597.75
16384 604.92
32768 608.34
65536 607.86
131072 626.89
262144 623.12
524288 611.56
1048576 644.33
2097152 661.96
4194304 654.56
------------------------------------------------------------------------
---------------------------------------------------------
And if I use our earlier install (setup for the Infiniband hardware) for
a case within a single node, that works also:
------------------------------------------------------------------------
---------------------------------------------------------
% /apps/new_cluster/mvapich2-1.7rc1_PGI7.2-5/install_dir/bin/mpiexec
-hosts compute-1-15,compute-1-15 -n 2
/apps/new_cluster/mvapich2-1.7rc1_PGI7.2-5/osu_benchmarks/osu_bibw
stty: standard input: Invalid argument
# OSU MPI Bi-Directional Bandwidth Test v3.3
# Size Bi-Bandwidth (MB/s)
1 2.83
2 5.74
4 11.54
8 22.66
16 46.08
32 90.02
64 175.56
128 346.18
256 650.94
512 1182.73
1024 1994.11
2048 3347.98
4096 4738.25
8192 5939.29
16384 6443.79
32768 6166.21
65536 6238.25
131072 6032.80
262144 9905.82
524288 9807.30
1048576 9789.05
2097152 9126.49
4194304 5609.62
------------------------------------------------------------------------
---------------------------------------------------------
Any thoughts on the best way to configure and run this on our new
hardware?
Thanks,
Mike
--
Mike Zulauf
Meteorologist, Lead Senior
Asset Optimization
Iberdrola Renewables
1125 NW Couch, Suite 700
Portland, OR 97209
Office: 503-478-6304 Cell: 503-913-0403
This message is intended for the exclusive attention of the recipient(s) indicated. Any information contained herein is strictly confidential and privileged. If you are not the intended recipient, please notify us by return e-mail and delete this message from your computer system. Any unauthorized use, reproduction, alteration, filing or sending of this message and/or any attached files may lead to legal action being taken against the party(ies) responsible for said unauthorized use. Any opinion expressed herein is solely that of the author(s) and does not necessarily represent the opinion of the Company. The sender does not guarantee the integrity, speed or safety of this message, and does not accept responsibility for any possible damage arising from the interception, incorporation of viruses, or any other damage as a result of manipulation.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20120330/2adbe0af/attachment-0001.html
More information about the mvapich-discuss
mailing list