[mvapich-discuss] Slow MV2_USE_SHMEM_COLL

[Jonathan Perkins]

In addition to the info requested previously, can you also try using
MVAPICH2 v2.1 to see if the results are still degraded or not?

On Fri, Oct 9, 2015 at 11:49 AM Jonathan Perkins <
perkinjo at cse.ohio-state.edu> wrote:

> Thanks for your note Marcin.  We'll discuss this issue and get back to you.
>
> However, can you send more information such as the cpu model name as
> reported by `cat /proc/cpuinfo' as well as the options used to build
> MVAPICH2 as reported by mpiname -a.  Thanks in advance.
>
> On Fri, Oct 9, 2015 at 11:39 AM Marcin Rogowski <marcin.rogowski at gmail.com>
> wrote:
>
>> --===============0526641324750062117==
>> Content-Type: multipart/alternative;
>> boundary="001a11c25b487209240521adc7b1"
>>
>> --001a11c25b487209240521adc7b1
>> Content-Type: text/plain; charset="UTF-8"
>>
>> Hello,
>>
>> I have been trying to diagnose what causes a huge slow down of one part of
>> our application between MVAPICH2 1.9 and 2.0.1 and eventually came up with
>> a test case that simply calls MPI_Gather of 16 MPI_CHARACTERS to process
>> 0.
>> Timings over 51 iterations are the following:
>>
>> (setenv MV2_USE_SHMEM_COLL 1)
>> $ mpirun_rsh -np 3000 -hostfile nodes ./a.out
>> took 20.7183160781860 seconds 0.406241491729138 per gather
>>
>> (setenv MV2_USE_SHMEM_COLL 0)
>> $ mpirun_rsh -np 3000 -hostfile nodes ./a.out
>> took 2.943396568298340E-002 seconds 5.771365820192823E-004 per gather
>>
>>
>> Interestingly, if hostfile file contains unique host names (by default we
>> use node1 repeated 'cpu cores' times followed by node2 etc.) the
>> observation does not hold - both Gathers are fast.
>>
>> The problem does not seem to appear before the release 2.0.1. Easy
>> solution
>> would be to disable collective shared memory optimizations or use unique
>> host lists however both solutions slow down different parts of the
>> application, on average exactly offsetting the benefits.
>>
>> Please let me know if you would like to know any details of our cluster
>> environment (24 core Xeons with QLogic's InfiniBand). I would be really
>> grateful if you can share any ideas and/or solutions to what could be
>> causing our problems and help us achieve optimal performance.
>>
>> Thank you.
>>
>>
>> Regards,
>> Marcin Rogowski
>> Saudi Aramco
>>
>> --001a11c25b487209240521adc7b1
>> Content-Type: text/html; charset="UTF-8"
>> Content-Transfer-Encoding: quoted-printable
>>
>> <div dir=3D"ltr">Hello,<br><br>I have been trying to diagnose what causes
>> a=
>>  huge slow down of one part of our application between MVAPICH2 1.9 and
>> 2.0=
>> .1 and eventually came up with a test case that simply calls MPI_Gather
>> of =
>> 16 MPI_CHARACTERS to process 0. Timings over 51 iterations are the
>> followin=
>> g:<br><br>(setenv MV2_USE_SHMEM_COLL 1)<br>$ mpirun_rsh -np 3000
>> -hostfile =
>> nodes ./a.out<br>took   20.7183160781860      seconds  0.406241491729138
>>  =
>>    per gather<br><br>(setenv MV2_USE_SHMEM_COLL 0)<br>$ mpirun_rsh -np
>> 3000=
>>  -hostfile nodes ./a.out<br>took  2.943396568298340E-002 seconds
>> 5.7713658=
>> 20192823E-004 per gather<br><br><br>Interestingly, if hostfile file
>> contain=
>> s unique host names (by default we use node1 repeated 'cpu cores'
>> t=
>> imes followed by node2 etc.) the observation does not hold - both Gathers
>> a=
>> re fast.<br><br>The problem does not seem to appear before the release
>> 2.0.=
>> 1. Easy solution would be to disable collective shared memory
>> optimizations=
>>  or use unique host lists however both solutions slow down different
>> parts =
>> of the application, on average exactly offsetting the
>> benefits.<br><br>Plea=
>> se let me know if you would like to know any details of our cluster
>> environ=
>> ment (24 core Xeons with QLogic's InfiniBand). I would be really
>> gratef=
>> ul if you can share any ideas and/or solutions to what could be causing
>> our=
>>  problems and help us achieve optimal performance.<br><br>Thank
>> you.<br><br=
>> ><br>Regards,<br>Marcin Rogowski<br>Saudi Aramco<br></div>
>>
>> --001a11c25b487209240521adc7b1--
>>
>> --===============0526641324750062117==
>> Content-Type: text/plain; charset="us-ascii"
>> MIME-Version: 1.0
>> Content-Transfer-Encoding: 7bit
>> Content-Disposition: inline
>>
>> _______________________________________________
>> mvapich-discuss mailing list
>> mvapich-discuss at cse.ohio-state.edu
>> http://mailman.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>>
>> --===============0526641324750062117==--
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20151009/60898a88/attachment.html>