[Mvapich-discuss] Question on shared memory communication

[zhiting zhu]

Hi Hari,

Thanks for the reply. Could I know what algorithm is used when the CMA
support is disabled? Is it by copying to shared memory and copying out by
the other process? If this process is abstracted by a queue, which queue
algorithm does it use?

Is there any way to find out what collective algorithm variant it's
running? Many MPI libraries have many variants of allreduce/reduce
algorithm. I would like to know which algorithm is used for a specific
message size. I'm comparing the collective algorithm for intra machine
communication between OpenMPI and MVAPICH2. The latency comparison is
similar but the allreduce performance has a large gap.

Best,
Zhiting

On Thu, Apr 13, 2023 at 4:09 PM Subramoni, Hari <subramoni.1 at osu.edu> wrote:

> Hi, Zhiting.
>
>
>
> Please accept our sincere apologies for the delay.
>
>
>
> The correct channel for MVAPICH2 2.3.7 is ch3:mrail. The Nemesis netmod is
> an older, depreciated version.
>
>
>
> The environment variable MV2_USE_SHMEM_COLL enables and disables the use
> of our hierarchical process topology aware collective communication schemes.
>
>
>
> If you would like to disable CMA support, please set the environment
> variable MV2_USE_SMP_CMA=0 at runtime.
>
>
>
> Several of these designs have been implemented in various MVAPICH
> libraries.
>
>
>
> Best,
>
> Hari.
>
>
>
> *From:* Mvapich-discuss <mvapich-discuss-bounces+subramon=
> cse.ohio-state.edu at lists.osu.edu> *On Behalf Of *zhiting zhu via
> Mvapich-discuss
> *Sent:* Friday, April 7, 2023 7:58 PM
> *To:* Mvapich-discuss at lists.osu.edu
> *Subject:* [Mvapich-discuss] Question on shared memory communication
>
>
>
> Hi, I'm a new user of the MPI lib. I have a few questions.   - I notice
> that in the feature page for shared memory communication, there're two
> options:  Shared-Memory-CH3 and Shared-Memory-Nemesis. The descriptions are
> exactly the same.
>
> Hi,
>
>
>
> I'm a new user of the MPI lib. I have a few questions.
>
>
>
> - I notice that in the feature page for shared memory communication,
> there're two options: Shared-Memory-CH3 and Shared-Memory-Nemesis. The
> descriptions are exactly the same. What's the difference between these two?
> I find that in the documentation MV2_USE_SHMEM_COLL applies to OFA-IB-CH3
> and OFA-iWARP-CH3. It seems I have to use shared-memory-ch3?
>
>
>
> FFLAGS="-w -fallow-argument-mismatch -O2" ../configure
> --with-device=ch3:mrail --disable-shared
> --prefix=$SCRIPT_DIR/mvapich2_install/
>
>
>
> - I found a description on what nemesis is:
> https://urldefense.com/v3/__https://www.mcs.anl.gov/uploads/cels/papers/TM-292.pdf__;!!KGKeukY!zHYBu1pDGkC_yQgXsirCohm32ObzjPcch94mBm12em_C3z5ECfm0cRibogZIssVjKD12GgAcgEffknKk6y_UPyKurfnJJMRj$ 
> <https://urldefense.com/v3/__https:/www.mcs.anl.gov/uploads/cels/papers/TM-292.pdf__;!!KGKeukY!x9GebWlzOwJDIaxL9-bETHPm371fheTuZHtVSJ15G3kugdUOeEYmtRp9BpmzRGMtI9WUiPHAwYNL56dSdR9oXMi1td2HJ6HR$>
> Is there any documentation about the shared memory algorithms used in
> ch3:mrail? I'm looking for the case that doesn't use CMA/XPMEM/LiMiC.
>
>
>
> - I also found the paper: Design and Evaluation of Shared Memory
> CommunicationBenchmarks on Emerging Architectures using MVAPICH2
>
> Is any of these designs incorporated in the MVAPICH2? I'm trying to look
> at the high level api mentioned in this paper(msg_enqueue/codemsg_dequeue)
> but I find nothing.
>
>
>
> Best,
>
> Zhiting
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osu.edu/pipermail/mvapich-discuss/attachments/20230413/2dcf39da/attachment-0007.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: latency_2.jpg
Type: image/jpeg
Size: 311842 bytes
Desc: not available
URL: <http://lists.osu.edu/pipermail/mvapich-discuss/attachments/20230413/2dcf39da/attachment-0014.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: allreduce_8.jpg
Type: image/jpeg
Size: 332225 bytes
Desc: not available
URL: <http://lists.osu.edu/pipermail/mvapich-discuss/attachments/20230413/2dcf39da/attachment-0015.jpg>