From panda at cse.ohio-state.edu  Thu Feb  9 08:08:29 2023
From: panda at cse.ohio-state.edu (Panda, Dhabaleswar)
Date: Thu, 9 Feb 2023 13:08:29 +0000
Subject: [Mvapich] Announcing the release of MPI4Dask 0.3
In-Reply-To: <CO1PR01MB7242DC900D601204DD2A8DD2DAD99@CO1PR01MB7242.prod.exchangelabs.com>
References: <CO1PR01MB7242DC900D601204DD2A8DD2DAD99@CO1PR01MB7242.prod.exchangelabs.com>
Message-ID: <CO1PR01MB7242B2B8275047E3ECED2F51DAD99@CO1PR01MB7242.prod.exchangelabs.com>

The OSU High-Performance Big Data (HiBD) team is pleased to announce
the release of MPI4Dask 0.3, which is a custom version of the Dask
Distributed library with support for high-performance MPI
communication backend. The MPI device in MPI4Dask uses mpi4py over the
MVAPICH2, MVAPICH2-X, or MVAPICH2-GDR library and targets modern HPC
clusters built with CPUs, GPUs, and high-performance interconnects.

This release of the MPI4Dask package is equipped with the following
features:

* MPI4Dask 0.3 (Features since 0.2 release are indicated as NEW):

    - (NEW) Based on Dask Distributed 2022.8.1
    - Compliant with user-level Dask APIs and packages
    - Support for MPI-based communication GPU-based Dask applications
    - Support for MPI-based communication CPU-based Dask applications
    - Implements point-to-point communication co-routines
    - Efficient chunking mechanism implemented for large messages
    - Built on top of mpi4py over the MVAPICh2, MVAPICH2-X, and
      MVAPICH2-GDR library
    - Supports starting execution of Dask programs using Dask-MPI
    - Tested with
      - Mellanox InfiniBand adapters (FDR, EDR, and HDR)
      - (NEW) Various multi-core platforms
      - (NEW) Various benchmarks used by the community (MatMul, Slicing,
              Sum Transpose, cuDF Merge, etc.)
      - (NEW) Scaling out to hundreds of workers (processes)
      - (NEW) NVIDIA V100 and A100 GPUs


For downloading MPI4Dask 0.3 package, the associated user
guide, please visit the following URL:

http://hibd.cse.ohio-state.edu

Sample performance numbers for MPI4Dask using benchmarks can
be viewed by visiting the `Performance' tab of the above website.

All questions, feedback and bug reports are welcome. Please post to
mvapich-discuss at lists.osu.edu.

Thanks,

The High-Performance Big Data (HiBD) Team
http://hibd.cse.ohio-state.edu

PS: The number of organizations using the HiBD stacks has crossed 355
(from 38 countries). Similarly, the number of downloads from the HiBD
site has crossed 46,900.  The HiBD team would like to thank all its
users and organizations!!