[mvapich-discuss] OpenIB Presentation on MVAPICH/MVAPICH2 has

[Dhabaleswar Panda]

> Hi All
> 
> Does anyone know any work on implementing (hardware) fault tolerance on
> MPI? (i.e. node/connection failures in the middle of a calculation
> should be handled in software). I downloaded some academic software that
> claims to achieve this but I could not get it to run correctly even on a
> Linux PC cluster (and our target hardware is a lot more sophisticated)

Thanks for your interest in the fault-tolerance stuff in MPI.

> Has any work on this regard been done/being planned in Prof. Panda's
> lab? Any pointers will be greatly appreciated.

Yes. Please take a look at the slides which I presented at the OpenIB
workshop last week. The slides are available from MVAPICH web page as
well as from OpenIB web page.

We are working on solutions along three directions (slide #45) to
provide varying degrees of fault-tolerance:

1. Mem-to-Mem Reliability (Slides #46-47): This support will provide
software-level detection of I/O-bus (such as PCI-x/PCI-Ex) errors at
the MPI level and will provide reliable data transfer from memory of
the source node to the memory of the destination node. This feature
will be available in MVAPICH 0.9.7 in the next few weeks.

2. Network-level Fault Tolerance with Automatic Path Migration (APM)
(Slides #49-51): InfiniBand provides APM supports which migrates an
existing path of communication to another path in case a link or
switch fails in an existing path. This can happen in the middle of any
computation and the underlying MPI library will take care of it. We
have incorporated this support to MVAPICH and testing it. It will be
released in the upcoming releases of MVAPICH and MVAPICH2.

3. Process-Level Fault Tolerance with Checkpoint Restart (Slides
#52-57): This provides the highest level of fault tolerance including
node failure. With this support enabled, the MPI application will go
through periodic checkpoints. If any failure occurs (node or
connection), the application can be restarted from the last
check-point.  Once again, we have this solution working in the lab
with MVAPICH2. We are carrying out in-depth testing. We plan to make a
release of this feature in the upcoming MVAPICH2 release.

Hope this helps. 

Thanks, 

DK


> Thanks
> Durga
> 
> -----Original Message-----
> From: mvapich-discuss-bounces at cse.ohio-state.edu
> [mailto:mvapich-discuss-bounces at cse.ohio-state.edu] On Behalf Of
> Dhabaleswar Panda
> Sent: Friday, February 10, 2006 10:47 PM
> To: mvapich-discuss at cse.ohio-state.edu
> Cc: Dhabaleswar Panda
> Subject: [mvapich-discuss] OpenIB Presentation on MVAPICH/MVAPICH2 has
> beenlinked to the MVAPICH page
> 
> The presentation made at the OpenIB workshop in Sonoma is now linked
> to the MVAPICH page. I have also sent the slides to Matt for it to be
> linked to the main OpenIB page. 
> 
> This presentation describes the current status of MVAPICH and MVAPICH2
> projects, latest performance numbers (such as SDR-DDR comparisons) and
> upcoming features. Especially, the initial set of performance numbers
> with the following upcoming features are included:
> 
> - SRQ with Flow control for scalability to multi-thousand nodes
>    - basic performance benefits
>    - reduced memory requirements as systems scale and performance
>      benefits to applications
> 
> - Fault Tolerance features
>    - Memory-to-memory reliability
>    - Network-level Fault Tolerance with Automatic Path Migration (APM)
>    - Process-level Fault Tolerance with Checkpoint-Restart
> 
> - Multi-threading Support
> 
> - Multi-network Support through uDAPL
> 
> - Adaptive Connection Management
>    - On-demand based schemes for scalability to multi-thousand nodes
> 
> We are working towards rolling out these features in successive
> MVAPICH and MVAPICH2 releases during the next 2-3 months.
> 
> Please feel free to take a look at the slides and let us know if you
> have any comments or suggestions on these upcoming features.
> 
> Thanks, 
> 
> DK
> 
> _______________________________________________
> mvapich-discuss mailing list
> mvapich-discuss at cse.ohio-state.edu
> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
> 
> _______________________________________________
> mvapich-discuss mailing list
> mvapich-discuss at cse.ohio-state.edu
> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>