[mvapich-discuss] Xen and VMM bypass with MVAPICH2

[Constantinos Evangelinos]

We are setting up a new mini-cluster with IB ConnectX hardware (which 
supposedly have support for virtualization) and have realized that the 
software stack we need to use is far from clear. For ease of use with 
Eucalyptus we prefer to use Ubuntu 9.04 (9.10 will not work with our Tesla 
cards nicely) - which works sort-of-ok-ish with OFED 1.3.x - but it appears 
that the published work with released code (Xen-IB) is meant for older OFED 
and Xen 3.0. The Novell VM-IB work seems to come only with some specific 
version of SLES. The Mellanox drivers are for older IB hardware (and OFED). 
Several publications seem to suggest that they modified the Xen-IB codebase 
from mercurial but without too many details and without releasing their source 
code. 

Any ideas on how we can proceed without having to reinvent the wheel?
Will MVAPICH2 work out of the box on the guest OS with the modified IB kernel 
modules or does it require its own set of patches? And finally, as we plan to 
also try VMware ESX on the cluster (for which Mellanox has supported drivers), 
has anyone any experience with getting MVAPICH2 to work with it in an HPC 
mode?

Constantinos
-- 
Dr. Constantinos Evangelinos                    Room 54-1518, EAPS/MIT
Earth, Atmospheric and Planetary Sciences       77 Massachusetts Avenue
Massachusetts Institute of Technology           Cambridge, MA 02139
+1-617-324-3386/+1-617-253-4464 (fax)           USA