[mvapich-discuss] RE: Po

[Choudhury, Durga]

Hi Wei

Thanks for the reply. Our nodes are NOT connected by infiniband but by a
proprietary backplane switch fabric, and this is why we cannot port the
OSU gen2 or VAPI channel directly. We are in the process of implementing
uDAPL for our switch fabric after which we can run MPI over uDAPL, but
the uDAPL specs are so infiniband centric that we are having a hard time
doing it.

As of now, we have a TCP/IP stack running over our switch fabric and
this is how we currently run MPI over the ch3:sock device, with limited
performance. Changing the device to ch3:ssm ought to give us a
significant performance boost.

I'll try asking the question in the ANL mailing list; in the meanwhile,
any leads from the users of this list will be highly appreciated.

Thanks
Durga

-----Original Message-----
From: wei huang [mailto:huanwei at cse.ohio-state.edu] 
Sent: Tuesday, February 21, 2006 10:44 AM
To: Choudhury, Durga; mvapich-discuss at cse.ohio-state.edu
Subject: Po

>Here is a question that I *think* is non-trivial, and again, any help
is
>deeply appreciated. Has anyone tried using ch3 device ssm channel on
>anything other than an Intel platform? (This is the channel that uses
>sockets for remote communication and shared memory for local
>communication). Our architecture is a cluster of SMP MIPS64 cores, so
>this device/channel would be ideal. However, in the inline function
>MPIDU_Process_lock() defined in
>src/mpid/common/locks/mpidu_process_locks.h, the three #ifdefs
>(HAVE_INTERLOCKEDEXCHANGE, HAVE__INTERLOCKEDEXCHANGE and
>HAVE_COMPARE_AND_SWAP) all seem to be defined only for intel processors
>(actually the first two seem to be defined only for the IA64
>architecture.)

Hi Durga,

Not sure if you really mean to use socket for inter-node communication.

As you know, MVAPICH2 uses InfiniBand for inter-node communication, this
will be a much better way if your cluster is equipped with InfiniBand.
MVAPICH2 also has shared memory for intra-node operations. So it would
be
ideal for you to choose: osu_ch3:vapi or osu_ch3:gen2 if you have
InfiniBand. (SMP should be used by default)

ssm channel is with the original MPICH2 package from Argonne national
lab,
on which our MVAPICH2 is based. If you really mean to use that device
instead of devices under osu_ch3, you can post questions to
mpich2-maint at mcs.anl.gov or mpich-discuss at mcs.anl.gov.

Thanks.

Regards,
Wei Huang

774 Dreese Lab, 2015 Neil Ave,
Dept. of Computer Science and Engineering
Ohio State University
OH 43210
Tel: (614)292-8501