[mvapich-discuss] Fwd: MVAPICH2 + BLCR performance problem on multi-core cluster

[Dhabaleswar Panda]

> Hello,
>
> I have a performance problem when using mvapich2 compiled with BLCR support
> on infiniband cluster with following parameters:
>
> Node: 2xQuad Core Intel Xeon 2.33 GHz
> O/S: RHEL4.5
> File System: GPFS
> We are using MVAPICH2-1.0.2p1 with BLCR-0.6.5.
>
> I've done 3 test runs of my program using 8 MPI processes:
> 1) All of 8 processes on one node
> 2) by 4 processes on two nodes
> 3) by 2 processes on 4 nodes
>
> *Results MVAPICH2 configured for BLCR support:*
> *[ccs-dev at n5304]$ mpiexec -machinefile ./mf1 -np 8 ./test*
> *Calc time: 341.3279, send/recv time = 297.817*
> *[ccs-dev at n5304]$ mpiexec -machinefile ./mf2 -np 8 ./test*
> *Calc time: 85.7075, send/recv time = 42.2270*
> *[ccs-dev at n5304]$ mpiexec -machinefile ./mf3 -np 8 ./test*
> *Calc time: 84.6182, send/recv time = 40.3554*
>
> *Results MVAPICH2 configured without BLCR support:*
> *[ccs-dev at n5304]$ mpiexec -machinefile ./mf1 -np 8 ./test*
> *Calc time: 51.5888, send/recv time = 8.0186*
> *[ccs-dev at n5304]$ mpiexec -machinefile ./mf2 -np 8 ./test*
> *Calc time: 53.6679, send/recv time = 10.1187*
> *[ccs-dev at n5304]$ mpiexec -machinefile ./mf3 -np 8 ./test*
> *Calc time: 63.6611, send/recv time = 20.0127
> *
>
> So when using MVAPICH2 configured with BLCR support I have much
> time which is spent on communication between processes.
> Is it concerned with the fact of shared-memory support automatic disabling
> in such build?

The previous version of BLCR didn't have capability to check-point
shared-memory. That's why the shared-memory support was disabled in
MVAPICH2 1.0 to make it work with BLCR. The performance degradation is
coming because of the lack of shared memory support.

> If it is so, do you plan to include support of both BLCR & shared-memory
> communications in future releases?

Since the lastest version of BLCR supports check-pointing shared-memory,
we are working on enabling this in MVAPICH2. The next version of MVAPICH2
(1.1) will support this capability. You will see the best performance as
well as the checkpointing capability.

Thanks,

DK

> And maybe there are another ways to improve performance of MPI program
> running on multi-core node?
>
>
> Thanks.
>
> Maya
>