[mvapich-discuss] Fw: Re: Fw: Question for help, thanks a lot!
zhangwl
zhangwl at ncic.ac.cn
Mon Jun 4 21:09:00 EDT 2007
A question on MPI over infiniband bandwidth test. Thanks for your help!
zhangwl
2007-06-05
·¢¼þÈË£º Jiuxing Liu
·¢ËÍʱ¼ä£º 2007-06-04 22:05:10
ÊÕ¼þÈË£º zhangwl at ncic.ac.cn
³ËÍ£º
Ö÷Ì⣺ Re: Fw: Question for help, thanks a lot!
Hi,
I have left Ohio State and do not maintain MVAPICH any more.
Can you forward your message to the MVAPICH mailing list (mvapich-discuss at cse.ohio-state.edu )? I think that people there will be more than happy to help you.
Thanks,
-Jiuxing
"zhangwl" <zhangwl at ncic.ac.cn>
06/04/2007 07:58 AM
ToJiuxing Liu/Watson/IBM at IBMUS
cc
SubjectFw: Question for help, thanks a lot!
Hello, Dr. Jiuxing Liu,
When testing mpi over infiniband performance, I find a problem that bandwidth at larger messages are lower instead both intra and inter node. The detailed tests and results are attached as follows.
My system is:
-- 2.2GHz Dual Core AMD Opteron(tm) Processor 275, 8GB Mem
-- Linux 2.6.9-42.ELsmp x86_64
-- openib-1.1 { Detected the following HCAs: 1) mthca0 [ Mellanox PCI-X ] }
1. Test inter-node bandwidth with -DVIADEV_RGET_SUPPORT .
setup_ch_gen2 starts... -D_X86_64_ -DEARLY_SEND_COMPLETION -DMEMORY_SCALE -DVIADEV_RGET_SUPPORT -DLAZY_MEM_UNREGISTER -DCH_GEN2 -D_SMP_ -D_SMP_RNDV_ -D_MLX_PCI_X_ -I/usr/local/ofed/include -O3
$ mpirun_rsh -rsh -np 2 inode28 inode30 ./osu_bw
# OSU MPI Bandwidth Test (Version 2.2)
# Size Bandwidth (MB/s)
1 0.243180
2 0.507795
4 1.008787
8 2.030054
16 4.008455
32 8.113140
64 16.160978
128 33.764735
256 67.708075
512 161.522157
1024 335.222506
2048 491.421716
4096 568.259955
8192 606.043232
16384 662.063392
32768 738.589843
65536 783.586601
131072 807.462616
262144 820.750931
524288 685.880335
1048576 660.237959
2097152 659.233480
4194304 659.946110
2. Test inter-node bandwidth with -DVIADEV_RPUT_SUPPORT .
setup_ch_gen2 starts... -D_X86_64_ -DEARLY_SEND_COMPLETION -DMEMORY_SCALE -DVIADEV_RPUT_SUPPORT -DLAZY_MEM_UNREGISTER -DCH_GEN2 -D_SMP_ -D_SMP_RNDV_ -D_MLX_PCI_X_ -I/usr/local/ofed/include -O3
$ mpirun_rsh -rsh -np 2 inode28 inode30 ./osu_bw
# OSU MPI Bandwidth Test (Version 2.2)
# Size Bandwidth (MB/s)
1 0.248081
2 0.516046
4 1.034260
8 2.069607
16 4.110799
32 8.282444
64 16.593745
128 34.620911
256 69.113305
512 163.455879
1024 341.066875
2048 496.503655
4096 569.049428
8192 606.183374
16384 624.840449
32768 713.280615
65536 769.011487
131072 800.359506
262144 814.869019
524288 679.025085
1048576 652.137840
2097152 650.207077
4194304 650.629356
3. Test intra-node bandwidth with -DVIADEV_RPUT_SUPPORT .
$ mpirun_rsh -rsh -np 2 inode28 inode28 ./osu_bw
# OSU MPI Bandwidth Test (Version 2.2)
# Size Bandwidth (MB/s)
1 2.173175
2 4.449079
4 9.049134
8 20.301348
16 42.489627
32 85.085168
64 153.869271
128 286.734337
256 480.187573
512 741.525232
1024 932.896797
2048 1145.834426
4096 1291.731546
8192 1388.989562
16384 1428.285773
32768 1453.529249
65536 1431.307671
131072 1445.227803
262144 1393.404399
524288 1168.315567
1048576 1071.952093
2097152 1072.327638
4194304 1064.196619
I have seen test results on your homepage (http://mvapich.cse.ohio-state.edu/performance/mvapich/opteron/MVAPICH-opteron-gen2-DDR.shtml, http://mvapich.cse.ohio-state.edu/performance/mvapich/intra_opteron.shtml), that inter-node bandwidth results seem normal but intra-node bandwidth results are like mine. And bandwidth results in you paper BUILDING MULTIRAIL INFINIBAND CLUSTERS: MPI-LEVEL DESIGN AND PERFORMANCE EVALUATION: SC2004(Fig. 9) seem that striping or binding optimization will remove improve this problem.
What do you think will be the problem source for my bandwidth tests? In order to get optimal bandwidth value, what do you think I should modify based on default options in original MVAPICH 0.9.8 packet? Does STRIPING or BINDING haven't been added to mvapich0.9.8?
Thanks a lot for your any reply!
Best Regards,
Wenli
zhangwl
2007-06-04
=====================================================
Zhang Wenli,
NCIC, Institute of Computing Technology
Chinese Academy of Sciences
NO. 6, Ke Xue Yuan South Road, Zhongguancun,
Beijing, P.R.China
NCIC, P.O.Box 2704 Zip Code 100080
Tel: 86-10-62601041
Fax: 86-10-62527487
Email: zhangwl at ncic.ac.cn
=====================================================
Zhang Wenli,
NCIC, Institute of Computing Technology
Chinese Academy of Sciences
NO. 6, Ke Xue Yuan South Road, Zhongguancun,
Beijing, P.R.China
NCIC, P.O.Box 2704 Zip Code 100080
Tel: 86-10-62601041
Fax: 86-10-62527487
Email: zhangwl at ncic.ac.cn
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20070605/73b7b997/attachment-0001.html
More information about the mvapich-discuss
mailing list