[mvapich-discuss] time spent in mpi_init and MV2_ON_DEMAND_THRESHOLD

[Gregory Bauer]

Qi-

Thanks for the patch. I rebuilt mvapich2-0.9.8p2 with the patch applied.

Testing up to 4096 tasks looks good. I think the issue is solved.

-Greg

Qi Gao wrote:

> Hi Greg,
>
> We've been looking into this problem and found that the slow startup 
> in 'on
> demand' case is because in that case the MPI program uses too
> much of PMI interface to exchange startup information, while in the
> 'all-to-all' case an advanced protocol using InfiniBand to exchange 
> startup
> information is used.
>
> I've ported this protocol to the 'on demand' case. The patch against
> mvapich2-0.9.8p2 is in the attachment. Now it should not performance 
> worse
> than 'all-to-all' case.
>
> Let me know if you have any more problems.
>
> Regards,
> --Qi
>
> ----- Original Message ----- From: "Gregory Bauer" <gbauer at ncsa.uiuc.edu>
> To: <mvapich-discuss at cse.ohio-state.edu>
> Sent: Thursday, May 03, 2007 11:45 AM
> Subject: [mvapich-discuss] time spent in mpi_init and
> MV2_ON_DEMAND_THRESHOLD
>
>
>> We are new at running MVAPICH2 and are seeing some scaling issues with
>> application start-up on an Intel dual quad-core with Infiniband
>> interconnect. Using the typical MPI "hello world' we see that when 
>> running
>> 512 tasks (nodes=64:ppn=8)  and greater, the application makes very slow
>> progress in mpi_init(), unless MV2_ON_DEMAND_THRESHOLD is set to a count
>> larger than the task count. So it appears that static connection 
>> start-up
>> is faster than 'on demand' which is seems counter-intuitive. At task
>> counts greater than 512, the 'on demand' scheme is too slow to be
>> practical.
>>
>> We must have something incorrectly configured, but what?
>>
>> Thanks.
>> -Greg
>>
>> stack trace of the hello process
>> #0  0x0000002a9588a0df in __read_nocancel () from
>> /lib64/tls/libpthread.so.0
>> #1  0x000000000044f513 in PMIU_readline ()
>> #2  0x0000000000443e51 in PMI_KVS_Get ()
>> #3  0x00000000004158b2 in MPIDI_CH3I_SMP_init ()
>> #4  0x00000000004456f9 in MPIDI_CH3_Init ()
>> #5  0x000000000042dab6 in MPID_Init ()
>> #6  0x000000000040f469 in MPIR_Init_thread ()
>> #7  0x000000000040f0bc in PMPI_Init ()
>> #8  0x0000000000403347 in main (argc=Variable "argc" is not available.
>> ) at hello.c:17
>>
>> strace snipet of the associated python process
>> select(16, [4 5 6 7 8 10 12 13 15], [], [], {5, 0}) = 1 (in [7], left 
>> {5,
>> 0})
>> recvfrom(7, "00000094", 8, 0, NULL, NULL) = 8
>> recvfrom(7, "(dp1\nS\'cmd\'\np2\nS\'response_to_pmi"..., 94, 0, NULL,
>> NULL) = 94
>> sendto(8, "00000094(dp1\nS\'cmd\'\np2\nS\'respons"..., 102, 0, NULL, 
>> 0) =
>> 102
>> select(16, [4 5 6 7 8 10 12 13 15], [], [], {5, 0}) = 1 (in [7], left 
>> {5,
>> 0})
>> recvfrom(7, "00000094", 8, 0, NULL, NULL) = 8
>> recvfrom(7, "(dp1\nS\'cmd\'\npProcess 28455 detached
>>
>>
>> Configuration information:
>>
>> $ /usr/local/mvapich2-0.9.8p1-gcc/bin/mpich2version
>> Version:           MVAPICH2-0.9.8
>> Device:            osu_ch3:mrail
>> Configure
>> Options: --prefix=/usr/local/mvapich2-0.9.8p1-gcc 
>> --with-device=osu_ch3:mrail
>>  --with-rdma=gen2 --with-pm=mpd --enable-romio --without-mpe
>>
>> $ cat /usr/local/ofed/BUILD_ID
>> OFED-1.1
>> openib-1.1 (REV=9905)
>>
>> $ dmesg | grep ib_
>> ib_core: no version for "struct_module" found: kernel tainted.
>> ib_mthca: Mellanox InfiniBand HCA driver v0.08 (February 14, 2006)
>> ib_mthca: Initializing 0000:0a:00.0
>>
>> $ uname -a
>> Linux honest1 2.6.9-42.0.8.EL_lustre.1.4.9smp #1 SMP Fri Feb 16 01:23:52
>> MST 2007 x86_64 x86_64 x86_64 GNU/Linux
>>
>> $ cat /etc/*release
>> Red Hat Enterprise Linux AS release 4 (Nahant Update 4)
>>
>>
>> _______________________________________________
>> mvapich-discuss mailing list
>> mvapich-discuss at cse.ohio-state.edu
>> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>>