[mvapich-discuss] MV2_USE_LAZY_MEM_UNREGISTER and memory usage?

[Xie Min]

We do some tests these days.

Because we share our testing system with others, and a complete HPL
will run more than two hours,
we don't run the whole HPCC to completion yet.

Without any environment variable setting, we run HPL about 10 times,
each HPL test will
run for about 12 minutes. Several tests still became deadlock, the
"port_rcv_packets" counter of IB card
stop to increase.

One of our parallel applications met some problems, it often segment
fault. But when we set the
MV2_ON_DEMAND_THRESHOLD to the value equals to the tasks number, it
can run successfully.

So we set only one envrionment variable MV2_ON_DEMAND_THRESHOLD=1024,
and run HPL tests again
for about 10 times (128 * 8, 1024 tasks, "RES" of each task is about
1.4GB), each HPL test will run for about 12 minutes.
All tests runs normally, the "port_rcv_packets" counter of IB card
increase continually in the test running time.

If on demand connection is used, seems there are some pthread lock
operations in vbuf.c, I am not sure if this will have
some relationship with the deadlock.

2009/3/24 Dhabaleswar Panda <panda at cse.ohio-state.edu>:
>> Thanks for the new patch.
>>
>> We did several tests today, but the testing system is busy today, so
>> we don't have enough node resource and time to do test.
>> We didn't run the HPCC to completion yet.
>>
>> For 512 tasks, seems this patch works, but we just run it for 30
>> minutes, in which HPL was running normally.
>
> Good to know about this.
>
>> For 1024 tasks, in one test, the HPL is deadlock when running for
>> about 5 minute (while using the old patch, HPL will be deadlock at the
>> start), but our testing system is not stable today, maybe the deadlock
>> is caused by network failure. We need to find enough
>> resource and time to do more tests.
>
> Once you have a stable system, let us know how it works for 1024 tasks.
>
> Thanks,
>
> DK
>
>> 2009/3/19 Matthew Koop <koop at cse.ohio-state.edu>:
>> > Xie,
>> > Can you try the following attached patch instead of the other patch? We
>> > found some places where a deadlock may have been allowed to occur.
>> >
>> > Thank you,
>> > Matt
>> >
>> > On Sat, 14 Mar 2009, Xie Min wrote:
>> >
>> >> Today, we do some other tests.
>> >>
>> >> For 128 tasks HPCC (16 nodes * 8), we can run the whole test
>> >> successfully and get final result.
>> >>
>> >> But for 512 tasks HPCC (64 nodes * 8), HPL is freezed too when
>> >> MV2_USE_LAZY_MEM_UNREGISTER=1.
>> >>
>> >> I attach an input file for 512 tasks HPCC (about 1.6GB for each task),
>> >> maybe you can try it on your systems to see if it will produce the
>> >> same problem.
>> >>
>> >> Thanks.
>> >>
>> >> 2009/3/12 Matthew Koop <koop at cse.ohio-state.edu>:
>> >> > Xie,
>> >> >
>> >> > Thanks for sending this information along. We've spent some time
>> >> > investigating the issue and came up with a patch that will hopefully
>> >> > resolve your issue. I've attached it to this email and it should be
>> >> > applied at the base directory.
>> >> >
>> >> > Please let us know if this helps the problem,
>> >> >
>> >> > Matt
>> >> >
>> >> > On Mon, 9 Feb 2009, Xie Min wrote:
>> >> >
>> >> >> The hpcc we used is HPCC 1.0.0, but we just tried HPCC 1.3.1, seems
>> >> >> has the same problem.
>> >> >>
>> >> >> In the attachment we attached two hpccinf.txt files for 64 HPCC tasks,
>> >> >> the hpccinf.txt.13 is the "RES" of about 1.3GB, while hpccinf.txt.16
>> >> >> is the "RES" of about 1.6/1.7GB. Whould you please try them on your
>> >> >> systems (with MV2_USE_LAZY_MEM_UNREGISTER=1), thanks.
>> >> >>
>> >> >> BTW, the OFED version we used is 1.3.1, physical memory on each node
>> >> >> is 16GB, use 8 nodes for 64 tasks.
>> >> >>
>> >> >>
>> >> >>
>> >> >> 2009/2/7 Matthew Koop <koop at cse.ohio-state.edu>:
>> >> >> >
>> >> >> > Thanks for the additional information. I've tried here with HPCC 1.3.1 and
>> >> >> > I haven't been able to see any difference in the 'RES' or 'VIRT' memory
>> >> >> > while running.
>> >> >> >
>> >> >> > Would it be possible to send me your hpccinf.txt file so I can more
>> >> >> > closely try to reproduce the problem? We also have AS5 with kernel 2.6.18
>> >> >> > as well.
>> >> >> >
>> >> >> > Thanks,
>> >> >> >
>> >> >> > Matt
>> >> >> >
>> >> >> > On Thu, 5 Feb 2009, Xie Min wrote:
>> >> >> >
>> >> >> >> We use Redhat AS5, kernel is 2.6.18 with lustre 1.6.6, and we don't
>> >> >> >> modify kernel source.
>> >> >> >>
>> >> >> >> We test HPCC on two clusters:
>> >> >> >> In one cluster, each node is booted using Boot over IB, it has no
>> >> >> >> harddisk, so NO swap space. We run 64 HPCC tasks on 8 nodes (so each
>> >> >> >> CPU core in the node will run one HPCC task), when each HPCC task use
>> >> >> >> 1.2/1.3G memory, it will be killed by OS because of "Out of memory"
>> >> >> >> error. But when MV2_USE_LAZY_MEM_UNREGISTER=0, task can use 1.7G
>> >> >> >> memory and run successfully.
>> >> >> >>
>> >> >> >> In another cluster, each node has harddisk, it booted from local disk,
>> >> >> >> and it HAS space space. We run 64 HPCC tasks on 8 nodes too. When each
>> >> >> >> HPCC use 1.3G memory, we use "top" to show the memory usage
>> >> >> >> information, we found swap will be used when HPCC is running for a
>> >> >> >> while, and the node begin to run very slowly and cannot respond to
>> >> >> >> keyboard input. But when MV2_USE_LAZY_MEM_UNREGISTER=0, each task can
>> >> >> >> be set to 1.7G memory scale and run successfully.
>> >> >> >>
>> >> >> >> I tried another mvapich2 parameters: MV2_USE_LAZY_MEM_UNREGISTER=1,
>> >> >> >> and MV2_NDREG_ENTRIES=8. In this configuration, HPCC is still be
>> >> >> >> killed by OS with "Out of memory" error when the memory scale of each
>> >> >> >> task is set to 1.3GB.
>> >> >> >>
>> >> >> >> 2009/2/5 Matthew Koop <koop at cse.ohio-state.edu>:
>> >> >> >> > Hi,
>> >> >> >> >
>> >> >> >> > What OS/distro are you running? Are there any changes you made, such as
>> >> >> >> > page size, etc from the base?
>> >> >> >> >
>> >> >> >> > I'm taking a look at this issue on our machine as well, although I'm not
>> >> >> >> > seeing the memory change that you reported.
>> >> >> >> >
>> >> >> >> > Matt
>> >> >> >> >
>> >> >> >> >
>> >> >> >>
>> >> >> >
>> >> >> >
>> >> >>
>> >> >
>> >>
>> >
>> _______________________________________________
>> mvapich-discuss mailing list
>> mvapich-discuss at cse.ohio-state.edu
>> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>>
>
>