[mvapich-discuss] Problems with migration in mvapich2-1.8rc1

Tue Apr 17 00:07:39 EDT 2012

Ivan,

I see from your example that you're trying to migrate all processes out of
compute-0-10.
This is the node which hosts the 'root mpispawn' process. Typically, there
is one mpispawn
process per node. The current design does not allow the migration of MPI
processes that are residing
in the same node that hosts the root mpispawn. It allows the migration of
processes
from ALL other nodes.

For instance, consider a 4 process MPI job that is run with the following
hostfile:

compute-0-8
compute-0-8
compute-0-9
compute-0-9

Here, you have 2 nodes which hosts two processes each. 'compute-0-8' is the
node which
will be hosting the root mpispawn process, so you will not be able to
migrate ranks 0 and 1 away from it,
but you can migrate ranks 2 and 3 out of compute-0-9 with the command
'mv2_trigger compute-0-9'

Also, disabling prelinking is a *requirement* to get process migration
working. The FAQ entry in BLCR's documentation
explains why this is a requirement:
https://upc-bugs.lbl.gov/blcr/doc/html/FAQ.html#prelink

A couple of other minor things to note:

- You can remove the "--enable-checkpointing --with-hydra-ckpointlib=blcr "
configure options.
MVAPICH2 inherits these flags from MPICH2, and using them will enable the
checkpointing
support in MPICH2 and is not needed for the checkpointing / migration
support in MVAPICH2's ch3 channel.

- The 'sparehosts' option takes a list of spare nodes that will be used for
consecutive migrations.
Specifying the same node name twice, as in your case, would mean that the
first and second migration
will consider compute-0-9 to be the spare node. You might not want that.

- I noticed that you've disabled aggregation (and RDMA-based migration in
turn). You can enable that
if you have FUSE installed in your system, to eliminate the file-copy
overheads and to take benefits
of MVAPICH2's pipelined process-migration protocol.

Do let us know if you have additional questions.

--
Raghu <http://www.cse.ohio-state.edu/%7Erajachan/>

2012/4/16 Iván Cores González <ivan.coresg at udc.es>

> Hi.
> Sorry for the delay in replying to your answer. I've change the
> configuration options,
> now, the configuration of mvapich is:
>
> *./configure --prefix=$HOME/mvapich/mvapich2-1.8rc1/build
> --with-device=ch3:mrail *
> --with-rdma=gen2 --enable-shared --enable-ckpt
> --with-blcr=$HOME/blcr-0.8.4/build
> --enable-ckpt-migration --enable-checkpointing
> --with-hydra-ckpointlib=blcr
> --with-ftb=$HOME/ftb/ftb-0.6.2/build --disable-ckpt-aggregation
>
> I test the BLCR installation and works fine with the cr_run and cr_restart
> commands.
> But, when I run the application with:
>
> mpirun_rsh -np 2 -hostfile mpd.hosts -sparehosts mpd.hostsMIG
> MV2_CKPT_FILE=$HOME/ckpt/check ./a.out
>
>
> where mpd.hosts is:
> compute-0-10
> compute-0-10
>
> and mpd.hostsMIG is:
> compute-0-9
> compute-0-9
>
> and I execute:
> mv2_trigger compute-0-10
>
> the checkpoint files are created, but something is wrong:
> check.0.0
> 100% 9591KB   9.4MB/s   00:00
> check.0.1
> 100% 9507KB   9.3MB/s   00:01
> Connection to compute-0-10 closed.
> [compute-0-9.local:mpispawn_1][child_handler] MPI process (rank: 0, pid:
> 29671) terminated with signal 11 -> abort job
> [pluton.des.udc.es:mpirun_rsh][process_mpispawn_connection] mpispawn_1
> from node compute-0-9 aborted: MPI process error (1)
>
> I think it could be a problem with the restart, but I can not find the
> mistake.
> I can not execute the "prelink --undo --all" command because I don't have
> root privileges.
> Could this be the cause of the problem? Am I doing something wrong?
>
> Thanks,
> Iván Cores.
>
>
> ------------------------------
> *De: *"Raghunath" <rajachan at cse.ohio-state.edu>
> *Para: *"Iván Cores González" <ivan.coresg at udc.es>
> *CC: *mvapich-discuss at cse.ohio-state.edu
> *Enviados: *Lunes, 2 de Abril 2012 16:47:35
> *Asunto: *Re: [mvapich-discuss] Problems with migration in mvapich2-1.8rc1
>
>
> Hi,
>
> Thanks for posting this to the list.
> It looks like you have configured MVAPICH2 with the QLogic PSM-CH3
> interface.
> Currently, the process Migration support in MVAPICH2 is available only
> with the
> OFA-IB-CH3 interface (--with-device=ch3:mrail --with-rdma=gen2)
> for Mellanox IB adapters, which is also the default one.
>
> Do let us know if you have additional questions on this.
>
> Thanks,
> --
> Raghu
>
>
> 2012/4/2 Iván Cores González <ivan.coresg at udc.es>
>
>> Hello,
>> I am testing the new mvapich2-1.8rc1 version in a small cluster with
>> infiniband and I have problems trying the migration features.
>>
>> I install FTB, BLCR and MVAPICH without problems. The configuration of
>> mvapich is:
>>
>> *./configure --prefix=$HOME/mvapich/mvapich2-1.8rc1/build
>> --with-device=ch3:psm --enable-shared --enable-ckpt
>> --with-blcr=$HOME/blcr-0.8.4/build --enable-ckpt-migration
>> --enable-checkpointing --with-hydra-ckpointlib=blcr
>> --with-ftb=$HOME/ftb/ftb-0.6.2/build --disable-ckpt-aggregation*
>>
>> Once I have executed the ftb daemos (ftb_database_server in the front-end
>> and the ftb_agent in the computing nodes) and loaded the BLCR modules I
>> only run the application with:
>>
>> mpirun_rsh -np 2 -hostfile mpd.hosts -sparehosts mpd.hostsMIG ./a.out
>>
>> where mpd.hosts is:
>> compute-0-10
>> compute-0-10
>>
>> and mpd.hostsMIG is:
>> compute-0-9
>> compute-0-9
>>
>> However, when I execute
>> mv2_trigger compute-0-10
>>
>> or the other options to migrate (pkill -SIGUSR2 mpispawn) nothing
>> happens. Only the FTB information is showed but the job continue working in
>> the same node.
>> I can not execute the "prelink --undo --all" command because I have not
>> root privileges. Could this be the cause of the problem?
>> Am I doing something wrong?
>>
>> Thanks,
>> Iván Cores.
>>
>>
>>
>>
>>
>> _______________________________________________
>> mvapich-discuss mailing list
>> mvapich-discuss at cse.ohio-state.edu
>> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20120417/336d13ec/attachment-0001.html