[mvapich-discuss] how to build mvapich2 so to make it work on SR-IOV + KVM Enviroment

[Xiaoyi Lu]

Hi, Pharthiphan,

In the user guide I sent to you, can you search the following paragraph?

Step 3. Execute ivshmem-tools.sh to set up IVSHMEM devices for virtual machines.

We have a tool to help you to hotplug the IVSHMEM device.

Xiaoyi 

Sent from my iPhone

> On Mar 12, 2018, at 9:28 AM, Pharthiphan Asokan <pasokan at ddn.com> wrote:
> 
> Hi Xiaoyi,
> 
> How to configure IVSHMEM on existing KVM guests ? if some link can shared for some tips would be useful
> 
> Regards,
> Pharthiphan
> ________________________________________
> From: Pharthiphan Asokan
> Sent: Monday, March 12, 2018 9:48 PM
> To: Xiaoyi Lu
> Cc: mvapich-discuss at cse.ohio-state.edu
> Subject: RE: [mvapich-discuss] how to build mvapich2 so to make it work on SR-IOV + KVM Enviroment
> 
> Hi Xiaoyi
> 
> can you please help me with the complete commandline to configure IVSHMEM,I could get through this step.
> 
> To configure IVSHMEM for virtual machines, administrators can add a device model option to QEMU command line as follows.
> 
> -device ivshmem,shm=ivshmem-id,size=256m
> 
> [root at vcn03 pasokan]# /opt/mvapich2/virt/2.2/bin/mpirun_rsh -np 4 vcn03 vcn03 vcn04 vcn04  MV2_VIRT_USE_IVSHMEM=1 /mnt/lustre_client/pasokan/a.out
> Failed to find IVShmem device, will fallback to SR-IOV
> Failed to find IVShmem device, will fallback to SR-IOV
> Failed to find IVShmem device, will fallback to SR-IOV
> Failed to find IVShmem device, will fallback to SR-IOV
> [src/mpid/ch3/channels/mrail/src/gen2/rdma_iba_priv.c:1487] Could not modify qpto RTR
> [src/mpid/ch3/channels/mrail/src/gen2/rdma_iba_priv.c:1487] Could not modify qpto RTR
> [src/mpid/ch3/channels/mrail/src/gen2/rdma_iba_priv.c:1487] Could not modify qpto RTR
> [src/mpid/ch3/channels/mrail/src/gen2/rdma_iba_priv.c:1487] Could not modify qpto RTR
> Hello world from processor vcn03, rank 1 out of 4 processors
> Hello world from processor vcn03, rank 0 out of 4 processors
> failed while avail wqe is 63, rail 0
> [vcn03:mpi_rank_1][post_srq_send] src/mpid/ch3/channels/mrail/src/gen2/ibv_send.c:878: ibv_post_sr (post_send_desc): Invalid argument (22)
> failed while avail wqe is 63, rail 0
> [vcn03:mpi_rank_0][post_srq_send] src/mpid/ch3/channels/mrail/src/gen2/ibv_send.c:878: ibv_post_sr (post_send_desc): Invalid argument (22)
> [vcn03:mpispawn_0][readline] Unexpected End-Of-File on file descriptor 7. MPI process died?
> [vcn03:mpispawn_0][mtpmi_processops] Error while reading PMI socket. MPI process died?
> [vcn03:mpispawn_0][child_handler] MPI process (rank: 1, pid: 28583) exited with status 255
> [vcn03:mpispawn_0][child_handler] MPI process (rank: 0, pid: 28582) exited with status 255
> Hello world from processor vcn04, rank 3 out of 4 processors
> Hello world from processor vcn04, rank 2 out of 4 processors
> failed while avail wqe is 63, rail 0
> [vcn04:mpi_rank_3][post_srq_send] src/mpid/ch3/channels/mrail/src/gen2/ibv_send.c:878: ibv_post_sr (post_send_desc): Invalid argument (22)
> failed while avail wqe is 63, rail 0
> [vcn04:mpi_rank_2][post_srq_send] src/mpid/ch3/channels/mrail/src/gen2/ibv_send.c:878: ibv_post_sr (post_send_desc): Invalid argument (22)
> [vcn04:mpispawn_1][readline] Unexpected End-Of-File on file descriptor 6. MPI process died?
> [vcn04:mpispawn_1][mtpmi_processops] Error while reading PMI socket. MPI process died?
> [vcn04:mpispawn_1][child_handler] MPI process (rank: 3, pid: 27658) exited with status 255
> [vcn04:mpispawn_1][child_handler] MPI process (rank: 2, pid: 27657) exited with status 255
> [root at vcn03 pasokan]# [vcn04:mpispawn_1][report_error] connect() failed: Connection refused (111)
> 
> Thanks
> Pharthiphan
> ________________________________________
> From: Xiaoyi Lu [lu.932 at osu.edu]
> Sent: Monday, March 12, 2018 9:27 PM
> To: Pharthiphan Asokan
> Cc: mvapich-discuss at cse.ohio-state.edu
> Subject: Re: [mvapich-discuss] how to build mvapich2 so to make it work on SR-IOV + KVM Enviroment
> 
> Hi, Pharthiphan,
> 
> Thanks for your interest in our project.
> 
> For virtualized environments (like SR-IOV + KVM), we strongly recommend our users to use the MVAPICH2-Virt package. You can download it from http://mvapich.cse.ohio-state.edu/downloads/.
> 
> You can find the user guide of MVAPICH2-Virt from http://mvapich.cse.ohio-state.edu/userguide/virt/.
> 
> Thanks,
> Xiaoyi
> 
>> On Mar 12, 2018, at 1:27 AM, Pharthiphan Asokan <pasokan at ddn.com> wrote:
>> 
>> Hi All,
>> 
>> how to build and run mvapich2 so to make it work on SR-IOV + KVM Enviroment
>> 
>> I tried the following, then ended up with Could not modify boot qp to RTR
>> 
>> [root at vcn01 ~]# /opt/ddn/mvapich/bin/mpirun -hosts vcn01,vcn02 -ppn 1 /opt/ddn/ior/bin/IOR-mvapich -a POSIX -w -r -t
>> 1m -b 1m -o /tmp/pasokan_fuse/test1
>> [src/mpid/ch3/channels/mrail/src/gen2/ring_startup.c:292] error(22): Could not modify boot qp to RTR
>> [src/mpid/ch3/channels/mrail/src/gen2/ring_startup.c:292] error(22): Could not modify boot qp to RTR
>> ===================================================================================
>> =
>> BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
>> =
>> PID 24752 RUNNING AT vcn02
>> =
>> EXIT CODE: 1
>> =
>> CLEANING UP REMAINING PROCESSES
>> =
>> YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
>> ===================================================================================
>> [proxy:0:0 at vcn01] HYDU_sock_write (utils/sock/sock.c:286): write error (Broken pipe)
>> [proxy:0:0 at vcn01] main (pm/pmiserv/pmip.c:265): unable to send EXIT_STATUS command upstream
>> [mpiexec at vcn01] HYDT_bscu_wait_for_completion (tools/bootstrap/utils/bscu_wait.c:76): one of the processes terminated
>> badly; aborting
>> [mpiexec at vcn01] HYDT_bsci_wait_for_completion (tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting
>> for completion
>> [mpiexec at vcn01] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:218): launcher returned error waiting for
>> completion
>> [mpiexec at vcn01] main (ui/mpich/mpiexec.c:344): process manager error waiting for completion
>> [root at vcn01 ~]#
>> 
>> Regrads,
>> Pharthiphan
>> _______________________________________________
>> mvapich-discuss mailing list
>> mvapich-discuss at cse.ohio-state.edu
>> http://mailman.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20180312/b72edbcb/attachment.html>