[mvapich-discuss] how to build mvapich2 so to make it work on SR-IOV + KVM Enviroment
Xiaoyi Lu
lu.932 at osu.edu
Mon Mar 12 13:41:02 EDT 2018
Hi, Pharthiphan,
What are your KVM and libvirt version numbers?
Xiaoyi
Sent from my iPhone
> On Mar 12, 2018, at 10:29 AM, Pharthiphan Asokan <pasokan at ddn.com> wrote:
>
> I tried to do this manully and got following error
>
> # virsh qemu-monitor-command vcn01 --pretty '{"execute":"device_add","arguments":{"driver":"ivshmem","id":"hpivshmem","shm":"g290da24","size":"256m"}}'
> {
> "id": "libvirt-44",
> "error": {
> "class": "GenericError",
> "desc": "'ivshmem' is not a valid device model name"
> }
> }
>
> From: Pharthiphan Asokan
> Sent: Monday, March 12, 2018 10:28 PM
> To: Xiaoyi Lu
> Cc: mvapich-discuss at cse.ohio-state.edu
> Subject: RE: [mvapich-discuss] how to build mvapich2 so to make it work on SR-IOV + KVM Enviroment
>
> Hi Xiaoyi,
>
> virtual machines are not deployed through openstack
>
> /opt/mvapich2/virt/2.2/share/mvapich2-virt/ivshmem-tools.sh attach vcn01 g290da24 256
> Returning instance name of server vcn01
> /opt/mvapich2/virt/2.2/share/mvapich2-virt/ivshmem-tools.sh: line 50: nova: command not found
> Returning hostname where server vcn01 is running on
> /opt/mvapich2/virt/2.2/share/mvapich2-virt/ivshmem-tools.sh: line 54: nova: command not found
> INSTANCE:
> HOST:
> Error: Returning instance and/or host failed, exit.
>
> Regards,
> Pharthiphan
> From: Xiaoyi Lu [lu.932 at osu.edu]
> Sent: Monday, March 12, 2018 10:07 PM
> To: Pharthiphan Asokan
> Cc: mvapich-discuss at cse.ohio-state.edu
> Subject: Re: [mvapich-discuss] how to build mvapich2 so to make it work on SR-IOV + KVM Enviroment
>
> Hi, Pharthiphan,
>
> In the user guide I sent to you, can you search the following paragraph?
>
> Step 3. Execute ivshmem-tools.sh to set up IVSHMEM devices for virtual machines.
>
> We have a tool to help you to hotplug the IVSHMEM device.
>
> Xiaoyi
>
> Sent from my iPhone
>
> On Mar 12, 2018, at 9:28 AM, Pharthiphan Asokan <pasokan at ddn.com> wrote:
>
>> Hi Xiaoyi,
>>
>> How to configure IVSHMEM on existing KVM guests ? if some link can shared for some tips would be useful
>>
>> Regards,
>> Pharthiphan
>> ________________________________________
>> From: Pharthiphan Asokan
>> Sent: Monday, March 12, 2018 9:48 PM
>> To: Xiaoyi Lu
>> Cc: mvapich-discuss at cse.ohio-state.edu
>> Subject: RE: [mvapich-discuss] how to build mvapich2 so to make it work on SR-IOV + KVM Enviroment
>>
>> Hi Xiaoyi
>>
>> can you please help me with the complete commandline to configure IVSHMEM,I could get through this step.
>>
>> To configure IVSHMEM for virtual machines, administrators can add a device model option to QEMU command line as follows.
>>
>> -device ivshmem,shm=ivshmem-id,size=256m
>>
>> [root at vcn03 pasokan]# /opt/mvapich2/virt/2.2/bin/mpirun_rsh -np 4 vcn03 vcn03 vcn04 vcn04 MV2_VIRT_USE_IVSHMEM=1 /mnt/lustre_client/pasokan/a.out
>> Failed to find IVShmem device, will fallback to SR-IOV
>> Failed to find IVShmem device, will fallback to SR-IOV
>> Failed to find IVShmem device, will fallback to SR-IOV
>> Failed to find IVShmem device, will fallback to SR-IOV
>> [src/mpid/ch3/channels/mrail/src/gen2/rdma_iba_priv.c:1487] Could not modify qpto RTR
>> [src/mpid/ch3/channels/mrail/src/gen2/rdma_iba_priv.c:1487] Could not modify qpto RTR
>> [src/mpid/ch3/channels/mrail/src/gen2/rdma_iba_priv.c:1487] Could not modify qpto RTR
>> [src/mpid/ch3/channels/mrail/src/gen2/rdma_iba_priv.c:1487] Could not modify qpto RTR
>> Hello world from processor vcn03, rank 1 out of 4 processors
>> Hello world from processor vcn03, rank 0 out of 4 processors
>> failed while avail wqe is 63, rail 0
>> [vcn03:mpi_rank_1][post_srq_send] src/mpid/ch3/channels/mrail/src/gen2/ibv_send.c:878: ibv_post_sr (post_send_desc): Invalid argument (22)
>> failed while avail wqe is 63, rail 0
>> [vcn03:mpi_rank_0][post_srq_send] src/mpid/ch3/channels/mrail/src/gen2/ibv_send.c:878: ibv_post_sr (post_send_desc): Invalid argument (22)
>> [vcn03:mpispawn_0][readline] Unexpected End-Of-File on file descriptor 7. MPI process died?
>> [vcn03:mpispawn_0][mtpmi_processops] Error while reading PMI socket. MPI process died?
>> [vcn03:mpispawn_0][child_handler] MPI process (rank: 1, pid: 28583) exited with status 255
>> [vcn03:mpispawn_0][child_handler] MPI process (rank: 0, pid: 28582) exited with status 255
>> Hello world from processor vcn04, rank 3 out of 4 processors
>> Hello world from processor vcn04, rank 2 out of 4 processors
>> failed while avail wqe is 63, rail 0
>> [vcn04:mpi_rank_3][post_srq_send] src/mpid/ch3/channels/mrail/src/gen2/ibv_send.c:878: ibv_post_sr (post_send_desc): Invalid argument (22)
>> failed while avail wqe is 63, rail 0
>> [vcn04:mpi_rank_2][post_srq_send] src/mpid/ch3/channels/mrail/src/gen2/ibv_send.c:878: ibv_post_sr (post_send_desc): Invalid argument (22)
>> [vcn04:mpispawn_1][readline] Unexpected End-Of-File on file descriptor 6. MPI process died?
>> [vcn04:mpispawn_1][mtpmi_processops] Error while reading PMI socket. MPI process died?
>> [vcn04:mpispawn_1][child_handler] MPI process (rank: 3, pid: 27658) exited with status 255
>> [vcn04:mpispawn_1][child_handler] MPI process (rank: 2, pid: 27657) exited with status 255
>> [root at vcn03 pasokan]# [vcn04:mpispawn_1][report_error] connect() failed: Connection refused (111)
>>
>> Thanks
>> Pharthiphan
>> ________________________________________
>> From: Xiaoyi Lu [lu.932 at osu.edu]
>> Sent: Monday, March 12, 2018 9:27 PM
>> To: Pharthiphan Asokan
>> Cc: mvapich-discuss at cse.ohio-state.edu
>> Subject: Re: [mvapich-discuss] how to build mvapich2 so to make it work on SR-IOV + KVM Enviroment
>>
>> Hi, Pharthiphan,
>>
>> Thanks for your interest in our project.
>>
>> For virtualized environments (like SR-IOV + KVM), we strongly recommend our users to use the MVAPICH2-Virt package. You can download it from http://mvapich.cse.ohio-state.edu/downloads/.
>>
>> You can find the user guide of MVAPICH2-Virt from http://mvapich.cse.ohio-state.edu/userguide/virt/.
>>
>> Thanks,
>> Xiaoyi
>>
>>>> On Mar 12, 2018, at 1:27 AM, Pharthiphan Asokan <pasokan at ddn.com> wrote:
>>>
>>> Hi All,
>>>
>>> how to build and run mvapich2 so to make it work on SR-IOV + KVM Enviroment
>>>
>>> I tried the following, then ended up with Could not modify boot qp to RTR
>>>
>>> [root at vcn01 ~]# /opt/ddn/mvapich/bin/mpirun -hosts vcn01,vcn02 -ppn 1 /opt/ddn/ior/bin/IOR-mvapich -a POSIX -w -r -t
>>> 1m -b 1m -o /tmp/pasokan_fuse/test1
>>> [src/mpid/ch3/channels/mrail/src/gen2/ring_startup.c:292] error(22): Could not modify boot qp to RTR
>>> [src/mpid/ch3/channels/mrail/src/gen2/ring_startup.c:292] error(22): Could not modify boot qp to RTR
>>> ===================================================================================
>>> =
>>> BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
>>> =
>>> PID 24752 RUNNING AT vcn02
>>> =
>>> EXIT CODE: 1
>>> =
>>> CLEANING UP REMAINING PROCESSES
>>> =
>>> YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
>>> ===================================================================================
>>> [proxy:0:0 at vcn01] HYDU_sock_write (utils/sock/sock.c:286): write error (Broken pipe)
>>> [proxy:0:0 at vcn01] main (pm/pmiserv/pmip.c:265): unable to send EXIT_STATUS command upstream
>>> [mpiexec at vcn01] HYDT_bscu_wait_for_completion (tools/bootstrap/utils/bscu_wait.c:76): one of the processes terminated
>>> badly; aborting
>>> [mpiexec at vcn01] HYDT_bsci_wait_for_completion (tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting
>>> for completion
>>> [mpiexec at vcn01] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:218): launcher returned error waiting for
>>> completion
>>> [mpiexec at vcn01] main (ui/mpich/mpiexec.c:344): process manager error waiting for completion
>>> [root at vcn01 ~]#
>>>
>>> Regrads,
>>> Pharthiphan
>>> _______________________________________________
>>> mvapich-discuss mailing list
>>> mvapich-discuss at cse.ohio-state.edu
>>> http://mailman.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20180312/a97afc31/attachment-0001.html>
More information about the mvapich-discuss
mailing list