[mvapich-discuss] how to build mvapich2 so to make it work on SR-IOV + KVM Enviroment
Xiaoyi Lu
lu.932 at osu.edu
Fri Mar 16 22:14:03 EDT 2018
(Just for closing this discussion)
After the discussion with our user, the issue got fixed by using the latest version of MVAPICH2 (MVAPICH2 2.3rc1). This can be downloaded from http://mvapich.cse.ohio-state.edu/downloads/.
Thanks,
Xiaoyi
> On Mar 12, 2018, at 1:48 PM, Xiaoyi Lu <lu.932 at osu.edu> wrote:
>
> Thanks. Let’s take a look and get back to you later.
>
> Xiaoyi
>
> Sent from my iPhone
>
> On Mar 12, 2018, at 10:44 AM, Pharthiphan Asokan <pasokan at ddn.com> wrote:
>
>> Hi Xiaoyi,
>>
>> # /usr/libexec/qemu-kvm --version
>> QEMU emulator version 1.5.3 (qemu-kvm-1.5.3-141.el7_4.6), Copyright (c) 2003-2008 Fabrice Bellard
>> #
>>
>> libvirt-3.2.0-14.el7_4.9.x86_64
>>
>>
>> Regards,
>> Pharthiphan
>> From: Xiaoyi Lu [lu.932 at osu.edu]
>> Sent: Monday, March 12, 2018 11:11 PM
>> To: Pharthiphan Asokan
>> Cc: mvapich-discuss at cse.ohio-state.edu
>> Subject: Re: [mvapich-discuss] how to build mvapich2 so to make it work on SR-IOV + KVM Enviroment
>>
>> Hi, Pharthiphan,
>>
>> What are your KVM and libvirt version numbers?
>>
>> Xiaoyi
>>
>> Sent from my iPhone
>>
>> On Mar 12, 2018, at 10:29 AM, Pharthiphan Asokan <pasokan at ddn.com> wrote:
>>
>>> I tried to do this manully and got following error
>>>
>>> # virsh qemu-monitor-command vcn01 --pretty '{"execute":"device_add","arguments":{"driver":"ivshmem","id":"hpivshmem","shm":"g290da24","size":"256m"}}'
>>> {
>>> "id": "libvirt-44",
>>> "error": {
>>> "class": "GenericError",
>>> "desc": "'ivshmem' is not a valid device model name"
>>> }
>>> }
>>>
>>> From: Pharthiphan Asokan
>>> Sent: Monday, March 12, 2018 10:28 PM
>>> To: Xiaoyi Lu
>>> Cc: mvapich-discuss at cse.ohio-state.edu
>>> Subject: RE: [mvapich-discuss] how to build mvapich2 so to make it work on SR-IOV + KVM Enviroment
>>>
>>> Hi Xiaoyi,
>>>
>>> virtual machines are not deployed through openstack
>>>
>>> /opt/mvapich2/virt/2.2/share/mvapich2-virt/ivshmem-tools.sh attach vcn01 g290da24 256
>>> Returning instance name of server vcn01
>>> /opt/mvapich2/virt/2.2/share/mvapich2-virt/ivshmem-tools.sh: line 50: nova: command not found
>>> Returning hostname where server vcn01 is running on
>>> /opt/mvapich2/virt/2.2/share/mvapich2-virt/ivshmem-tools.sh: line 54: nova: command not found
>>> INSTANCE:
>>> HOST:
>>> Error: Returning instance and/or host failed, exit.
>>>
>>> Regards,
>>> Pharthiphan
>>> From: Xiaoyi Lu [lu.932 at osu.edu]
>>> Sent: Monday, March 12, 2018 10:07 PM
>>> To: Pharthiphan Asokan
>>> Cc: mvapich-discuss at cse.ohio-state.edu
>>> Subject: Re: [mvapich-discuss] how to build mvapich2 so to make it work on SR-IOV + KVM Enviroment
>>>
>>> Hi, Pharthiphan,
>>>
>>> In the user guide I sent to you, can you search the following paragraph?
>>>
>>> Step 3. Execute ivshmem-tools.sh to set up IVSHMEM devices for virtual machines.
>>>
>>> We have a tool to help you to hotplug the IVSHMEM device.
>>>
>>> Xiaoyi
>>>
>>> Sent from my iPhone
>>>
>>> On Mar 12, 2018, at 9:28 AM, Pharthiphan Asokan <pasokan at ddn.com> wrote:
>>>
>>>> Hi Xiaoyi,
>>>>
>>>> How to configure IVSHMEM on existing KVM guests ? if some link can shared for some tips would be useful
>>>>
>>>> Regards,
>>>> Pharthiphan
>>>> ________________________________________
>>>> From: Pharthiphan Asokan
>>>> Sent: Monday, March 12, 2018 9:48 PM
>>>> To: Xiaoyi Lu
>>>> Cc: mvapich-discuss at cse.ohio-state.edu
>>>> Subject: RE: [mvapich-discuss] how to build mvapich2 so to make it work on SR-IOV + KVM Enviroment
>>>>
>>>> Hi Xiaoyi
>>>>
>>>> can you please help me with the complete commandline to configure IVSHMEM,I could get through this step.
>>>>
>>>> To configure IVSHMEM for virtual machines, administrators can add a device model option to QEMU command line as follows.
>>>>
>>>> -device ivshmem,shm=ivshmem-id,size=256m
>>>>
>>>> [root at vcn03 pasokan]# /opt/mvapich2/virt/2.2/bin/mpirun_rsh -np 4 vcn03 vcn03 vcn04 vcn04 MV2_VIRT_USE_IVSHMEM=1 /mnt/lustre_client/pasokan/a.out
>>>> Failed to find IVShmem device, will fallback to SR-IOV
>>>> Failed to find IVShmem device, will fallback to SR-IOV
>>>> Failed to find IVShmem device, will fallback to SR-IOV
>>>> Failed to find IVShmem device, will fallback to SR-IOV
>>>> [src/mpid/ch3/channels/mrail/src/gen2/rdma_iba_priv.c:1487] Could not modify qpto RTR
>>>> [src/mpid/ch3/channels/mrail/src/gen2/rdma_iba_priv.c:1487] Could not modify qpto RTR
>>>> [src/mpid/ch3/channels/mrail/src/gen2/rdma_iba_priv.c:1487] Could not modify qpto RTR
>>>> [src/mpid/ch3/channels/mrail/src/gen2/rdma_iba_priv.c:1487] Could not modify qpto RTR
>>>> Hello world from processor vcn03, rank 1 out of 4 processors
>>>> Hello world from processor vcn03, rank 0 out of 4 processors
>>>> failed while avail wqe is 63, rail 0
>>>> [vcn03:mpi_rank_1][post_srq_send] src/mpid/ch3/channels/mrail/src/gen2/ibv_send.c:878: ibv_post_sr (post_send_desc): Invalid argument (22)
>>>> failed while avail wqe is 63, rail 0
>>>> [vcn03:mpi_rank_0][post_srq_send] src/mpid/ch3/channels/mrail/src/gen2/ibv_send.c:878: ibv_post_sr (post_send_desc): Invalid argument (22)
>>>> [vcn03:mpispawn_0][readline] Unexpected End-Of-File on file descriptor 7. MPI process died?
>>>> [vcn03:mpispawn_0][mtpmi_processops] Error while reading PMI socket. MPI process died?
>>>> [vcn03:mpispawn_0][child_handler] MPI process (rank: 1, pid: 28583) exited with status 255
>>>> [vcn03:mpispawn_0][child_handler] MPI process (rank: 0, pid: 28582) exited with status 255
>>>> Hello world from processor vcn04, rank 3 out of 4 processors
>>>> Hello world from processor vcn04, rank 2 out of 4 processors
>>>> failed while avail wqe is 63, rail 0
>>>> [vcn04:mpi_rank_3][post_srq_send] src/mpid/ch3/channels/mrail/src/gen2/ibv_send.c:878: ibv_post_sr (post_send_desc): Invalid argument (22)
>>>> failed while avail wqe is 63, rail 0
>>>> [vcn04:mpi_rank_2][post_srq_send] src/mpid/ch3/channels/mrail/src/gen2/ibv_send.c:878: ibv_post_sr (post_send_desc): Invalid argument (22)
>>>> [vcn04:mpispawn_1][readline] Unexpected End-Of-File on file descriptor 6. MPI process died?
>>>> [vcn04:mpispawn_1][mtpmi_processops] Error while reading PMI socket. MPI process died?
>>>> [vcn04:mpispawn_1][child_handler] MPI process (rank: 3, pid: 27658) exited with status 255
>>>> [vcn04:mpispawn_1][child_handler] MPI process (rank: 2, pid: 27657) exited with status 255
>>>> [root at vcn03 pasokan]# [vcn04:mpispawn_1][report_error] connect() failed: Connection refused (111)
>>>>
>>>> Thanks
>>>> Pharthiphan
>>>> ________________________________________
>>>> From: Xiaoyi Lu [lu.932 at osu.edu]
>>>> Sent: Monday, March 12, 2018 9:27 PM
>>>> To: Pharthiphan Asokan
>>>> Cc: mvapich-discuss at cse.ohio-state.edu
>>>> Subject: Re: [mvapich-discuss] how to build mvapich2 so to make it work on SR-IOV + KVM Enviroment
>>>>
>>>> Hi, Pharthiphan,
>>>>
>>>> Thanks for your interest in our project.
>>>>
>>>> For virtualized environments (like SR-IOV + KVM), we strongly recommend our users to use the MVAPICH2-Virt package. You can download it from http://mvapich.cse.ohio-state.edu/downloads/.
>>>>
>>>> You can find the user guide of MVAPICH2-Virt from http://mvapich.cse.ohio-state.edu/userguide/virt/.
>>>>
>>>> Thanks,
>>>> Xiaoyi
>>>>
>>>>> On Mar 12, 2018, at 1:27 AM, Pharthiphan Asokan <pasokan at ddn.com> wrote:
>>>>>
>>>>> Hi All,
>>>>>
>>>>> how to build and run mvapich2 so to make it work on SR-IOV + KVM Enviroment
>>>>>
>>>>> I tried the following, then ended up with Could not modify boot qp to RTR
>>>>>
>>>>> [root at vcn01 ~]# /opt/ddn/mvapich/bin/mpirun -hosts vcn01,vcn02 -ppn 1 /opt/ddn/ior/bin/IOR-mvapich -a POSIX -w -r -t
>>>>> 1m -b 1m -o /tmp/pasokan_fuse/test1
>>>>> [src/mpid/ch3/channels/mrail/src/gen2/ring_startup.c:292] error(22): Could not modify boot qp to RTR
>>>>> [src/mpid/ch3/channels/mrail/src/gen2/ring_startup.c:292] error(22): Could not modify boot qp to RTR
>>>>> ===================================================================================
>>>>> =
>>>>> BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
>>>>> =
>>>>> PID 24752 RUNNING AT vcn02
>>>>> =
>>>>> EXIT CODE: 1
>>>>> =
>>>>> CLEANING UP REMAINING PROCESSES
>>>>> =
>>>>> YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
>>>>> ===================================================================================
>>>>> [proxy:0:0 at vcn01] HYDU_sock_write (utils/sock/sock.c:286): write error (Broken pipe)
>>>>> [proxy:0:0 at vcn01] main (pm/pmiserv/pmip.c:265): unable to send EXIT_STATUS command upstream
>>>>> [mpiexec at vcn01] HYDT_bscu_wait_for_completion (tools/bootstrap/utils/bscu_wait.c:76): one of the processes terminated
>>>>> badly; aborting
>>>>> [mpiexec at vcn01] HYDT_bsci_wait_for_completion (tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting
>>>>> for completion
>>>>> [mpiexec at vcn01] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:218): launcher returned error waiting for
>>>>> completion
>>>>> [mpiexec at vcn01] main (ui/mpich/mpiexec.c:344): process manager error waiting for completion
>>>>> [root at vcn01 ~]#
>>>>>
>>>>> Regrads,
>>>>> Pharthiphan
>>>>> _______________________________________________
>>>>> mvapich-discuss mailing list
>>>>> mvapich-discuss at cse.ohio-state.edu
>>>>> http://mailman.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>>>>
> _______________________________________________
> mvapich-discuss mailing list
> mvapich-discuss at cse.ohio-state.edu
> http://mailman.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
More information about the mvapich-discuss
mailing list