Xen在使用SR-IOV網卡時xl pci-detach 失敗-有解無憂

Hi,
I've come across a need to hotplug PCI devices between dom0 and domU using SR-IOV NIC. But I'm experiencing problems when trying to detach VF more than one PV guests.
I can attach VF to DomU successful as follow:

參考

# xl pci-assignable-list
0000:05:10.0
0000:05:10.2
# xl pci-attach 1 05:10.0
# xl pci-attach 2 05:10.2

It seems that everything is working fine and it can work well in DomU. But when I detach VF it report errors as follow:

參考

# xl pci-detach 2 05:10.2
libxl: error: libxl_device.c:1269:libxl__wait_for_backend: Backend /local/domain/0/backend/pci/2/0 not ready

Each guest config is the same like this:

參考

name = "ubuntu-pv-1"
bootloader = "pygrub"
memory = 256
vcpus = 1
vif = [ 'bridge=xenbr0' ]
disk = [ 'file:/home/ye/ubuntu-pv/ubuntu-pv-1/ubuntu-pv-1.img,xvda,rw' ]
pci_permissive = 1

Follow xl dmesg log:

參考

Xen 4.9.1
(XEN) Xen version 4.9.1 (ye@) (gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-28)) debug=n  Mon Jun  4 11:34:35 CST 2018
(XEN) Latest ChangeSet:
(XEN) Bootloader: GRUB 2.02~beta2
(XEN) Command line: placeholder dom0_mem=2048M,max:2048M dom0_max_vcpus=4 iommu=dom0-passthrough
(XEN) Xen image load base address: 0
(XEN) Video information:
(XEN)  VGA is text mode 80x25, font 8x16
(XEN)  VBE/DDC methods: none; EDID transfer time: 0 seconds
(XEN)  EDID info not retrieved because no DDC retrieval method detected
(XEN) Disc information:
(XEN)  Found 1 MBR signatures
(XEN)  Found 4 EDD information structures
(XEN) Xen-e820 RAM map:
(XEN)  0000000000000000 - 000000000009c000 (usable)
(XEN)  000000000009c000 - 00000000000a0000 (reserved)
(XEN)  00000000000e0000 - 0000000000100000 (reserved)
(XEN)  0000000000100000 - 000000007a289000 (usable)
(XEN)  000000007a289000 - 000000007af0b000 (reserved)
(XEN)  000000007af0b000 - 000000007b93b000 (ACPI NVS)
(XEN)  000000007b93b000 - 000000007bab7000 (ACPI data)
(XEN)  000000007bab7000 - 000000007bae9000 (usable)
(XEN)  000000007bae9000 - 000000007baff000 (ACPI data)
(XEN)  000000007baff000 - 000000007bb00000 (usable)
(XEN)  000000007bb00000 - 0000000090000000 (reserved)
(XEN)  00000000feda8000 - 00000000fedac000 (reserved)
(XEN)  00000000ff310000 - 0000000100000000 (reserved)
(XEN)  0000000100000000 - 0000000880000000 (usable)
(XEN) New Xen image base address: 0x79c00000
(XEN) ACPI: RSDP 000FE320, 0024 (r2 DELL  )
(XEN) ACPI: XSDT 7BAB50E8, 00BC (r1 DELL   PE_SC3          0       1000013)
(XEN) ACPI: FACP 7BAB1000, 00F4 (r4 DELL   PE_SC3          0 DELL        1)
(XEN) ACPI: DSDT 7BA99000, 10768 (r2 DELL   PE_SC3          3 DELL        1)
(XEN) ACPI: FACS 7B8E3000, 0040
(XEN) ACPI: MCEJ 7BAB4000, 0130 (r1 INTEL                  1 INTL  100000D)
(XEN) ACPI: WD__ 7BAB3000, 0134 (r1 DELL   PE_SC3          1 DELL        1)
(XEN) ACPI: SLIC 7BAB2000, 0024 (r1 DELL   PE_SC3          1 DELL        1)
(XEN) ACPI: HPET 7BAB0000, 0038 (r1 DELL   PE_SC3          1 DELL        1)
(XEN) ACPI: APIC 7BAAF000, 0AFC (r2 DELL   PE_SC3          0 DELL        1)
(XEN) ACPI: MCFG 7BAAE000, 003C (r1 DELL   PE_SC3          1 DELL        1)
(XEN) ACPI: MSCT 7BAAD000, 0090 (r1 DELL   PE_SC3          1 DELL        1)
(XEN) ACPI: SLIT 7BAAC000, 006C (r1 DELL   PE_SC3          1 DELL        1)
(XEN) ACPI: SRAT 7BAAA000, 1130 (r3 DELL   PE_SC3          1 DELL        1)
(XEN) ACPI: SSDT 7B956000, 1424A9 (r2 DELL   PE_SC3       4000 INTL 20121114)
(XEN) ACPI: SSDT 7B953000, 217F (r2 DELL   PE_SC3          2 INTL 20121114)
(XEN) ACPI: SSDT 7B952000, 006E (r2 DELL   PE_SC3          2 INTL 20121114)
(XEN) ACPI: PRAD 7B951000, 0132 (r2   DELL PE_SC3          2 INTL 20121114)
(XEN) ACPI: DMAR 7BAFE000, 0120 (r1 DELL   PE_SC3          1 DELL        1)
(XEN) ACPI: HEST 7BAFD000, 017C (r1 DELL   PE_SC3          2 DELL        1)
(XEN) ACPI: BERT 7BAFC000, 0030 (r1 DELL   PE_SC3          2 DELL        1)
(XEN) ACPI: ERST 7BAFB000, 0230 (r1 DELL   PE_SC3          2 DELL        1)
(XEN) ACPI: EINJ 7BAFA000, 0150 (r1 DELL   PE_SC3          2 DELL        1)
(XEN) System RAM: 32674MB (33458528kB)
(XEN) SRAT: Node 1 has no memory. BIOS Bug or mis-configured hardware?
(XEN) Domain heap initialised DMA width 32 bits
(XEN) ACPI: 32/64X FACS address mismatch in FADT - 7b8e3000/0000000000000000, using 32
(XEN) IOAPIC[0]: apic_id 8, version 32, address 0xfec00000, GSI 0-23
(XEN) IOAPIC[1]: apic_id 9, version 32, address 0xfec01000, GSI 24-47
(XEN) IOAPIC[2]: apic_id 10, version 32, address 0xfec40000, GSI 48-71
(XEN) Enabling APIC mode:  Phys.  Using 3 I/O APICs
(XEN) Not enabling x2APIC (upon firmware request)
(XEN) xstate: size: 0x340 and states: 0x7
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Platform timer is 14.318MHz HPET
(XEN) Detected 2200.002 MHz processor.
(XEN) Initing memory sharing.
(XEN) Intel VT-d iommu 0 supported page sizes: 4kB, 2MB, 1GB.
(XEN) Intel VT-d iommu 1 supported page sizes: 4kB, 2MB, 1GB.
(XEN) Intel VT-d Snoop Control enabled.
(XEN) Intel VT-d Dom0 DMA Passthrough enabled.
(XEN) Intel VT-d Queued Invalidation enabled.
(XEN) Intel VT-d Interrupt Remapping enabled.
(XEN) Intel VT-d Posted Interrupt not enabled.
(XEN) Intel VT-d Shared EPT tables enabled.
(XEN) I/O virtualisation enabled
(XEN)  - Dom0 mode: Passthrough
(XEN) Interrupt remapping enabled
(XEN) Enabled directed EOI with ioapic_ack_old on!
(XEN) ENABLING IO-APIC IRQs
(XEN)  -> Using old ACK method
(XEN) Allocated console ring of 128 KiB.
(XEN) VMX: Supported advanced features:
(XEN)  - APIC MMIO access virtualisation
(XEN)  - APIC TPR shadow
(XEN)  - Extended Page Tables (EPT)
(XEN)  - Virtual-Processor Identifiers (VPID)
(XEN)  - Virtual NMI
(XEN)  - MSR direct-access bitmap
(XEN)  - Unrestricted Guest
(XEN)  - APIC Register Virtualization
(XEN)  - Virtual Interrupt Delivery
(XEN)  - Posted Interrupt Processing
(XEN)  - VMCS shadowing
(XEN)  - VM Functions
(XEN)  - Virtualisation Exceptions
(XEN)  - Page Modification Logging
(XEN) HVM: ASIDs enabled.
(XEN) HVM: VMX enabled
(XEN) HVM: Hardware Assisted Paging (HAP) detected
(XEN) HVM: HAP page sizes: 4kB, 2MB, 1GB
(XEN) Brought up 40 CPUs
(XEN) Dom0 has maximum 840 PIRQs
(XEN) *** LOADING DOMAIN 0 ***
(XEN)  Xen  kernel: 64-bit, lsb, compat32
(XEN)  Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x2078000
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN)  Dom0 alloc.:   0000000860000000->0000000864000000 (502576 pages to be allocated)
(XEN)  Init. ramdisk: 000000087eb30000->000000087ffff057
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN)  Loaded kernel: ffffffff81000000->ffffffff82078000
(XEN)  Init. ramdisk: 0000000000000000->0000000000000000
(XEN)  Phys-Mach map: 0000008000000000->0000008000400000
(XEN)  Start info:    ffffffff82078000->ffffffff820784b4
(XEN)  Page tables:   ffffffff82079000->ffffffff8208e000
(XEN)  Boot stack:    ffffffff8208e000->ffffffff8208f000
(XEN)  TOTAL:         ffffffff80000000->ffffffff82400000
(XEN)  ENTRY ADDRESS: ffffffff81c051f0
(XEN) Dom0 has maximum 4 VCPUs
(XEN) Scrubbing Free RAM on 2 nodes using 10 CPUs
(XEN) ............................done.
(XEN) Initial low memory virq threshold set at 0x4000 pages.
(XEN) Std. Loglevel: Errors and warnings
(XEN) Guest Loglevel: Nothing (Rate-limited: Errors and warnings)
(XEN) Xen is relinquishing VGA console.
(XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch input to Xen)
(XEN) Freed 440kB init memory
(XEN) deassign 0000:05:10.2 from dom2 failed (-19)

And libxl-driver log:

參考

# cat /var/log/libvirt/libxl/libxl-driver.log
xc: detail: sysctl operation failed -- need to rebuild the user-space tool set?
libxl: error: libxl.c:4364:libxl_get_physinfo: getting physinfo: Permission denied
xc: debug: hypercall buffer: total allocations:7 total releases:7
xc: debug: hypercall buffer: current allocations:0 maximum allocations:1
xc: debug: hypercall buffer: cache current size:1
xc: debug: hypercall buffer: cache hits:6 misses:1 toobig:0

I'm running in Ubuntu 14 using Xen-4.9. I have tested in other machine using the same environment but encountering the same problem.

轉載請註明出處，本文鏈接：https://www.uj5u.com/qita/50598.html

標籤：虛擬化

上一篇：特征工程

下一篇：阿里云建站全程序