Virtual Interrupt Deactivation Failure in GICv2 on ARM AArch64 Systems

Virtual Interrupt Handling and GICv2 Behavior in AArch64 Hypervisors

The issue at hand revolves around the failure of virtual interrupt deactivation in a system utilizing the Generic Interrupt Controller version 2 (GICv2) on an ARM AArch64 architecture. The system in question involves a hypervisor running at Exception Level 2 (EL2), managing a virtual machine that interacts with a virtualized GICv2. The specific problem manifests when the guest operating system (Linux) writes to the GICV_EOIR (End of Interrupt Register) to deactivate a virtual interrupt, but the corresponding physical interrupt is not deactivated as expected. This results in the interrupt only triggering once, despite the guest OS successfully processing the interrupt and writing to the GICV_EOIR.

The virtual interrupt in question is a virtual timer interrupt with interrupt number 27. The hypervisor traps the interrupt from the guest, injects it into the virtual machine, and the guest processes it. The GICH_LR (List Register) state transitions correctly from pending (01) to active (10) and finally to deactivated (00). However, the physical interrupt remains active, and no maintenance interrupt is triggered despite the guest writing to GICV_EOIR. This behavior contradicts the expected operation of GICv2, where writing to GICV_EOIR should deactivate both the virtual and physical interrupts under specific conditions.

The conditions for successful deactivation, as per the GICv2 documentation, are:

• The GICV_CTLR.EOImode bit must be set to 0.
• The GICH_LRn.HW bit must be set to 1.

Both conditions have been verified to be true in this scenario. The GICV_CTLR.EOImode bit is 0, and the GICH_LRn.HW bit remains set to 1 throughout the interrupt lifecycle. Despite this, the physical interrupt is not deactivated, and no maintenance interrupt is generated.

Potential Causes of Virtual Interrupt Deactivation Failure

The failure of virtual interrupt deactivation in GICv2 can be attributed to several potential causes, each of which must be carefully examined to identify the root cause of the issue.

Incorrect Maintenance Interrupt Configuration

One possible cause is the incorrect configuration of the maintenance interrupt. The GICH_HCR (Hypervisor Control Register) is configured with LRENPIE (List Register Entry Not Present Interrupt Enable) set to 1 and the global enable bit (En) set to 1. However, the LRENPIE bit is designed to generate a maintenance interrupt when a deactivate operation (write to GICV_EOIR or GICV_DIR) is performed, and the specified INTID does not match any of the List Registers. In this case, since the INTID matches a List Register, the EOICount is not incremented, and no maintenance interrupt is generated. This behavior is expected, but it does not explain why the physical interrupt is not deactivated.

Core Affinity and Physical Interrupt Routing

Another potential cause is related to core affinity and the routing of physical interrupts. The virtual timer interrupt is a Peripheral Private Interrupt (PPI), which is specific to a particular core. If the virtual machine is not running on the same physical core where the physical interrupt was originally generated, the deactivation of the virtual interrupt may not propagate correctly to the physical interrupt. This could result in the physical interrupt remaining active even after the virtual interrupt has been deactivated.

GICV_EOIR Write Timing and Synchronization Issues

Timing and synchronization issues could also play a role in the failure of interrupt deactivation. If the write to GICV_EOIR is not properly synchronized with the state of the GICv2 hardware, the deactivation signal may not be correctly processed. This could result in the physical interrupt remaining active. Additionally, if there are delays or race conditions in the hypervisor’s handling of the interrupt deactivation, the physical interrupt may not be deactivated in time.

Hypervisor Implementation Bugs

Bugs in the hypervisor’s implementation of the GICv2 virtualization could also be a cause. If the hypervisor does not correctly handle the deactivation of virtual interrupts, or if there are errors in the mapping between virtual and physical interrupts, the physical interrupt may not be deactivated as expected. This could be due to incorrect handling of the GICH_LR registers, improper configuration of the GICV_CTLR, or other implementation errors.

Detailed Troubleshooting Steps and Solutions for GICv2 Virtual Interrupt Deactivation

To resolve the issue of virtual interrupt deactivation failure in GICv2, a systematic approach to troubleshooting and problem resolution is required. The following steps outline a detailed process for identifying and addressing the root cause of the issue.

Step 1: Verify GICv2 Configuration and Register States

The first step is to thoroughly verify the configuration and state of the GICv2 registers, particularly those involved in virtual interrupt handling. This includes:

• GICV_CTLR: Ensure that the EOImode bit is set to 0, as required for proper interrupt deactivation.
• GICH_LR: Verify that the HW bit (bit 31) remains set to 1 throughout the interrupt lifecycle, and that the state transitions correctly from pending (01) to active (10) and finally to deactivated (00).
• GICH_HCR: Confirm that the LRENPIE bit is set to 1 and that the global enable bit (En) is also set to 1. This ensures that the virtual CPU interface is enabled and that maintenance interrupts are configured correctly.

Step 2: Check Core Affinity and Physical Interrupt Routing

Next, verify that the virtual machine is running on the same physical core where the physical interrupt was originally generated. This is particularly important for PPIs, which are core-specific. If the virtual machine is not running on the correct core, the deactivation of the virtual interrupt may not propagate to the physical interrupt. To check this:

• Core Affinity: Ensure that the virtual machine is pinned to the correct physical core. This can be done by checking the affinity settings in the hypervisor and the guest OS.
• Physical Interrupt Routing: Verify that the physical interrupt is correctly routed to the core where the virtual machine is running. This may involve checking the GICD (Distributor) configuration and the routing tables in the hypervisor.

Step 3: Investigate Timing and Synchronization Issues

Timing and synchronization issues can be difficult to diagnose, but they are a common cause of interrupt handling problems. To investigate these issues:

• Synchronization Barriers: Ensure that appropriate synchronization barriers are in place to guarantee that the write to GICV_EOIR is properly synchronized with the state of the GICv2 hardware. This may involve adding memory barriers or other synchronization mechanisms in the hypervisor code.
• Interrupt Latency: Measure the latency between the guest writing to GICV_EOIR and the hypervisor processing the deactivation. If there are significant delays, this could indicate a timing issue that needs to be addressed.

Step 4: Debug Hypervisor Implementation

If the above steps do not resolve the issue, it may be necessary to debug the hypervisor’s implementation of GICv2 virtualization. This involves:

• Code Review: Carefully review the hypervisor code responsible for handling virtual interrupts, particularly the code that interacts with the GICH_LR registers and the GICV_EOIR. Look for any potential bugs or incorrect assumptions.
• Logging and Tracing: Add detailed logging and tracing to the hypervisor code to capture the state of the GICv2 registers and the flow of interrupt handling. This can help identify where the deactivation process is failing.
• Testing with Different Interrupts: Test the system with different types of interrupts (e.g., SPIs, LPIs) to see if the issue is specific to PPIs or if it occurs with other interrupt types as well.

Step 5: Implement Workarounds and Fixes

Based on the findings from the previous steps, implement appropriate workarounds or fixes to resolve the issue. Possible solutions include:

• Core Affinity Fixes: If the issue is related to core affinity, ensure that the virtual machine is always running on the correct physical core. This may involve modifying the hypervisor’s scheduling algorithm or the guest OS’s affinity settings.
• Synchronization Fixes: If timing or synchronization issues are identified, add appropriate synchronization mechanisms to ensure that the write to GICV_EOIR is properly processed by the GICv2 hardware.
• Hypervisor Bug Fixes: If bugs are found in the hypervisor’s implementation, correct them to ensure proper handling of virtual interrupts. This may involve fixing incorrect register handling, improving the mapping between virtual and physical interrupts, or addressing other implementation errors.

Step 6: Validate the Fixes

Finally, validate the fixes by thoroughly testing the system with the virtual timer interrupt and other types of interrupts. Ensure that:

• Interrupts Trigger Correctly: Verify that interrupts are triggered correctly and that the guest OS can process them as expected.
• Interrupts Deactivate Correctly: Confirm that writing to GICV_EOIR results in the correct deactivation of both the virtual and physical interrupts.
• Maintenance Interrupts: Ensure that maintenance interrupts are generated as expected when required.

By following these detailed troubleshooting steps, the issue of virtual interrupt deactivation failure in GICv2 can be systematically identified and resolved, ensuring reliable interrupt handling in the virtualized environment.