GICv3 LPI Interrupt Delivery Failure After Bootloader Transition
The issue at hand involves a custom bootloader that loads an image from an NVMe device, utilizing LPI (Locality-specific Peripheral Interrupts) interrupts routed through the GICv3 ITS (Interrupt Translation Service). The bootloader successfully loads the next image into RAM and transfers control to it. However, the second image re-initializes the GIC, including the ITS device, and subsequently, no LPI interrupts are delivered. A critical observation is that the LPI enable bit in the redistributor register remains stuck at 1, preventing it from being reset to 0. This behavior suggests a potential misconfiguration or hardware-software interaction issue within the GICv3 architecture, specifically related to the LPI enable bit and ITS re-initialization.
The GICv3 architecture is designed to handle a large number of interrupts efficiently, with LPIs being a key feature for message-based interrupts. The ITS is responsible for translating device-specific interrupt identifiers into virtual interrupts that the GIC can handle. The redistributor is a key component that manages interrupt routing and prioritization for each CPU. The LPI enable bit in the redistributor register controls whether LPIs are enabled for a specific CPU. If this bit cannot be reset, it indicates a deeper issue within the GICv3 configuration or state management.
The problem manifests during the transition from the bootloader to the second image, where the GICv3 and ITS are re-initialized. The inability to reset the LPI enable bit suggests that the GICv3 state is not being fully cleared or reconfigured during this transition. This could be due to several factors, including improper initialization sequences, missing cache or memory barrier operations, or hardware-specific quirks in the GICv3 implementation.
Redistributor Register State Corruption and ITS Re-initialization Issues
One possible cause of the LPI enable bit being stuck at 1 is state corruption within the redistributor registers. The redistributor is responsible for managing interrupt routing and prioritization for each CPU, and its registers must be properly initialized and cleared during system startup. If the redistributor registers are not correctly reset during the transition from the bootloader to the second image, the LPI enable bit may remain set, preventing LPIs from being delivered.
Another potential cause is improper re-initialization of the ITS device. The ITS is responsible for translating device-specific interrupt identifiers into virtual interrupts that the GIC can handle. If the ITS is not correctly re-initialized, it may fail to properly route LPIs to the redistributor, resulting in no LPI interrupts being delivered. This could be due to missing or incorrect configuration steps during the ITS re-initialization process.
Additionally, the issue could be related to cache coherency or memory barrier operations. The GICv3 architecture relies on proper cache management to ensure that interrupt-related data structures are correctly synchronized between the CPU and the GIC. If cache coherency is not maintained, the redistributor registers may not reflect the correct state, leading to the LPI enable bit being stuck at 1. Similarly, missing memory barriers could result in out-of-order writes to the redistributor registers, causing the LPI enable bit to remain set.
Finally, the problem could be due to hardware-specific quirks or errata in the GICv3 implementation. Some GICv3 implementations may have specific requirements or limitations that are not documented in the architecture specification. For example, certain GICv3 implementations may require additional steps to reset the LPI enable bit or may have specific timing requirements for ITS re-initialization.
Resetting the LPI Enable Bit and Ensuring Proper ITS Re-initialization
To address the issue of the LPI enable bit being stuck at 1, the following troubleshooting steps and solutions can be implemented:
First, ensure that the redistributor registers are properly reset during the transition from the bootloader to the second image. This involves clearing the redistributor registers, including the LPI enable bit, before re-initializing the GICv3. The redistributor registers can be accessed through memory-mapped I/O, and the LPI enable bit can be cleared by writing a 0 to the appropriate bit in the redistributor control register. It is important to follow the GICv3 architecture specification for the correct register offsets and bit positions.
Next, verify that the ITS device is correctly re-initialized during the transition. The ITS must be properly configured to translate device-specific interrupt identifiers into virtual interrupts that the GIC can handle. This involves setting up the ITS tables, including the device table, collection table, and interrupt translation table. The ITS must also be enabled and configured to route interrupts to the correct redistributor. Ensure that all necessary configuration steps are followed, as outlined in the GICv3 architecture specification.
Additionally, ensure that cache coherency and memory barrier operations are properly implemented. The GICv3 architecture relies on proper cache management to ensure that interrupt-related data structures are correctly synchronized between the CPU and the GIC. Use data synchronization barriers (DSB) and instruction synchronization barriers (ISB) to ensure that all writes to the redistributor registers are completed before proceeding with the next steps. This will help prevent out-of-order writes and ensure that the redistributor registers reflect the correct state.
If the issue persists, consider checking for hardware-specific quirks or errata in the GICv3 implementation. Some GICv3 implementations may have specific requirements or limitations that are not documented in the architecture specification. Consult the hardware documentation or contact the hardware vendor for any known issues or workarounds related to the LPI enable bit or ITS re-initialization.
Finally, if all else fails, consider using a debugger to inspect the state of the redistributor registers and ITS configuration during the transition from the bootloader to the second image. This can help identify any discrepancies or unexpected behavior that may be causing the LPI enable bit to remain stuck at 1. Use the debugger to step through the initialization sequence and verify that all registers are being correctly set and cleared.
In summary, the issue of the LPI enable bit being stuck at 1 in the GICv3 redistributor register can be addressed by ensuring proper reset and re-initialization of the redistributor and ITS device, implementing cache coherency and memory barrier operations, and checking for hardware-specific quirks or errata. By following these steps, the LPI enable bit can be successfully reset, allowing LPI interrupts to be delivered as expected.
