Distributing Single SPI to Multiple PEs Concurrently in GIC-600

ARM Cortex-A55 and GIC-600: SPI Interrupt Distribution Challenges in AMP Systems

In systems utilizing the ARM Cortex-A55 cores alongside the GIC-600 interrupt controller, a common challenge arises when attempting to distribute a single Shared Peripheral Interrupt (SPI) to multiple Processing Elements (PEs) concurrently. This scenario is particularly relevant in Asymmetric Multiprocessing (AMP) systems, where one core runs a baremetal application while others operate under a Linux environment. The goal is to ensure that a precise synchronization signal, such as an External Interrupt (EINT) from a GPIO, can interrupt multiple subsystems simultaneously. However, the GIC-600’s architecture imposes limitations on how SPIs can be targeted, making this task non-trivial.

The GIC-600, compliant with the GICv3 architecture, provides two primary targeting options for SPIs: Targeted and 1ofN. In the Targeted mode, the interrupt is directed to a specific PE as configured by software. In the 1ofN mode, the GIC selects a target PE from a group each time the interrupt becomes pending. However, neither mode supports the concurrent delivery of a single SPI to multiple PEs. This limitation necessitates alternative approaches to achieve the desired behavior, such as using Software Generated Interrupts (SGIs) or connecting the interrupt source to multiple INTIDs at design time.

GICv3 SPI Targeting Modes and Their Limitations

The GICv3 architecture, which underpins the GIC-600, offers two distinct modes for targeting SPIs: Targeted and 1ofN. In the Targeted mode, the software explicitly specifies a target PE for each SPI. Once configured, the GIC will only deliver the interrupt to the designated PE. This mode is straightforward but lacks flexibility when multiple PEs need to respond to the same interrupt.

The 1ofN mode, on the other hand, allows the GIC to dynamically select a target PE from a group each time the interrupt becomes pending. This mode provides a degree of load balancing, as the GIC can distribute interrupts across multiple PEs over time. However, it is important to note that support for the 1ofN mode is optional in the GICv3 specification, meaning that not all implementations may offer this feature. Furthermore, even in 1ofN mode, a given instance of an SPI can only be acknowledged by a single PE at any given time. This restriction prevents the concurrent delivery of the same SPI to multiple PEs, which is a critical requirement in certain AMP configurations.

The inability to concurrently deliver a single SPI to multiple PEs stems from the GICv3 architecture’s design principles, which prioritize simplicity and efficiency in interrupt handling. Allowing multiple PEs to acknowledge the same SPI would introduce complexities in managing interrupt states and could lead to race conditions or inconsistent system behavior. As a result, developers must explore alternative strategies to achieve the desired functionality.

Implementing SPI Distribution via SGIs and Multiple INTIDs

Given the limitations of the GICv3 architecture, there are two primary strategies for distributing a single SPI to multiple PEs in an AMP system: using Software Generated Interrupts (SGIs) and connecting the interrupt source to multiple INTIDs at design time.

The first approach involves targeting the SPI at a single PE and having that PE send SGIs to the other PEs as part of its interrupt handler. SGIs are a type of interrupt that can be generated by software and are typically used for inter-processor communication. In this scenario, the PE that receives the SPI can generate SGIs to notify the other PEs of the event. This method allows multiple PEs to respond to the same interrupt, albeit indirectly. However, it introduces additional latency, as the secondary PEs must wait for the SGI to be generated and delivered. Furthermore, this approach requires careful coordination to ensure that the SGIs are sent and handled correctly, which can complicate the software design.

The second approach involves connecting the interrupt source to multiple INTIDs on the GIC at design time. By wiring the interrupt to multiple inputs on the GIC, each input can be configured to target a different PE. This method allows the same interrupt signal to be delivered to multiple PEs concurrently, as each INTID is treated as a separate interrupt by the GIC. However, this approach requires careful planning during the hardware design phase, as it involves modifying the interrupt routing at the physical level. Additionally, it increases the complexity of the interrupt handling software, as each PE must manage its own instance of the interrupt.

Both approaches have their trade-offs, and the choice between them depends on the specific requirements and constraints of the system. In general, the SGI-based approach is more flexible and can be implemented in software, making it suitable for systems where hardware modifications are not feasible. On the other hand, the multiple INTID approach offers lower latency and more deterministic behavior, making it preferable for systems with stringent timing requirements.

Detailed Troubleshooting and Implementation Steps

To implement the SGI-based approach, follow these steps:

1. Configure the SPI to target a single PE in the GIC. This can be done by setting the appropriate bits in the GICD_IROUTER register for the SPI in question. The target PE should be the one that will handle the initial interrupt and generate the SGIs.

2. In the interrupt handler for the target PE, generate SGIs to notify the other PEs of the event. This can be done using the GICD_SGIR register. The SGIs should be targeted at the other PEs in the system, and the handler should ensure that the SGIs are sent before exiting.

3. Implement interrupt handlers for the SGIs on the secondary PEs. These handlers should perform the necessary actions in response to the interrupt, such as updating shared data structures or triggering further processing.

4. Ensure that the interrupt handlers on all PEs are properly synchronized to avoid race conditions or inconsistent states. This may involve using memory barriers, spinlocks, or other synchronization mechanisms.

To implement the multiple INTID approach, follow these steps:

1. Modify the hardware design to connect the interrupt source to multiple inputs on the GIC. Each input should be assigned a unique INTID, and the routing should be configured to target a different PE for each INTID.

2. Configure each INTID in the GIC to target the appropriate PE. This can be done by setting the appropriate bits in the GICD_IROUTER register for each INTID.

3. Implement interrupt handlers for each INTID on the corresponding PEs. These handlers should perform the necessary actions in response to the interrupt, such as updating shared data structures or triggering further processing.

4. Ensure that the interrupt handlers on all PEs are properly synchronized to avoid race conditions or inconsistent states. This may involve using memory barriers, spinlocks, or other synchronization mechanisms.

In both approaches, it is important to thoroughly test the system to ensure that the interrupts are being delivered and handled correctly. This may involve using debug tools to monitor interrupt delivery and handling, as well as stress testing the system to verify its behavior under heavy load.

Conclusion

Distributing a single SPI to multiple PEs concurrently in a system using the ARM Cortex-A55 and GIC-600 presents a significant challenge due to the limitations of the GICv3 architecture. However, by leveraging SGIs or multiple INTIDs, it is possible to achieve the desired behavior with careful planning and implementation. Each approach has its trade-offs, and the choice between them should be guided by the specific requirements and constraints of the system. By following the detailed steps outlined above, developers can successfully implement SPI distribution in their AMP systems, ensuring reliable and efficient interrupt handling across multiple PEs.