ARM Cortex-A53 Interrupt Distribution and GIC Configuration Issues

Ethernet Interrupts Limited to Core0 Despite SMP Linux Configuration

In a typical ARM Cortex-A53-based system running a 64-bit SMP Linux kernel (version 4.9), interrupts are expected to be distributed across all available cores to ensure balanced workload and optimal performance. However, in this scenario, Ethernet interrupts are observed to be arriving exclusively on Core0, even though Core0 is already heavily occupied with other interrupt handling tasks. This results in increased latency and reduced throughput for Ethernet packet transmission. Manually redistributing the Ethernet interrupts to other cores using smp_affinity improves performance significantly, indicating that the default interrupt distribution mechanism is not functioning as expected.

The ARM Cortex-A53 processor relies on the Generic Interrupt Controller (GIC) for interrupt management. The GIC is responsible for receiving interrupts from peripherals and distributing them to the appropriate CPU cores. In an SMP Linux environment, the kernel is expected to handle interrupt distribution across cores automatically. However, this behavior depends on proper configuration of the GIC, the Linux kernel, and the Device Tree Source (DTS). Misconfigurations or missing settings in any of these components can lead to suboptimal interrupt distribution, as seen in this case.

GIC Initialization and SMP Linux Interrupt Distribution Mechanisms

The root cause of the issue lies in the interaction between the GIC, the Linux kernel, and the SMP architecture. The GIC is initialized via the Device Tree Source (DTS), which describes the hardware configuration to the Linux kernel. If the GIC initialization is incomplete or incorrect, the kernel may not be able to utilize the GIC’s interrupt distribution capabilities effectively. Additionally, the Linux kernel relies on certain configuration options (e.g., CONFIG_SMP) and runtime mechanisms (e.g., irqbalance) to distribute interrupts across cores. If these mechanisms are not properly enabled or configured, interrupts may default to a single core.

The GIC supports two modes of operation: 1) Distributor mode, where the GIC distributes interrupts to CPU cores based on priority and affinity settings, and 2) CPU interface mode, where each CPU core interacts with the GIC independently. For interrupt distribution to work correctly in an SMP environment, the GIC must be configured in Distributor mode, and the Linux kernel must be aware of this configuration. The kernel’s SMP support also plays a critical role in managing interrupt affinity and balancing.

In this case, the Ethernet interrupts are not being distributed across cores, suggesting that either the GIC is not configured correctly, or the Linux kernel is not utilizing the GIC’s distribution capabilities. The use of smp_affinity to manually redistribute interrupts confirms that the GIC hardware is functional, but the automatic distribution mechanism is not working as intended.

Configuring GIC and Linux Kernel for Optimal Interrupt Distribution

To resolve the issue, several steps must be taken to ensure proper configuration of the GIC and the Linux kernel. These steps include verifying the GIC initialization in the DTS, enabling relevant kernel configuration options, and using tools like irqbalance to manage interrupt distribution at runtime.

Verifying GIC Initialization in the Device Tree Source (DTS)

The Device Tree Source (DTS) is responsible for describing the hardware configuration to the Linux kernel. For the GIC to function correctly, the DTS must include the appropriate entries for the GIC node. A typical GIC node in the DTS looks like this:

gic: interrupt-controller@address {
    compatible = "arm,gic-400";
    #interrupt-cells = <3>;
    interrupt-controller;
    reg = <0xaddress 0x1000>,
          <0xaddress 0x2000>,
          <0xaddress 0x4000>,
          <0xaddress 0x8000>;
    interrupts = <GIC_PPI 9 (GIC_CPU_MASK_SIMPLE(4) | IRQ_TYPE_LEVEL_HIGH)>;
};

Key points to verify in the DTS include:

• The compatible property must match the GIC version (e.g., "arm,gic-400" for GIC-400).
• The reg property must specify the correct base addresses for the GIC distributor and CPU interfaces.
• The interrupts property must define the interrupt lines for the GIC.

If the GIC node is missing or incorrectly configured, the Linux kernel will not be able to initialize the GIC properly, leading to interrupt distribution issues.

Enabling SMP and GIC Support in the Linux Kernel

The Linux kernel must be configured to support SMP and the GIC. This involves enabling the following configuration options in the kernel build:

• CONFIG_SMP: Enables Symmetric Multi-Processing (SMP) support, allowing the kernel to utilize multiple CPU cores.
• CONFIG_ARM_GIC: Enables support for the Generic Interrupt Controller (GIC).
• CONFIG_IRQ_DOMAIN: Enables interrupt domain support, which is required for the GIC to map interrupts to CPU cores.

These options can be enabled in the kernel configuration file (.config) or via the make menuconfig interface. After enabling these options, rebuild the kernel and ensure that the new kernel image is deployed to the target system.

Using irqbalance for Runtime Interrupt Distribution

While the Linux kernel includes basic support for interrupt distribution, it may not always distribute interrupts optimally. The irqbalance tool can be used to dynamically balance interrupts across CPU cores at runtime. irqbalance monitors the system’s interrupt load and adjusts the affinity of interrupts to ensure that no single core is overwhelmed.

To use irqbalance, install the package on the target system and start the service:

sudo apt-get install irqbalance
sudo systemctl start irqbalance
sudo systemctl enable irqbalance

irqbalance works in conjunction with the kernel’s SMP and GIC support to provide a more balanced interrupt distribution. It is particularly useful in systems with high interrupt loads or uneven workloads.

Implementing Manual Interrupt Affinity with smp_affinity

In cases where automatic interrupt distribution is not sufficient, manual affinity settings can be used to assign specific interrupts to specific CPU cores. The smp_affinity file in the /proc/irq/ directory controls the affinity of each interrupt. For example, to assign Ethernet interrupts to Core1, use the following command:

echo 2 > /proc/irq/<irq_number>/smp_affinity

Here, <irq_number> is the interrupt number for the Ethernet controller, and 2 represents Core1 (since the affinity value is a bitmask, where each bit corresponds to a CPU core). This approach can be used to fine-tune interrupt distribution for specific workloads.

Debugging and Monitoring Interrupt Distribution

To diagnose interrupt distribution issues, use tools like cat /proc/interrupts to view the interrupt counts for each CPU core. This provides insight into how interrupts are being distributed and whether any cores are being overloaded. Additionally, the mpstat command can be used to monitor CPU utilization and identify cores that are handling excessive interrupt loads.

cat /proc/interrupts
mpstat -P ALL 1

These tools help identify imbalances in interrupt distribution and guide further tuning of the GIC and kernel configuration.

By following these steps, the issue of Ethernet interrupts being limited to Core0 can be resolved, ensuring optimal performance and balanced workload distribution across all CPU cores in the ARM Cortex-A53 system.