ARM Cortex-A53 Spinlock Failure in Multi-Core Environment
The issue at hand involves the failure of a spinlock implementation on an ARM Cortex-A53 multi-core system, specifically on a Xilinx ZCU102 board. The spinlock is designed to synchronize the execution of all Cortex-A53 cores, but only CPU 0 is able to acquire the lock. The other cores perceive the lock as still being held, even after CPU 0 has released it. This behavior suggests a problem with cache coherency or memory visibility across cores, which is critical for proper synchronization in a multi-core environment.
The system is running bare-metal code, with the following components:
- FSBL (First Stage Boot Loader) and PMUFW (Platform Management Unit Firmware) provided by Xilinx.
- Bootloader BL31, compiled from Arm Trusted Firmware for the ZynqMP platform.
- Custom application code running at EL2 (Exception Level 2) in AArch64 mode.
The spinlock implementation uses ARM’s load-acquire (ldaxr) and store-release (stlxr) instructions, which are designed to ensure proper memory ordering and synchronization. However, the lock variable, located in the BSS section at address 0x0000000008014000, is not being correctly observed by all cores after being released by CPU 0. This discrepancy indicates that the memory updates are not being propagated correctly across the cores, likely due to misconfigured memory attributes or cache coherency settings.
Misconfigured Shareability Attributes and Cache Coherency Settings
The root cause of the spinlock failure lies in the misconfiguration of the Virtual Memory System Architecture (VMSA) and cache coherency settings. Specifically, the shareability attributes in the page table entries for the memory region containing the lock variable were not set correctly. In ARM architectures, memory regions must be marked as shareable to ensure that updates to memory are visible to all cores in a multi-core system. Without this attribute, each core may maintain its own local copy of the memory, leading to inconsistencies.
The ARM Cortex-A53 processor supports three levels of shareability:
- Non-shareable (NSH): Memory is not shared between cores.
- Inner Shareable (ISH): Memory is shared within a cluster of cores.
- Outer Shareable (OSH): Memory is shared across clusters or with external devices.
For multi-core synchronization, the memory region containing the lock variable must be marked as Inner Shareable (ISH). This ensures that all cores within the same cluster see a consistent view of the memory. Additionally, the cache coherency mechanism must be enabled to ensure that updates to the lock variable are propagated across the cores’ caches.
The issue is further compounded by the fact that the system is running in EL2, which has its own set of memory management registers (e.g., TCR_EL2, MAIR_EL2, TTBR0_EL2). These registers control the translation and memory attributes for EL2, and any misconfiguration here can lead to incorrect memory behavior. In this case, the shareability attribute was not set in the page table entries, leading to the observed spinlock failure.
Implementing Correct Shareability Attributes and Cache Management
To resolve the spinlock issue, the following steps must be taken to ensure proper memory visibility and cache coherency across all cores:
1. Configure Shareability Attributes in the Page Table
The page table entries for the memory region containing the lock variable must be updated to include the Inner Shareable (ISH) attribute. This ensures that all cores within the cluster see a consistent view of the memory. The shareability attribute is controlled by the SH field in the page table descriptor, which can be set as follows:
- SH = 0b10: Inner Shareable (ISH)
- SH = 0b11: Outer Shareable (OSH)
For the lock variable, the page table entry should be configured with SH = 0b10 to mark the memory as Inner Shareable.
2. Enable Cache Coherency
The ARM Cortex-A53 processor includes a hardware coherency mechanism that ensures cache coherency across cores. This mechanism must be enabled by setting the SMPEN bit in the CPU Extended Control Register (S3_1_c15_c2_1). The following assembly code can be used to enable hardware coherency:
/* Enable hardware coherency between cores */
mrs x0, S3_1_c15_c2_1 // Read EL1 CPU Extended Control Register
orr x0, x0, #(1 << 6) // Set the SMPEN bit
msr S3_1_c15_c2_1, x0 // Write EL1 CPU Extended Control Register
isb // Instruction Synchronization Barrier
This code sets the SMPEN bit, which enables cache coherency between cores. The isb instruction ensures that the change takes effect immediately.
3. Verify Memory Attributes in the Translation Table
The translation table must be carefully configured to ensure that the memory attributes are correctly applied. The following table summarizes the key memory attributes that must be set for the lock variable:
| Attribute | Value | Description |
|---|---|---|
| Memory Type | Normal Memory | The lock variable should be mapped as normal memory, not device memory. |
| Cacheability | Write-Back Cacheable | The memory should be cacheable to benefit from the cache coherency mechanism. |
| Shareability | Inner Shareable | The memory must be marked as Inner Shareable to ensure visibility across cores. |
The translation table entries should be updated to reflect these attributes. For example, the following descriptor sets the memory as Normal Memory, Write-Back Cacheable, and Inner Shareable:
DESCRIPTOR: level 2 index: 000 --> 00600000c0000405 (00000000c0000000)
4. Use Proper Memory Barriers
When implementing spinlocks, it is crucial to use the correct memory barriers to ensure proper memory ordering. The ARM architecture provides several memory barrier instructions, including dmb (Data Memory Barrier), dsb (Data Synchronization Barrier), and isb (Instruction Synchronization Barrier). These barriers ensure that memory operations are completed in the correct order and that all cores see a consistent view of memory.
In the spinlock implementation, the stlr (Store-Release) instruction is used to release the lock. This instruction ensures that all previous memory operations are completed before the lock is released. However, additional barriers may be necessary depending on the specific implementation and the memory model being used.
5. Debugging and Verification
After making the necessary changes, it is important to verify that the spinlock is functioning correctly. This can be done by:
- Using a debugger to inspect the lock variable and ensure that it is being updated correctly by all cores.
- Adding logging or print statements to track the acquisition and release of the lock by each core.
- Running stress tests to ensure that the spinlock works correctly under heavy contention.
If the issue persists, further debugging may be required to identify any additional misconfigurations or hardware issues.
Conclusion
The spinlock failure in the ARM Cortex-A53 multi-core system is caused by a misconfiguration of the shareability attributes in the page table and the cache coherency settings. By correctly configuring the memory attributes, enabling hardware coherency, and using proper memory barriers, the spinlock can be made to function correctly across all cores. This ensures that all cores see a consistent view of the lock variable, allowing for proper synchronization in a multi-core environment.
