ARM Cortex-A53 Cores Crashing During 64-bit to 32-bit Mode Transition
The issue at hand involves ARM Cortex-A53 cores 1, 2, and 3 crashing when transitioning from a power-on reset state to an execution state during a boot process that switches from 64-bit mode to 32-bit mode. The system initially runs a Built-In Test (BIT) in the Secondary Stage Boot Loader (SSBL) in 64-bit mode, after which cores 1 to 3 are placed into a power-on reset state. Core 0 then hands off control to a 32-bit operating system, during which cores 1 to 3 are brought back into an execution state. The crash occurs immediately upon this transition. Interestingly, introducing a delay of 200 to 500 milliseconds before placing cores 1 to 3 into the power-on reset state resolves the issue. This suggests a timing or synchronization problem in the hardware-software interaction during the mode transition.
The Cortex-A53 is a 64-bit ARMv8-A processor that supports both AArch64 (64-bit) and AArch32 (32-bit) execution states. The transition between these states is a critical operation that requires careful handling of the processor’s architectural state, including registers, caches, and memory management units (MMUs). The crash indicates a potential misconfiguration or race condition during this transition, particularly when multiple cores are involved.
The problem is exacerbated by the fact that the system is switching execution modes while also managing the power states of multiple cores. This dual transition—both in execution mode and power state—creates a complex scenario where timing, synchronization, and state management must be meticulously handled to avoid instability.
Power State Transition Timing and Cache Coherency Issues
One of the primary suspects in this scenario is the timing of the power state transition and its interaction with the cache coherency mechanisms. When cores 1 to 3 are placed into a power-on reset state, their caches and architectural states are invalidated. However, if these cores are brought back into an execution state too quickly, there may not be sufficient time for the system to ensure cache coherency across all cores. This can lead to inconsistent memory views, causing crashes when the cores attempt to execute instructions or access memory.
The Cortex-A53 employs a distributed cache coherency mechanism, typically managed by the ARM CoreLink CCI (Cache Coherent Interconnect). This interconnect ensures that all cores have a consistent view of memory. However, during power state transitions, the coherency protocol must be carefully managed to avoid race conditions. If a core is brought out of reset before the coherency protocol has fully stabilized, it may access stale or invalid data, leading to undefined behavior.
Another potential cause is the handling of the Translation Table Base Register (TTBR) and the Memory Management Unit (MMU) during the transition from 64-bit to 32-bit mode. The TTBR and MMU configurations are different between AArch64 and AArch32 states, and an improper transition can result in incorrect memory mappings. If cores 1 to 3 are brought online before the MMU is properly configured for the 32-bit mode, they may attempt to access memory using incorrect translations, leading to crashes.
Additionally, the delay introduced before placing cores 1 to 3 into the power-on reset state may be allowing sufficient time for the system to stabilize the coherency protocol and complete the MMU reconfiguration. This suggests that the issue is timing-sensitive and related to the synchronization of hardware states during the transition.
Implementing Proper Cache Invalidation and MMU Reconfiguration
To address the issue, a systematic approach to cache invalidation and MMU reconfiguration must be implemented. The following steps outline the necessary actions to ensure a stable transition from 64-bit to 32-bit mode while managing the power states of multiple cores.
Cache Invalidation and Coherency Management
Before placing cores 1 to 3 into the power-on reset state, it is essential to ensure that all caches are properly invalidated and that the coherency protocol is in a stable state. This can be achieved by performing a full cache clean and invalidate operation on all cores. The Data Cache Clean and Invalidate by Set/Way (DC CISW) instruction can be used to clean and invalidate the data cache, while the Instruction Cache Invalidate All (ICIALLU) instruction can be used to invalidate the instruction cache.
After the caches are invalidated, a Data Synchronization Barrier (DSB) instruction should be executed to ensure that all cache operations are complete before proceeding. This ensures that no stale data remains in the caches when the cores are brought back online.
MMU Reconfiguration and TTBR Handling
During the transition from 64-bit to 32-bit mode, the MMU must be reconfigured to use the appropriate translation tables for the AArch32 state. This involves updating the TTBR0 and TTBR1 registers to point to the new translation tables. It is crucial to ensure that these updates are performed atomically and that the MMU is disabled during the reconfiguration to prevent any incorrect memory accesses.
Once the TTBRs are updated, the MMU can be re-enabled, and a TLB Invalidate All (TLBIALL) instruction should be executed to ensure that any stale translations are removed from the TLB. This ensures that the cores will use the correct memory mappings when they are brought back into the execution state.
Power State Transition Timing
The introduction of a delay before placing cores 1 to 3 into the power-on reset state suggests that the system requires additional time to stabilize the coherency protocol and complete the MMU reconfiguration. To ensure a stable transition, it is recommended to implement a delay that is sufficient to allow the system to complete these operations. The exact duration of the delay may vary depending on the specific hardware implementation, but a delay of 200 to 500 milliseconds, as observed in the original scenario, appears to be effective.
In addition to the delay, it is advisable to implement a synchronization mechanism to ensure that all cores are in a known state before proceeding with the power state transition. This can be achieved using a barrier or semaphore mechanism that ensures all cores have completed their cache invalidation and MMU reconfiguration before any core is brought back into the execution state.
Verification and Testing
After implementing the above steps, it is essential to verify the stability of the system through rigorous testing. This includes running the BIT test and the 32-bit mode boot process multiple times to ensure that the issue does not reoccur. Additionally, stress testing should be performed to ensure that the system remains stable under various load conditions.
If the issue persists, further analysis may be required to identify any additional factors contributing to the instability. This may include examining the behavior of the CoreLink CCI, analyzing the timing of the power state transitions, and reviewing the implementation of the cache coherency protocol.
Conclusion
The crashing of ARM Cortex-A53 cores 1, 2, and 3 during the transition from 64-bit to 32-bit mode is a complex issue that involves careful management of cache coherency, MMU reconfiguration, and power state transitions. By implementing proper cache invalidation, MMU reconfiguration, and synchronization mechanisms, the issue can be resolved, ensuring a stable and reliable system operation. The introduction of a delay before placing the cores into the power-on reset state provides a practical workaround, but a more robust solution involves addressing the underlying synchronization and state management issues. Through systematic troubleshooting and rigorous testing, the stability of the system can be ensured, allowing for a smooth transition between execution modes and power states.
