Excessive Secondary Core Boot Delay on ARMv8 Zynq MPSoC Platforms

ARM Cortex-A53 Secondary Core Boot Latency of 466ms During PSCI SMC Call

The issue at hand involves an unusually high latency of approximately 466ms when booting secondary Cortex-A53 cores on an ARMv8-based Zynq MPSoC platform. The primary core boots the kernel in just 74ms, but the secondary cores take significantly longer to reach their entry point after being triggered via an SMC (Secure Monitor Call) with the PSCI (Power State Coordination Interface) identifier 0xC4000003. This delay is causing the overall SMP (Symmetric Multiprocessing) configuration of the operating system to take 574ms, which is unacceptable for time-sensitive applications. The delay is suspected to be related to the ARM Trusted Firmware (ATF) implementation or the PSCI handling mechanism.

The primary concern is identifying the root cause of this latency and implementing optimizations to reduce the secondary core boot time to a more reasonable range, ideally in the order of microseconds, as seen in other operating systems. The problem is particularly critical in real-time or performance-sensitive systems where boot time and core synchronization are paramount.

ARM Trusted Firmware Initialization Overhead and PSCI Handling Delays

The excessive boot latency of secondary cores can be attributed to several potential causes, primarily centered around the ARM Trusted Firmware (ATF) and its interaction with the PSCI implementation. Below are the most likely culprits:

ARM Trusted Firmware Initialization Overhead

The ARM Trusted Firmware is responsible for initializing the secure world environment, including setting up the PSCI framework. During the boot process, the ATF performs a series of secure world initializations, such as setting up the exception vectors, configuring the secure monitor, and initializing the PSCI handlers. If the ATF is not optimized for the specific platform, these initializations can introduce significant delays. For instance, the ATF might be performing redundant checks or initializations that are not strictly necessary for the Zynq MPSoC platform.

PSCI SMC Handling Latency

The PSCI framework is used to manage power states and core boot sequences in ARM-based systems. When the primary core issues an SMC call with the PSCI identifier 0xC4000003 to boot a secondary core, the request is handled by the ATF in the secure world. The latency in this process can be influenced by several factors:

• SMC Call Routing Overhead: The SMC call must traverse the secure monitor, which involves context switching between the normal and secure worlds. This transition can be slow if the secure monitor implementation is not optimized.
• PSCI Handler Complexity: The PSCI handler in the ATF might be performing extensive checks or operations before triggering the secondary core boot. For example, it might validate the core state, check power domain configurations, or perform unnecessary synchronization steps.
• Platform-Specific Delays: The Zynq MPSoC platform might have specific hardware constraints or quirks that are not fully accounted for in the ATF implementation. For instance, the platform might require additional delays for power domain stabilization or clock synchronization.

Secondary Core Initialization Sequence

Once the PSCI handler triggers the secondary core boot, the core must go through its own initialization sequence before reaching the entry point. This sequence includes:

• Reset Vector Configuration: The secondary core must fetch its reset vector from the platform-specific memory location. If this memory access is slow or involves additional checks, it can introduce delays.
• Cache and MMU Initialization: The secondary core might need to initialize its caches and MMU (Memory Management Unit) before executing the boot code. If these initializations are not optimized, they can contribute to the overall latency.
• Synchronization with Primary Core: The secondary core might need to wait for synchronization signals from the primary core, such as spinlocks or semaphores. If these synchronization mechanisms are not efficient, they can introduce significant delays.

Optimizing ARM Trusted Firmware and PSCI Handling for Faster Secondary Core Boot

To address the excessive secondary core boot latency, a systematic approach is required to identify and eliminate the bottlenecks in the ARM Trusted Firmware and PSCI handling mechanisms. Below are the detailed troubleshooting steps and potential solutions:

Profiling and Identifying Bottlenecks in ARM Trusted Firmware

The first step is to profile the ARM Trusted Firmware to identify the specific functions or operations that are contributing to the 466ms delay. This can be done using platform-specific profiling tools or by instrumenting the ATF code with timestamps. Key areas to focus on include:

• Secure Monitor Transition Overhead: Measure the time taken for the SMC call to traverse the secure monitor. If this transition is slow, consider optimizing the secure monitor implementation or reducing the number of context switches.
• PSCI Handler Execution Time: Profile the PSCI handler to identify any redundant checks or operations. For example, if the handler is performing extensive power domain validations, consider simplifying these checks or moving them to a less critical path.
• Platform-Specific Initializations: Review the platform-specific initializations performed by the ATF. If any of these initializations are not strictly necessary for the Zynq MPSoC platform, consider removing or optimizing them.

Optimizing PSCI SMC Handling

Once the bottlenecks have been identified, the next step is to optimize the PSCI SMC handling mechanism. This can be achieved through the following steps:

• Reducing SMC Call Overhead: Minimize the overhead of the SMC call by optimizing the secure monitor implementation. For example, reduce the number of context switches or streamline the exception handling mechanism.
• Simplifying PSCI Handler: Simplify the PSCI handler by removing redundant checks or operations. For instance, if the handler is validating the core state multiple times, consider consolidating these checks into a single validation step.
• Platform-Specific Optimizations: Implement platform-specific optimizations for the Zynq MPSoC platform. For example, if the platform requires additional delays for power domain stabilization, consider reducing these delays or implementing a more efficient power management mechanism.

Streamlining Secondary Core Initialization

Finally, optimize the secondary core initialization sequence to reduce the time taken to reach the entry point. This can be achieved through the following steps:

• Optimizing Reset Vector Fetch: Ensure that the reset vector is fetched from a fast memory location. If the reset vector is stored in slow memory, consider moving it to a faster memory region or caching it.
• Efficient Cache and MMU Initialization: Optimize the cache and MMU initialization sequence for the secondary core. For example, consider lazy initialization of the MMU or pre-initializing the cache before the secondary core starts executing.
• Efficient Synchronization Mechanisms: Implement efficient synchronization mechanisms between the primary and secondary cores. For example, use lightweight spinlocks or atomic operations instead of heavy-weight semaphores.

Implementing Data Synchronization Barriers and Cache Management

To ensure that the optimizations are effective, it is crucial to implement proper data synchronization barriers and cache management techniques. This includes:

• Data Synchronization Barriers (DSB): Use DSB instructions to ensure that all memory accesses are completed before proceeding to the next step. This is particularly important during the secondary core initialization sequence to avoid race conditions or stale data.
• Cache Invalidation and Flushing: Invalidate and flush the caches as needed to ensure that the secondary core has a consistent view of memory. For example, invalidate the instruction cache before the secondary core starts executing to ensure that it fetches the latest instructions.

Testing and Validation

After implementing the optimizations, thoroughly test and validate the system to ensure that the secondary core boot latency has been reduced to an acceptable level. This includes:

• Benchmarking: Measure the time taken to boot the secondary cores and compare it with the previous results. Ensure that the latency has been significantly reduced.
• Functional Testing: Verify that the system functions correctly with the optimized boot sequence. This includes testing the SMP configuration, synchronization mechanisms, and overall system stability.
• Regression Testing: Ensure that the optimizations do not introduce any regressions or side effects. Test the system under various workloads and conditions to ensure that it remains stable and performant.

By following these steps, the excessive secondary core boot latency on the ARMv8 Zynq MPSoC platform can be effectively addressed, resulting in a more efficient and responsive system.