Disabling Hardware Prefetchers on ARM Cortex-A57 for Cache Benchmarking

ARM Cortex-A57 Hardware Prefetcher Behavior and Cache Benchmarking Challenges

The ARM Cortex-A57 is a high-performance processor core designed for applications requiring significant computational power, such as mobile devices, networking equipment, and embedded systems. One of its key features is the inclusion of hardware prefetchers, which are designed to improve performance by predicting and preloading data into the cache before it is explicitly requested by the software. While this feature is beneficial for most applications, it can introduce complications when attempting to benchmark cache performance, as the prefetchers can distort measurements of cache misses, bandwidth, and other critical metrics.

Hardware prefetchers in the Cortex-A57 operate by observing memory access patterns and speculatively fetching data that is likely to be needed in the near future. This includes both data prefetching (loading data into the cache) and instruction prefetching (loading instructions into the instruction cache). For cache benchmarking, this behavior is problematic because it can artificially reduce the number of cache misses and alter the observed memory bandwidth, making it difficult to obtain accurate measurements.

To address this issue, it is necessary to disable the hardware prefetchers. However, the Cortex-A57 does not provide a straightforward mechanism for disabling prefetchers at the user level (EL0). Instead, this requires modifications at the operating system level (EL1) or even the bootloader level, depending on the specific implementation. This complexity arises because the controls for prefetching and speculation are tied to privileged registers and memory management unit (MMU) configurations, which are not accessible from user space.

Privileged Register Access and MMU Configuration for Prefetch Control

The ARM Cortex-A57 provides several mechanisms for controlling prefetching and speculation, but these mechanisms are generally accessible only at higher exception levels (EL1 or above). This includes the use of MMU translation tables to mark memory regions as either "Normal" or "Device" and to set permissions for read, write, and execute operations. Marking a region as "Device" prevents data speculation, while marking a region as non-executable prevents instruction prefetching. These controls are implemented through the MMU translation tables, which are managed by the operating system or bootloader.

In addition to MMU configurations, the Cortex-A57 includes implementation-defined (IMPDEF) registers that can be used to fine-tune prefetching behavior. These registers are described in the Technical Reference Manual (TRM) for the Cortex-A57 and typically require EL1 or higher privilege to access. For example, the CPUACTLR_EL1 register (CPU Auxiliary Control Register) includes bits that can be used to disable specific prefetchers, such as the L1 data prefetcher or the L2 prefetcher. However, the exact bits and their functions may vary depending on the specific implementation of the Cortex-A57.

To disable hardware prefetchers system-wide, it is necessary to modify these privileged registers during the boot process. This can be done by adding custom code to the bootloader or by modifying the operating system kernel to include the necessary register writes. For example, to disable the L1 data prefetcher, the following steps might be required:

1. Access the CPUACTLR_EL1 register at EL1 or higher.
2. Set the appropriate bit (e.g., bit 2 for the L1 data prefetcher) to disable the prefetcher.
3. Ensure that the changes are propagated to all cores in a multi-core system.

These steps require a deep understanding of the Cortex-A57 architecture and the specific implementation of the system in question. Additionally, care must be taken to ensure that disabling prefetchers does not inadvertently degrade system performance or stability.

Implementing Prefetch Disabling in Bootloader and Kernel Code

Disabling hardware prefetchers on the ARM Cortex-A57 requires modifications to the bootloader or operating system kernel, as the necessary controls are not accessible from user space. Below is a detailed guide on how to implement these changes, including example code snippets and considerations for system-wide implementation.

Bootloader Modifications

The bootloader is responsible for initializing the hardware and loading the operating system. It operates at a high privilege level (typically EL3 or EL2), making it an ideal place to disable hardware prefetchers before the operating system takes control. The following steps outline how to modify the bootloader to disable prefetchers:

1. Identify the Prefetch Control Registers: Consult the Cortex-A57 TRM to identify the registers and bits that control hardware prefetchers. For example, the CPUACTLR_EL1 register may include bits for disabling the L1 and L2 prefetchers.

2. Write to the Registers: Add code to the bootloader to write to the identified registers. This typically involves using assembly instructions to access the system registers. For example, to disable the L1 data prefetcher, the following assembly code might be used:

   MRS X0, CPUACTLR_EL1    // Read the current value of CPUACTLR_EL1
   ORR X0, X0, #(1 << 2)   // Set bit 2 to disable the L1 data prefetcher
   MSR CPUACTLR_EL1, X0    // Write the modified value back to CPUACTLR_EL1
   

3. Ensure System-Wide Application: In a multi-core system, the changes must be applied to all cores. This can be done by executing the register writes on each core during the boot process. The bootloader should include logic to detect and iterate over all available cores.

4. Verify the Changes: After modifying the bootloader, verify that the prefetchers have been disabled by running cache benchmarking tests and comparing the results to those obtained with prefetchers enabled.

Kernel Modifications

If modifying the bootloader is not feasible, the operating system kernel can be modified to disable hardware prefetchers. This approach is more complex, as it requires adding custom code to the kernel and ensuring that the changes are applied consistently across all cores and contexts. The following steps outline how to implement this in the kernel:

1. Add Kernel Module or Patch: Create a kernel module or patch that includes the necessary register writes. This module should be loaded during kernel initialization.

2. Access Privileged Registers: Use kernel-level assembly or inline assembly to access the prefetch control registers. For example, the following C code with inline assembly can be used to disable the L1 data prefetcher:

   static void disable_l1_prefetcher(void) {
       uint64_t val;
       __asm__ volatile("MRS %0, CPUACTLR_EL1" : "=r"(val));
       val |= (1 << 2);  // Set bit 2 to disable the L1 data prefetcher
       __asm__ volatile("MSR CPUACTLR_EL1, %0" : : "r"(val));
   }
   

3. Apply to All Cores: Ensure that the register writes are executed on all cores. This can be done by using kernel APIs to iterate over all CPUs and execute the code on each one.

4. Handle Context Switching: Ensure that the changes persist across context switches and power management events. This may require additional code to reapply the register writes after the system resumes from a low-power state.

5. Test and Validate: Thoroughly test the modified kernel to ensure that the prefetchers are disabled and that the system remains stable. Run cache benchmarking tests to verify the impact on performance metrics.

Considerations and Best Practices

Disabling hardware prefetchers can have significant implications for system performance and behavior. Before implementing these changes, consider the following:

• Performance Impact: Disabling prefetchers may reduce overall system performance, as the processor will no longer benefit from speculative data loading. This is particularly important for applications with irregular memory access patterns.

• System Stability: Ensure that the changes do not introduce instability or unexpected behavior. Thoroughly test the modified bootloader or kernel in a controlled environment before deploying it in production.

• Reversibility: Provide a mechanism to re-enable prefetchers if needed. This can be done by adding a runtime option or configuration parameter to the bootloader or kernel.

• Documentation: Document the changes and their impact on system behavior. This is especially important in collaborative or multi-developer environments.

By following these steps and considerations, it is possible to disable hardware prefetchers on the ARM Cortex-A57 for accurate cache benchmarking. However, this process requires a deep understanding of the Cortex-A57 architecture and careful implementation to avoid unintended consequences.