ARM Cortex-A53 Cache Coherency Issues Between EL1 and EL3 in Shared Memory Regions

ARM Cortex-A53 Cache Coherency Issues Between EL1 and EL3 in Shared Memory Regions

Cache Coherency Challenges in Multi-Exception Level Systems

In systems utilizing ARM Cortex-A53 processors, cache coherency between different Exception Levels (ELs) can present significant challenges, particularly when shared memory regions are involved. The Cortex-A53, being a part of the ARMv8-A architecture, supports multiple Exception Levels, each with its own Memory Management Unit (MMU) and cache configuration. When Linux runs at EL1 and a custom OS operates at EL3, ensuring cache coherency between these two levels becomes critical, especially when they share memory regions for data exchange.

The primary issue arises when the shared memory region is configured with cacheable attributes, leading to inconsistent data visibility between the cores running at EL1 and EL3. This inconsistency manifests as data corruption or stale data reads, which can severely impact system reliability. The problem is exacerbated when the shared memory region is accessed with mismatched memory attributes or when the Secure and Non-secure states are not properly configured.

Mismatched Memory Attributes and Secure/Non-secure State Configuration

One of the core reasons for cache coherency issues in ARM Cortex-A53 systems is the mismatched memory attributes between EL1 and EL3. Each Exception Level has its own translation tables and memory attribute configurations, which must be consistent for shared memory regions. If the memory attributes, such as cacheability and shareability, are not aligned between EL1 and EL3, the caches may not maintain coherency, leading to data inconsistencies.

Another critical factor is the Secure (S) and Non-secure (NS) state configuration. The ARMv8-A architecture defines separate physical address spaces for Secure and Non-secure states. When a memory region is accessed from different Exception Levels, the NS bit in the translation table descriptors determines whether the access is to the Secure or Non-secure physical address space. If the NS bit is not consistently configured across EL1 and EL3, the caches may treat the same physical address as two distinct locations, leading to coherency issues.

For example, if EL1 accesses a memory region with NS=1 (Non-secure) and EL3 accesses the same region with NS=0 (Secure), the caches will allocate these accesses to different cache lines, even though they refer to the same physical memory. This mismatch can result in data corruption or stale data reads, as the caches do not maintain coherency between Secure and Non-secure accesses.

Implementing Consistent Memory Attributes and Cache Management

To resolve cache coherency issues between EL1 and EL3 in ARM Cortex-A53 systems, it is essential to implement consistent memory attributes and proper cache management techniques. The following steps outline the necessary actions to ensure cache coherency in shared memory regions:

1. Consistent Memory Attribute Configuration: Ensure that the memory attributes, such as cacheability and shareability, are consistent between EL1 and EL3 for shared memory regions. This includes configuring the MAIR (Memory Attribute Indirection Register) with the same attributes for both Exception Levels. For example, if the shared memory region is configured as Normal, Inner Write-Back Write-Allocate (WBWA), and Outer Write-Back Write-Allocate (WBWA) in EL1, the same attributes must be used in EL3.

2. Proper NS Bit Configuration: Ensure that the NS bit in the translation table descriptors is consistently configured across EL1 and EL3. If the shared memory region is intended to be accessed in Non-secure state, both EL1 and EL3 must set the NS bit to 1 in their respective translation table entries. This ensures that both Exception Levels access the same physical address space, allowing the caches to maintain coherency.

3. Cache Maintenance Operations: Implement cache maintenance operations to ensure that data written to the shared memory region is visible to all cores. This includes using Data Synchronization Barriers (DSB) and Data Memory Barriers (DMB) to ensure that memory operations are completed before proceeding. Additionally, cache invalidation and cleaning operations may be required to ensure that stale data is not read from the caches.

4. Translation Table Configuration: Ensure that the translation tables for both EL1 and EL3 are correctly configured to map the shared memory region with the same attributes. This includes setting the appropriate MAIR index and ensuring that the page table entries are consistent between Exception Levels. For example, the descriptor used in EL3 should match the attributes used in EL1, such as the NS bit, AP (Access Permissions), and MAIR index.

5. Testing and Validation: Implement thorough testing and validation to ensure that the shared memory region is functioning correctly. This includes running test applications that write and read data from the shared memory region and verifying that the data is consistent between EL1 and EL3. Any discrepancies should be investigated and resolved by revisiting the memory attribute and cache management configurations.

Example Configuration for Shared Memory Region

To illustrate the proper configuration for a shared memory region between EL1 and EL3, consider the following example:

• MAIR Configuration: The MAIR register is configured with the following attributes:

  • Index 0: Device, nGnRnE
  • Index 1: Device, nGnRE
  • Index 2: Device, GRE
  • Index 3: Normal, Outer Non-cacheable, Inner Non-cacheable
  • Index 4: Normal, Outer Write-Back Non-transient, Inner Write-Back Non-transient
  • Index 5: Normal, Outer Write-Through Non-transient, Inner Write-Through Non-transient

  The MAIR register is set as follows:

  ldr x1, =0x0000BBFF440C0400
  msr MAIR_EL3, x1
  

• Translation Table Descriptor: The translation table descriptor for the shared memory region is configured with the following attributes:

  • NS bit: 1 (Non-secure)
  • AP: 01 (Read/write)
  • MAIR index: 4 (Normal, Outer Write-Back Non-transient, Inner Write-Back Non-transient)

  The descriptor is set as follows:

  ldr x2, =0x0060000000000E71
  

• Cache Maintenance Operations: After writing data to the shared memory region, a Data Synchronization Barrier (DSB) is executed to ensure that the write operation is completed:

  dsb sy
  

• Cache Invalidation: Before reading data from the shared memory region, the cache is invalidated to ensure that stale data is not read:

  dc ivac, x3
  

By following these steps and ensuring consistent memory attribute and cache management configurations, cache coherency issues between EL1 and EL3 in ARM Cortex-A53 systems can be effectively resolved. Proper testing and validation are essential to ensure that the shared memory region functions as intended, providing reliable data exchange between different Exception Levels.