SMMU-v3 Memory Attribute Configuration and Coherency Challenges
The System Memory Management Unit version 3 (SMMU-v3) is a critical component in modern ARM-based systems, enabling virtualization and memory protection for I/O devices. However, configuring memory attributes and ensuring coherency between the CPU and SMMU can be challenging, especially when dealing with non-coherent memory accesses. This post delves into the intricacies of SMMU-v3 memory attribute configuration, focusing on the coherency challenges that arise when the SMMU does not support coherent access (COHACC = 0). We will explore the root causes of these issues and provide detailed troubleshooting steps and solutions.
SMMU_CR1 and STE Memory Attribute Configuration Ambiguities
The SMMU_CR1 register and Stage 2 Translation Entries (STE) are central to configuring memory attributes for table and queue accesses in SMMU-v3. The SMMU_CR1 register includes fields such as TABLE_SH, TABLE_OC, and TABLE_IC, which define the shareability and cacheability attributes for table accesses. Similarly, QUEUE_SH, QUEUE_OC, and QUEUE_IC configure these attributes for queue accesses. The STE contains fields like S2IR0, S2OR0, and S2SH0, which define the cacheability and shareability attributes for Stage 2 translation table accesses.
A key ambiguity arises regarding whether these configurations refer to the CPU or the SMMU. For instance, when the CPU allocates memory for SMMU tables and queues, should the cacheability and shareability attributes match those of the CPU or the SMMU? This ambiguity is particularly problematic when the SMMU does not support coherent access (COHACC = 0). In such cases, the SMMU cannot rely on hardware coherency mechanisms and must use non-cacheable memory. However, the Linux SMMU-v3 driver often configures SMMU_CR1 with cacheable attributes (e.g., CR1_CACHE_WB) regardless of the COHACC setting, leading to potential coherency issues.
The STE configuration further complicates matters. When the SMMU uses Stage 2 translation tables created by the CPU, the cacheability and shareability attributes in the STE must align with those of the CPU. However, if the SMMU does not support coherent access, configuring these attributes incorrectly can lead to coherency problems, resulting in errors such as C_BAD_STREAMID or C_BAD_STE.
Cache Coherency and Memory Synchronization Issues in Non-Coherent SMMU Systems
When the SMMU does not support coherent access (COHACC = 0), software must ensure coherency between the CPU and SMMU through explicit cache operations. This requirement introduces several challenges. First, the CPU must allocate non-cacheable memory for SMMU tables and queues to avoid coherency issues. However, even with non-cacheable memory, the SMMU may still require explicit cache operations to synchronize memory accesses.
For example, when the CPU updates a translation table or writes to a queue, the SMMU may not immediately see these changes due to caching effects. In such cases, the CPU must perform cache maintenance operations, such as cleaning or invalidating the cache, to ensure that the SMMU accesses the most up-to-date data. Failure to perform these operations can result in errors such as C_BAD_STREAMID or C_BAD_STE, as the SMMU may access stale or incorrect data.
The need for cache operations on non-cacheable memory may seem counterintuitive. However, this requirement arises because the SMMU relies on the CPU’s cache operations as a synchronization mechanism. When the CPU performs a cache operation, it signals to the SMMU that the memory is now consistent, allowing the SMMU to proceed with its accesses. This behavior is particularly evident in systems where the SMMU does not support coherent access, as the SMMU cannot rely on hardware coherency mechanisms to ensure memory consistency.
Implementing Cache Maintenance and Memory Attribute Configuration for SMMU-v3
To address the coherency and memory attribute configuration challenges in SMMU-v3, software must implement a combination of cache maintenance operations and careful memory attribute configuration. The following steps outline a comprehensive approach to resolving these issues:
-
Determine SMMU Coherency Support: Before configuring memory attributes, software must check the SMMU_IDR0.COHACC bit to determine whether the SMMU supports coherent access. If COHACC = 0, the SMMU does not support coherent access, and software must use non-cacheable memory for SMMU tables and queues.
-
Configure SMMU_CR1 and STE Memory Attributes: When COHACC = 0, software must configure the SMMU_CR1 and STE memory attributes to match the non-cacheable memory allocated by the CPU. This includes setting the TABLE_OC, TABLE_IC, QUEUE_OC, QUEUE_IC, S2IR0, and S2OR0 fields to non-cacheable values. Additionally, the shareability attributes (TABLE_SH, QUEUE_SH, S2SH0) should align with the CPU’s shareability configuration.
-
Perform Cache Maintenance Operations: To ensure coherency between the CPU and SMMU, software must perform cache maintenance operations after updating SMMU tables or queues. These operations include cleaning the cache to the Point of Coherency (PoC) and issuing a Data Synchronization Barrier (DSB) to ensure that the cache operations are complete before the SMMU accesses the memory.
-
Handle Errors and Event Queues: In cases where errors such as C_BAD_STREAMID or C_BAD_STE occur, software must check the event queue for corresponding events. If the event queue does not receive new events after an error, it may indicate a coherency issue. Performing cache maintenance operations can resolve these issues and allow the event queue to receive new events.
-
Validate Configuration on Real Hardware and FVP: Finally, software must validate the memory attribute configuration and cache maintenance operations on both real hardware and the Fixed Virtual Platform (FVP). This validation ensures that the configuration works correctly in different environments and helps identify any platform-specific issues.
By following these steps, software can effectively address the coherency and memory attribute configuration challenges in SMMU-v3, ensuring reliable operation in both coherent and non-coherent systems.
| Step | Description |
|---|---|
| Determine SMMU Coherency Support | Check SMMU_IDR0.COHACC to determine if the SMMU supports coherent access. |
| Configure SMMU_CR1 and STE | Set memory attributes in SMMU_CR1 and STE to match non-cacheable memory. |
| Perform Cache Maintenance | Clean cache to PoC and issue DSB after updating SMMU tables or queues. |
| Handle Errors and Event Queues | Check event queue for errors and perform cache maintenance if necessary. |
| Validate Configuration | Test configuration on real hardware and FVP to ensure correctness. |
In conclusion, configuring memory attributes and ensuring coherency in SMMU-v3 requires a deep understanding of the hardware and careful software implementation. By addressing the ambiguities in memory attribute configuration and implementing robust cache maintenance operations, software can achieve reliable operation in both coherent and non-coherent SMMU systems.
