ARM Cortex-M7 Cache Coherency Challenges with Peripheral DMA Transfers
The ARM Cortex-M7 processor, with its advanced features like data cache and high-performance memory system, is widely used in embedded systems requiring efficient data processing. However, these features can introduce complexities when interfacing with peripheral DMA engines, such as the Ethernet GMAC (Gigabit Media Access Controller). The core issue arises from the interaction between the Cortex-M7’s data cache and the DMA engine, leading to coherency problems that manifest as HRESP errors or incorrect data transfers. This post delves into the root causes of these issues, explores potential solutions, and provides detailed troubleshooting steps to ensure reliable operation of DMA-driven peripherals like the GMAC.
Cache Coherency Breakdown During DMA Transfers
The Cortex-M7’s data cache is designed to accelerate memory access by storing frequently used data closer to the CPU. However, this introduces a coherency problem when peripherals like the GMAC use DMA to transfer data directly to or from memory. The DMA engine operates independently of the CPU and accesses memory directly, bypassing the cache. If the CPU has modified data in the cache but not yet written it back to memory (a "dirty" cache line), the DMA engine will read stale data from memory. Conversely, if the DMA engine writes data to memory, the CPU may read stale data from the cache instead of fetching the updated data from memory.
In the case of the ATSAME70 Cortex-M7 processor, the Ethernet GMAC driver experienced HRESP errors when the data cache was enabled. These errors indicate that the DMA engine encountered issues while accessing memory, often due to coherency problems. Disabling the data cache resolved the issue, but this is not a practical solution as it sacrifices performance. The challenge lies in maintaining cache coherency while leveraging the performance benefits of the cache.
Misaligned Buffers and Inadequate Cache Management
One of the primary causes of cache coherency issues in DMA transfers is improper buffer alignment and insufficient cache management. The Cortex-M7’s cache operates on 32-byte cache lines, meaning that memory accesses are grouped into 32-byte blocks. If a DMA buffer is not aligned to a 32-byte boundary, the cache management operations (such as flushing or invalidating) may not cover the entire buffer, leading to partial updates or stale data.
In the ATSAME70 GMAC driver, the buffers used for DMA transfers were not explicitly aligned to 32-byte boundaries, even though the datasheet specified this requirement. While the compiler happened to align the buffers correctly in some cases, this was not guaranteed. Additionally, the driver did not consistently flush or invalidate the cache before and after DMA transfers, leading to coherency issues. For example, when sending a packet, the driver wrote data to SRAM and then initiated the DMA transfer without ensuring that the data in the cache was flushed to memory. Similarly, when receiving data, the driver read from SRAM without invalidating the cache, potentially reading stale data.
Another contributing factor was the lack of detailed documentation and example code from the silicon vendor. The ATSAME70 datasheet contained ambiguities and contradictions regarding buffer alignment and cache management, making it difficult for developers to implement correct solutions. Furthermore, the provided CMSIS pack had incorrect register addresses, exacerbating the problem.
Implementing Cache Flushing, Invalidation, and Buffer Alignment
To resolve cache coherency issues in DMA transfers, a combination of cache management techniques and proper buffer alignment is required. Below are detailed steps to address these challenges:
Cache Flushing Before DMA Transfers
Before initiating a DMA transfer, it is essential to ensure that any modified data in the cache is written back to memory. This is achieved by flushing the cache for the relevant memory regions. On the Cortex-M7, the SCB_CleanDCache_by_Addr function can be used to flush specific cache lines. The function takes the starting address and size of the memory region as parameters and ensures that all dirty cache lines within that region are written back to memory.
For example, when sending a packet via the GMAC, the driver should flush the cache for the transmit buffer before starting the DMA transfer. This ensures that the DMA engine reads the most up-to-date data from memory.
SCB_CleanDCache_by_Addr((uint32_t*)txBuffer, txBufferSize);
Cache Invalidation After DMA Transfers
After a DMA transfer completes, the cache must be invalidated to ensure that the CPU reads the updated data from memory rather than stale data from the cache. The SCB_InvalidateDCache_by_Addr function is used for this purpose. It takes the starting address and size of the memory region and invalidates the corresponding cache lines.
For example, when receiving a packet via the GMAC, the driver should invalidate the cache for the receive buffer after the DMA transfer completes.
SCB_InvalidateDCache_by_Addr((uint32_t*)rxBuffer, rxBufferSize);
Buffer Alignment to Cache Line Size
To ensure that cache management operations cover the entire DMA buffer, the buffer must be aligned to the cache line size (32 bytes on the Cortex-M7). This can be achieved using compiler-specific attributes or alignment directives. For example, in GCC, the aligned attribute can be used to align a buffer to a 32-byte boundary.
uint8_t txBuffer[BUFFER_SIZE] __attribute__((aligned(32)));
uint8_t rxBuffer[BUFFER_SIZE] __attribute__((aligned(32)));
Polling Interrupt Registers Instead of Memory
Instead of polling memory locations to check for DMA completion, it is more efficient and reliable to poll the peripheral’s interrupt registers. This approach avoids potential cache coherency issues and reduces CPU overhead. The GMAC’s interrupt status register can be polled to determine when a DMA transfer has completed.
while (!(GMAC->GMAC_ISR & GMAC_ISR_RCOMP)) {
// Wait for DMA completion
}
Using Non-Cached Memory Sections (Optional)
In some cases, it may be beneficial to allocate DMA buffers in a non-cached memory section to avoid cache management overhead. This can be done using linker scripts to define a non-cached memory region and placing the DMA buffers in that region. However, this approach should be used judiciously, as it may reduce overall system performance by bypassing the cache for frequently accessed data.
MEMORY {
SRAM_NC (rwx) : ORIGIN = 0x20000000, LENGTH = 64K
}
SECTIONS {
.dma_buffers (NOLOAD) : {
*(.dma_buffers)
} > SRAM_NC
}
Debugging HRESP Errors
HRESP errors indicate that the DMA engine encountered a problem while accessing memory. These errors can be caused by misaligned buffers, insufficient cache management, or incorrect memory configurations. To debug HRESP errors, follow these steps:
- Verify that all DMA buffers are aligned to 32-byte boundaries.
- Ensure that the cache is flushed before DMA writes and invalidated after DMA reads.
- Check the memory map and access permissions to ensure that the DMA engine has access to the required memory regions.
- Use a debugger to inspect the DMA engine’s status registers and identify the specific cause of the HRESP error.
Vendor Documentation and Example Code
When working with complex peripherals like the GMAC, it is crucial to have accurate and detailed documentation. If the vendor’s datasheet or example code is inadequate, consider the following steps:
- Consult the ARM Cortex-M7 Technical Reference Manual for core-specific details.
- Review errata documents for known issues and workarounds.
- Engage with the vendor’s support team to clarify ambiguities and report bugs.
- Leverage community forums and open-source projects for additional insights and examples.
Performance Considerations
While cache management operations introduce some overhead, they are generally more efficient than disabling the cache entirely. Properly aligned buffers and targeted cache operations minimize the performance impact. Additionally, polling interrupt registers instead of memory locations reduces CPU overhead and improves system responsiveness.
Summary of Key Points
| Issue | Solution |
|---|---|
| Cache coherency during DMA | Flush cache before DMA writes, invalidate cache after DMA reads |
| Misaligned buffers | Align DMA buffers to 32-byte boundaries |
| HRESP errors | Debug using aligned buffers, cache management, and DMA status registers |
| Inadequate vendor documentation | Consult ARM manuals, errata, and community resources |
| Performance optimization | Use targeted cache operations and poll interrupt registers |
By following these steps, developers can effectively address cache coherency issues in Cortex-M7 systems with peripheral DMA engines like the GMAC. Proper cache management, buffer alignment, and debugging techniques ensure reliable and efficient operation, even in complex embedded systems.
