ARM Cortex-A Systems with NIC-400 and CCI-500: DDR Access Coherency Challenges
In ARM-based systems, particularly those utilizing the Cortex-A series of processors, the coexistence of multiple interconnect fabrics such as the NIC-400 and CCI-500 can introduce complex data coherency challenges when accessing shared DDR memory. The NIC-400 (Network Interconnect) is commonly used to connect hardware accelerators, peripherals, and other non-coherent masters to the memory subsystem, while the CCI-500 (Cache Coherent Interconnect) facilitates coherent access to DDR memory for CPUs and other coherent agents. When both interconnects are used simultaneously to access DDR, data coherency issues can arise due to the lack of automatic synchronization between the two fabrics. This can lead to scenarios where the CPU reads stale data from its caches while a hardware module writes new data to DDR through the NIC-400, or vice versa.
The core issue stems from the fact that the NIC-400 does not inherently support cache coherency, meaning it cannot snoop or invalidate CPU caches when accessing DDR. On the other hand, the CCI-500 ensures coherency among its connected agents (e.g., CPUs, GPUs) but does not extend this coherency to non-coherent masters connected via the NIC-400. This architectural divide creates a potential for data inconsistency, especially in systems where hardware modules and CPUs frequently share data in DDR.
To understand the problem in depth, consider a typical use case: a hardware accelerator connected to the NIC-400 writes processed data to DDR, while a CPU connected to the CCI-500 reads the same data for further computation. If the CPU’s caches are not invalidated after the hardware accelerator’s write, the CPU may read outdated data from its cache, leading to incorrect results. Similarly, if the CPU writes data to DDR and the hardware accelerator reads it before the write is flushed from the CPU’s cache, the accelerator may also operate on stale data.
Memory Access Path Divergence and Lack of Automatic Cache Synchronization
The root cause of data coherency issues in systems using both NIC-400 and CCI-500 lies in the divergence of memory access paths and the absence of automatic cache synchronization mechanisms between the two interconnects. The NIC-400 operates as a non-coherent interconnect, meaning it does not participate in the cache coherency protocol enforced by the CCI-500. As a result, any memory access initiated by a master connected to the NIC-400 bypasses the CPU caches entirely, leading to potential inconsistencies.
Another contributing factor is the timing of cache maintenance operations. In ARM architectures, cache maintenance operations such as invalidations and clean-and-invalidates are explicit and must be performed by software or hardware to ensure data coherency. If these operations are not executed at the appropriate times, data written by a NIC-400-connected master may not be visible to a CCI-500-connected CPU, or vice versa. This is particularly problematic in real-time systems where deterministic behavior is critical.
Additionally, the system’s memory map and address routing configuration can exacerbate coherency issues. If the DDR memory regions accessed by the NIC-400 and CCI-500 overlap, the lack of coherency enforcement between the two interconnects can lead to unpredictable behavior. For example, if a hardware accelerator writes to a specific DDR address through the NIC-400 and a CPU reads from the same address through the CCI-500, the CPU may receive stale data unless explicit cache maintenance operations are performed.
The ARM architecture provides mechanisms such as memory barriers and cache maintenance instructions to address these issues, but their correct implementation requires a deep understanding of the system’s memory hierarchy and access patterns. Misuse or omission of these mechanisms can result in subtle bugs that are difficult to diagnose and reproduce.
Enforcing Data Coherency Through Explicit Cache Maintenance and Memory Barriers
To resolve data coherency issues in systems using both NIC-400 and CCI-500, a combination of explicit cache maintenance operations and memory barriers must be employed. These techniques ensure that data written by one interconnect is visible to the other and that the order of memory accesses is preserved.
First, when a hardware accelerator or other non-coherent master connected to the NIC-400 writes data to DDR, the software running on the CPU must invalidate the corresponding cache lines before reading the data. This ensures that the CPU fetches the latest data from DDR rather than relying on potentially stale data in its caches. The ARMv8 architecture provides the DC IVAC (Data Cache Invalidate by Virtual Address to Point of Coherency) instruction for this purpose. Similarly, when the CPU writes data that needs to be accessed by a NIC-400-connected master, it must perform a cache clean operation using the DC CVAC (Data Cache Clean by Virtual Address to Point of Coherency) instruction to ensure the data is flushed to DDR.
Second, memory barriers must be used to enforce the correct ordering of memory accesses. The ARM architecture provides several types of memory barriers, including DMB (Data Memory Barrier), DSB (Data Synchronization Barrier), and ISB (Instruction Synchronization Barrier). These barriers ensure that memory operations are completed in the specified order, preventing reordering that could lead to coherency issues. For example, a DMB instruction can be used to ensure that all memory writes by the CPU are completed before a hardware accelerator begins reading from DDR.
Third, the system’s memory map and address routing should be carefully configured to minimize overlap between regions accessed by the NIC-400 and CCI-500. If overlap is unavoidable, software must manage coherency explicitly by performing cache maintenance operations and using memory barriers as described above.
Finally, debugging and verifying coherency in such systems requires a thorough understanding of the ARM architecture and the specific implementation of the NIC-400 and CCI-500 interconnects. Tools such as ARM’s CoreSight and DS-5 Debugger can be invaluable for tracing memory accesses and identifying coherency issues. Additionally, simulation and emulation platforms can help validate the system’s behavior before deploying it in production.
By following these best practices, developers can ensure data coherency in systems that use both NIC-400 and CCI-500 to access DDR, avoiding subtle bugs and achieving reliable performance.
