ARMv8 DMB NSHLD vs. ISHLD: Practical Differences and Use Cases

ARMv8 Memory Barriers and Shareability Domains: The Core Issue

In ARMv8 architectures, memory barriers (DMB) are critical for enforcing memory ordering guarantees across different processing elements (PEs) and shareability domains. The primary issue under discussion revolves around the practical differences between DMB NSHLD (Non-shareable Load Barrier) and DMB ISHLD (Inner Shareable Load Barrier). Specifically, the question is: When and why should a developer choose DMB ISHLD over DMB NSHLD?

The confusion arises because load barriers (DMB LD) are often perceived as less critical than store barriers (DMB ST), as loads do not directly modify memory. However, in multi-core or multi-threaded systems, the order in which loads are observed can have significant implications for program correctness, especially when shared memory is involved. The shareability domain of a barrier determines which observers (other PEs or devices) must respect the ordering guarantees enforced by the barrier.

For example, DMB NSHLD ensures that load operations are ordered only within the non-shareable domain of the executing PE, meaning other PEs in the same inner or outer shareable domain may not observe the same ordering. On the other hand, DMB ISHLD ensures that load ordering is visible to all observers within the inner shareable domain, which typically includes multiple PEs on the same cluster or core complex. Understanding these distinctions is crucial for designing correct and efficient multi-threaded or multi-core systems.

Memory Ordering and Shareability Domains: Key Concepts

To fully grasp the differences between DMB NSHLD and DMB ISHLD, it is essential to understand the concepts of memory ordering and shareability domains in ARMv8.

Memory Ordering in ARMv8

ARMv8 employs a weakly-ordered memory model, meaning that memory operations (loads and stores) may be reordered by the hardware unless explicitly constrained by memory barriers or other synchronization mechanisms. This reordering can occur for performance optimization, such as allowing loads to bypass stores to different addresses. However, this flexibility can lead to unexpected behavior in multi-threaded or multi-core systems, where one PE’s memory operations must be observed in a specific order by another PE.

Memory barriers (DMB, DSB, and ISB) are used to enforce ordering constraints. A DMB (Data Memory Barrier) ensures that memory operations before the barrier are completed before any memory operations after the barrier, with respect to the specified shareability domain. The LD variant of DMB (DMB LD) specifically enforces ordering for load operations, ensuring that loads before the barrier are observed before loads after the barrier.

Shareability Domains in ARMv8

Shareability domains define the scope of memory coherence and ordering guarantees. ARMv8 defines three primary shareability domains:

1. Non-shareable (NSH): Memory operations are only visible to the executing PE. This domain is typically used for private memory regions that are not accessed by other PEs or devices.
2. Inner Shareable (ISH): Memory operations are visible to all PEs within the same inner shareable domain. This domain usually includes multiple PEs on the same cluster or core complex.
3. Outer Shareable (OSH): Memory operations are visible to all PEs within the same outer shareable domain, which may include multiple clusters or core complexes.

The choice of shareability domain for a memory barrier depends on the scope of observers that need to respect the ordering guarantees. For example, if two threads are running on different PEs within the same inner shareable domain, a DMB ISH barrier is required to ensure that memory operations are ordered correctly across both PEs. If the threads are running on PEs in different shareability domains, a DMB OSH barrier may be necessary.

Practical Implications of Load Barriers

While store barriers (DMB ST) are often more straightforward to understand (e.g., ensuring that a store to a flag variable is observed after a store to a message buffer), load barriers (DMB LD) are more subtle. Load barriers ensure that the order in which loads are performed is respected, which can be critical in scenarios where one PE’s loads depend on the results of another PE’s stores.

For example, consider a producer-consumer scenario where one PE writes a message to a shared buffer and sets a flag to indicate that the message is ready. The consumer PE reads the flag and, if set, reads the message. To ensure correctness, the consumer must observe the flag and message in the correct order. A DMB LD barrier on the consumer PE ensures that the load of the flag is observed before the load of the message.

However, the choice of shareability domain for the load barrier depends on the relationship between the PEs. If the producer and consumer are in the same inner shareable domain, a DMB ISHLD barrier is required. If they are in different domains, a DMB OSHLD barrier may be necessary. Using a DMB NSHLD barrier in this scenario would only enforce ordering within the consumer PE’s non-shareable domain, which is insufficient if the producer is in a different domain.

Practical Scenarios and Correct Usage of DMB NSHLD vs. ISHLD

To illustrate the practical differences between DMB NSHLD and DMB ISHLD, let’s examine a few scenarios where the choice of barrier is critical for program correctness.

Scenario 1: Producer-Consumer with Shared Memory

Consider a producer-consumer scenario where two threads (Thread 0 and Thread 1) are running on different PEs within the same inner shareable domain. Thread 0 writes a message to a shared buffer and sets a flag to indicate that the message is ready. Thread 1 reads the flag and, if set, reads the message.

The code for Thread 0 might look like this:

MOV W0, #1          // Write message value
STR W0, [MSG]       // Store message to shared buffer
DMB ISHST           // Ensure message store is observed before flag store
MOV W1, #1          // Set flag value
STR W1, [FLAG]      // Store flag to shared buffer

The code for Thread 1 might look like this:

LDR W0, [FLAG]      // Load flag value
DMB ISHLD           // Ensure flag load is observed before message load
LDR W1, [MSG]       // Load message value

In this scenario, DMB ISHST on Thread 0 ensures that the store to MSG is observed before the store to FLAG by all observers in the inner shareable domain. Similarly, DMB ISHLD on Thread 1 ensures that the load of FLAG is observed before the load of MSG by all observers in the inner shareable domain. Using DMB NSHLD on Thread 1 would only enforce ordering within Thread 1’s non-shareable domain, which is insufficient because Thread 0 is in a different domain.

Scenario 2: Multi-PE Synchronization with Acknowledgment

Now consider an extended version of the producer-consumer scenario where Thread 1 acknowledges receipt of the message by clearing the flag. Thread 0 waits for the flag to be cleared before writing a new message. This introduces additional dependencies between the PEs’ memory operations.

The code for Thread 0 might look like this:

MOV W0, #1          // Write first message value
STR W0, [MSG]       // Store first message to shared buffer
DMB ISHST           // Ensure first message store is observed before flag store
MOV W1, #1          // Set flag value
STR W1, [FLAG]      // Store flag to shared buffer

// Wait for flag to be cleared
PollLoop:
LDR W2, [FLAG]      // Load flag value
CBNZ W2, PollLoop   // Loop until flag is cleared

DMB ISHLD           // Ensure flag load is observed before second message store
MOV W0, #2          // Write second message value
STR W0, [MSG]       // Store second message to shared buffer

The code for Thread 1 might look like this:

LDR W0, [FLAG]      // Load flag value
DMB ISHLD           // Ensure flag load is observed before message load
LDR W1, [MSG]       // Load message value

// Acknowledge receipt by clearing flag
MOV W2, #0          // Clear flag value
STR W2, [FLAG]      // Store cleared flag to shared buffer
DMB ISHST           // Ensure flag store is observed by Thread 0

In this scenario, DMB ISHLD on Thread 1 ensures that the load of FLAG is observed before the load of MSG by all observers in the inner shareable domain. Similarly, DMB ISHST on Thread 1 ensures that the store to FLAG is observed by Thread 0 before it writes the second message. Using DMB NSHLD or DMB NSHST in this scenario would not provide the necessary ordering guarantees across PEs.

Scenario 3: Single-PE Load Ordering

In some cases, a developer may only need to enforce load ordering within a single PE. For example, consider a scenario where a PE performs two loads from different memory locations and needs to ensure that the first load is observed before the second load. This might be necessary if the second load depends on the result of the first load.

The code might look like this:

LDR W0, [ADDR1]     // Load first value
DMB NSHLD           // Ensure first load is observed before second load
LDR W1, [ADDR2]     // Load second value

In this scenario, DMB NSHLD is sufficient because the ordering constraint only applies to the executing PE. There is no need to involve other PEs or shareability domains. However, if the loads were part of a larger multi-PE synchronization scheme, a stronger barrier (e.g., DMB ISHLD) might be necessary.

Troubleshooting and Best Practices for Using DMB NSHLD and ISHLD

When working with DMB NSHLD and DMB ISHLD, developers should follow these best practices to ensure correct and efficient memory ordering:

1. Identify the Shareability Domain: Determine the shareability domain of the memory operations involved. If multiple PEs or devices need to observe the ordering guarantees, use DMB ISHLD or DMB OSHLD as appropriate. If only the executing PE needs to observe the ordering, DMB NSHLD may be sufficient.

2. Pair Barriers Correctly: Ensure that load barriers (DMB LD) are paired with corresponding store barriers (DMB ST) in the same shareability domain. For example, if a producer PE uses DMB ISHST, the consumer PE should use DMB ISHLD.

3. Use the Least Restrictive Barrier: Use the least restrictive barrier that provides the necessary ordering guarantees. For example, if DMB NSHLD is sufficient, there is no need to use DMB ISHLD. However, be cautious when making this determination, as underestimating the required barrier strength can lead to subtle bugs.

4. Test and Validate: Use tools like the ARM Memory Model Tool to test and validate memory ordering scenarios. This can help identify potential issues and ensure that the chosen barriers provide the desired guarantees.

5. Document Assumptions: Clearly document the assumptions about shareability domains and memory ordering in the code. This can help other developers understand the reasoning behind the choice of barriers and avoid introducing bugs when modifying the code.

By following these best practices, developers can effectively use DMB NSHLD and DMB ISHLD to enforce memory ordering guarantees in ARMv8 systems, ensuring correct and efficient operation in multi-threaded and multi-core environments.