x86 Cacheability, Memory Types & Ordering

“The cache” is not one mechanism. A modern x86 access crosses address translation, a memory-type decision, store/load machinery, one or more coherent data caches, and possibly an external memory or I/O agent. Correct low-level software must keep four contracts separate: cacheability, coherence, ordering, and atomicity.

The path of one access #

For an ordinary load or store:

1. Paging translates the linear address and supplies page-level cache attributes.
2. The Page Attribute Table (PAT) interprets those page bits as a requested memory type.
3. Memory Type Range Registers (MTRRs) contribute the type assigned to the physical range.
4. The architectural precedence rules produce one effective memory type.
5. That type controls speculation, caching, combining, and ordering for the access.

The memory type belongs to the physical resource as reached through a mapping, not to the source-language object. Mapping the same physical page at two linear addresses with incompatible memory types is unsupported and can cause undefined behavior or system failure.

Cache hierarchy and cache lines #

Main memory latency is long relative to core execution. Hardware therefore keeps recently used instructions and data in a hierarchy, commonly split L1 instruction/data caches, a private or per-core L2, and a larger shared last-level cache. Exact capacities, associativity, inclusiveness, replacement, and TLB dimensions are model-specific and must be discovered from CPUID or model documentation rather than hard-coded from an old processor diagram.

The unit of data-cache allocation and coherence is a cache line, commonly 64 bytes on current Intel Core-family processors. Loading one byte can fetch its whole line. Two consequences follow:

• spatial locality can make sequential access cheap;
• unrelated variables on the same line can contend through false sharing even when no thread touches the other's bytes.

The architectural software contract rarely exposes a precise “L1 hit takes N cycles” constant. Such numbers vary by microarchitecture, clock domain, topology, contention, and access pattern; performance work belongs in CPU Caches & the Memory Hierarchy.

MESI coherence is ownership, not synchronization #

Intel describes a MESI-family protocol with four conceptual line states:

Coherence ensures that cores do not permanently keep contradictory writable copies of one coherent cache line. It does not imply that every core observes different addresses in source-code order. Store buffers, speculative loads, and permitted reordering still shape the software-visible memory model.

Locked read-modify-write operations usually obtain exclusive ownership of a cache line and rely on the coherence protocol rather than asserting a physical bus lock, provided the operand is cacheable and contained within one cache line. Split or unsuitable accesses can take a much more disruptive path.

Store buffers explain the central x86 relaxation #

A store does not have to stall retirement until it reaches the coherent cache hierarchy. It can enter a per-logical-processor store buffer, allowing later work to continue. A later load by the same processor can receive the stored value through store-to-load forwarding even before other processors can observe it.

For ordinary write-back memory, Intel's processor-ordering model is often summarized as x86-TSO. The useful operational rules include:

• loads are observed in load order;
• stores are observed in store order;
• a store is not observed before an older load from the same processor;
• a load may appear before an older store to a different location;
• locked operations have a single total order;
• each processor can see its own buffered store before other processors do.

The classic store-buffering outcome is therefore allowed:

; Initially x = 0 and y = 0, both ordinary WB memory.
; Processor 0                 ; Processor 1
mov dword [x], 1              ; mov dword [y], 1
mov eax, [y]                  ; mov ebx, [x]

Both EAX = 0 and EBX = 0 are possible: each load can occur before the other processor's older buffered store has become visible. Coherence is not violated because each address still has a coherent sequence of values.

Use the language or operating-system synchronization primitives that define the intended happens-before relation. Hand-placing instructions is appropriate only when implementing those primitives or interacting with hardware.

Atomicity is a separate axis #

Intel guarantees single-copy atomicity for naturally aligned ordinary accesses of 1, 2, 4, and 8 bytes under the documented conditions. Wider, unaligned, split-line, split-page, and device accesses require their own proof. An atomic access is not automatically an ordered publication protocol.

The LOCK prefix turns supported memory read-modify-write instructions into an indivisible operation and supplies strong ordering around that operation. XCHG with a memory operand is implicitly locked.

; Atomically add EAX to a shared 32-bit counter and return the old value.
; RDI points to a naturally aligned counter in coherent WB memory.
lock xadd dword [rdi], eax

This protects the single operation. It does not automatically make a larger data structure transactionally atomic; other fields still need an explicit protocol.

Fences and serialization are not synonyms #

The ordering instructions have different scopes:

LFENCE, SFENCE, and MFENCE are memory-ordering instructions, not general serializing instructions. Conversely, a serializing instruction constrains the instruction stream more broadly. Neither class implies that dirty cache lines are written all the way to physical non-volatile media.

Ordinary WB stores are already observed in store order, so adding SFENCE between two ordinary WB stores normally proves nothing new. It becomes essential for weakly ordered paths such as non-temporal stores or WC buffers:

; Conceptual publication of four non-temporal stores.
; RDI points to suitably aligned streaming data; RSI points to ready.
movnti [rdi + 0], eax
movnti [rdi + 4], ebx
movnti [rdi + 8], ecx
movnti [rdi + 12], edx
sfence                       ; drain/order the streaming stores
mov dword [rsi], 1           ; publish through the higher-level protocol

The final ready store still needs a concurrency contract understood by its reader. In C, C++, Rust, Java, or the kernel, use the platform's atomic/barrier API rather than assuming a plain source-level load maps to the intended assembly forever.

Architectural memory types #

The effective memory type controls more than whether bytes may reside in L1:

WB is the normal choice for RAM because it minimizes external traffic and participates in cache coherence. UC is the conservative choice for device registers with read or write side effects. WC can greatly improve throughput to a frame buffer or similar aperture, but its writes can combine, collapse, and appear in a weak order; software must explicitly drain them when the device-observation point matters.

An operand that crosses from one page to another page with a different memory type can be written more than once. Ordinary RAM tolerates this; a device register may not. Device mappings and register accesses must respect both page and natural hardware boundaries.

PAT and MTRRs solve different mapping problems #

MTRRs describe memory types over physical-address ranges, normally established by firmware and kept identical across logical processors. PAT lets the operating system select a type at page granularity through bits in the paging entries.

The IA32_PAT MSR has eight entries. A three-bit index formed from PAT, PCD, and PWT in the leaf paging entry selects one entry:

$$\text{PAT index} = (\text{PAT} \ll 2) \;|\; (\text{PCD} \ll 1) \;|\; \text{PWT}$$

For 4 KiB PTEs, the PAT bit is bit 7. For entries mapping 2 MiB or 1 GiB pages, it is bit 12 because bit 7 is PS. The selected PAT type and the MTRR type for the physical range then combine according to the architectural precedence table. It is incorrect to read PCD=1 in isolation and conclude that the final type is always a particular uncached behavior.

MTRRs include a default type, legacy fixed ranges below 1 MiB, and a model-specific count of variable power-of-two ranges. Firmware normally initializes them; an operating system changing MTRRs must coordinate all processors, disable/flush caches and translations as prescribed, update every processor consistently, and restore operation through a rendezvous. This is not an ordinary per-process mapping operation.

Large pages add another constraint: one TLB entry caches one memory type for the large mapping. If the physical range spans MTRR subranges with different types, behavior can be undefined. Use a uniform type across the large page or split it into 4 KiB mappings.

Cache-management instructions have narrow jobs #

CLFLUSH and INVLPG can name the same linear address while acting on unrelated cached facts. One concerns data associated with a physical cache line; the other concerns how a linear page reaches a physical page and which rights it has.

Cache-line writeback is also not synonymous with durable storage. Persistence depends on the platform's persistence domain, memory controller, device protocol, and any required drain operation. A fence orders relevant CPU operations; it does not invent durability beyond the hardware contract.

MMIO: where ordinary-memory intuition fails #

Memory-mapped I/O places device registers or apertures in the physical address map, but their accesses can have side effects:

• reading a status register may acknowledge or consume an event;
• writing a doorbell may trigger DMA;
• two adjacent registers may require a defined order;
• the device may accept only specific widths and alignments.

A correct MMIO path therefore needs all of the following:

1. the platform-provided physical resource and a mapping with the required effective memory type;
2. the device-specified access width and alignment;
3. compiler-visible MMIO accessors so the compiler cannot remove or merge operations;
4. CPU ordering appropriate to the memory type;
5. the bus/device completion rule, often a read-back or a device-specific status transition.

volatile in a high-level language addresses only a narrow compiler transformation problem. It does not select PAT/MTRR state, create CPU fences, make a compound operation atomic, or prove that a posted write reached the device.

Self-modifying and cross-modifying code #

Data-cache coherence alone does not define when already fetched or decoded instructions are discarded. Intel specifies synchronization sequences for self-modifying code and a handshake plus a serializing instruction on the executing processor for cross-modifying code.

The safe conceptual shape for cross-modification is:

1. modifying processor marks the code unavailable;
2. it stores the new instruction bytes;
3. it publishes completion through a synchronization variable;
4. executing processor observes completion;
5. executing processor runs a serializing instruction such as CPUID or SERIALIZE as required;
6. only then does it execute the modified region.

Do not cargo-cult a data fence or CLFLUSH as a universal substitute. The exact sequence depends on whether modification and execution occur on the same logical processor and on the architectural rules for the target processors.

Failure patterns and diagnosis #

The most reliable diagnostic question is: which cached or buffered fact is stale, and which architectural operation is defined to retire that exact fact? Data lines, translations, store-buffer entries, instruction-stream state, and device-posted writes require different answers.