Synchronization in Go: atomics and memory barriers

Low-Level Synchronization in Go: Atomics, Barriers, and Instruction Reordering

Understanding processor-level synchronization mechanisms isn't an academic luxury—it's a practical necessity for developers building multithreaded Go applications. In this article, we break down how the CPU optimizes code execution, why this breaks expected logic in concurrent programs, and which hardware primitives help restore order.

How the Processor Tricks the Programmer: Reordering and Out-of-Order Execution

When you write x = 1; ready = true, you expect strict sequential order. The processor? Not so much. It reorders independent instructions to fill pipeline stalls. This is called instruction reordering. The goal: maximize utilization of execution units and hide memory latencies.

CPUs use out-of-order execution: instructions run not in program order, but as their operands become ready. For example:

MOVQ A(SB), AX ; load from memory
MOVQ B(SB), BX ; load from memory
ADDQ DX, CX    ; add registers

The processor can start with ADDQ because the operands are already in registers, while sending loads to the background—they'll take hundreds of cycles. The logical result for the current thread stays the same, but other threads might see something different.

Pipeline stages where chaos emerges:

• Fetch → Decode → Register Rename → Dispatch → Execute → Commit

Key stage: Dispatch. Here, the scheduler picks instructions with ready operands, ignoring the original order. The Reorder Buffer (ROB) ensures instruction commits to architectural state happen in the correct sequence—but only for the current core.

Memory Reordering: When the Store Buffer Betrays You

The problem gets worse at the memory level. Writes to RAM aren't instantaneous—they land in the store buffer, then get asynchronously flushed to cache. For one core:

x = 1
ready = true

For another core, it might look like:

ready = true
x = 0

Because the ready=true write could leave the store buffer earlier. This is memory reordering—reordering of memory operations visible to other threads. Classic data race example:

G1: 
x = 1
ready = true

G2:
if ready {
  print(x) // might print 0!
}

Memory Barriers: MFENCE and Other Tools to Restore Order

Solution: memory fences (memory barriers). These instructions force synchronization of operation visibility between cores. On x86, the main barrier is MFENCE:

MOVQ $1, x(SB)
MFENCE           ; barrier
MOVQ $1, ready(SB)

MFENCE guarantees that all stores before it are visible to all cores before operations after it begin executing. It forbids four types of reordering:

• Store → Store
• Load → Load
• Load → Store
• Store → Load

Important: x86 has a relatively strong memory model—many reorderings are already hardware-forbidden. But Store→Load is still possible, so MFENCE remains essential for strict synchronization.

Types of barriers on different architectures:

• x86: MFENCE (full), SFENCE (store→store), LFENCE (load→load)
• ARM: DMB (Data Memory Barrier), ISH (inner shareable domain)
• RISC-V: FENCE (flexible, with direction specifiers)

Atomic Operations: High-Level Abstraction over LOCK

Go hides barriers behind atomic operations from the sync/atomic package. Under the hood, they use the LOCK prefix, which makes read-modify-write operations indivisible across all cores. Example XADDQ:

TEXT ·Xadd64(SB), NOSPLIT, $0-24
	MOVQ	ptr+0(FP), BX
	MOVQ	delta+8(FP), AX
	MOVQ	AX, CX
	LOCK
	XADDQ	AX, 0(BX)   ; atomically: tmp=*ptr; *ptr+=AX; AX=tmp
	ADDQ	CX, AX
	MOVQ	AX, ret+16(FP)
	RET

LOCK works by locking the cache line in the MESI protocol:

• Core acquires exclusive rights to the cache line
• Performs the operation
• Releases the line—changes become visible to other cores

Atomicity ≠ ordering! An atomic operation guarantees indivisibility but doesn't necessarily create a memory barrier. On x86, simple load/store are atomic but don't provide ordering.

Writing Our Own Atomic AND: Hands-On with Go Assembler

The Go standard library doesn't provide atomic.And. Let's implement it ourselves. Create the file Xand8_x86.s:

#include "textflag.h"

TEXT ·And8(SB), NOSPLIT, $0-9
	MOVQ	ptr+0(FP), BX      // BX = pointer
	MOVB	mask+8(FP), AX     // AX = mask (8 bits)
	LOCK
	ANDB	AX, 0(BX)          // *ptr &= mask
	RET

And the corresponding main.go:

package main

import "C"

func And8(ptr *uint8, mask uint8)

func main() {
	var smth uint8 = 3
	And8(&smth, uint8(2))
	println("Result:", smth) // outputs 2
}

Build and run with go run .—the ASM file will be picked up automatically. Note: we use ANDB with the LOCK prefix, even though x86 has no XAND instruction. This works—LOCK is compatible with basic logical operations.

Key Takeaways

• Instruction reordering — CPU optimization that changes the execution order of independent instructions to improve performance.
• Memory reordering — desynchronization of write operation visibility between cores due to the store buffer.
• Memory fences (barriers) — instructions that forcibly synchronize memory operation order between threads.
• Atomic operations — indivisible read-modify-write actions, implemented via LOCK and encapsulating barriers.
• LOCK + ANDB — a working way to implement a bitwise atomic operation missing from the Go standard library.

— Editorial Team