Returning x86-64 Structures: Registers and MEMORY ABI

How x86-64 Returns Values Using the System V ABI

The x86-64 compiler under the System V ABI uses the rax and rdx registers to return the first two integer values. For structs, classification rules apply: INTEGER for small sizes and MEMORY for larger ones. This determines whether the value is returned directly or through a hidden pointer passed in rdi.

Let’s examine the basic case of returning an int:

one_plus_one():
  mov DWORD PTR [rbp-4], 2
  mov eax, DWORD PTR [rbp-4]
  ret

The value is loaded into eax (the lower 32 bits of rax). Similarly, for 64-bit types, the full rax register is used.

Small Structs Up to 16 Bytes: Return via Registers

Small structs are returned directly in rax and rdx. Consider a 16-byte struct:

struct nums {
    std::int64_t first{};
    std::int64_t second{};
};
nums construct() {
    nums ret{10, 120};
    return ret;
}

Generated assembly:

construct():
  mov QWORD PTR [rbp-16], 10
  mov QWORD PTR [rbp-8], 120
  mov rax, QWORD PTR [rbp-16]
  mov rdx, QWORD PTR [rbp-8]
  ret

Fields are laid out sequentially: first goes into rax, second into rdx. The memory layout follows little-endian order, with the stack growing downward.

• INTEGER classification: Up to two 64-bit fields fit in registers.
• Alignment: Fields are aligned to 8-byte boundaries.
• Limitation: Total size must be ≤16 bytes for direct return.

Large Structs: Hidden Argument in rdi

Structs larger than 16 bytes are classified as MEMORY. The caller allocates memory and passes its address in rdi as a hidden first argument.

Example: a 32-byte struct:

struct many_nums {
    std::int64_t first{};
    std::int64_t second{};
    std::int64_t third{};
    std::int64_t fourth{};
};
many_nums construct_scary() {
    many_nums temp{10, 20, 30, 40};
    return temp;
}

In construct_scary:

construct_scary():
  mov QWORD PTR [rbp-8], rdi
  mov rax, QWORD PTR [rbp-8]
  mov QWORD PTR [rax], 10
  mov QWORD PTR [rax+8], 20
  mov QWORD PTR [rax+16], 30
  mov QWORD PTR [rax+24], 40
  ret

Repeated mov rax, [rbp-8] is an artifact of -O0. Optimized code uses rdi directly.

In main:

main:
  sub rsp, 32
  lea rax, [rbp-32]
  mov rdi, rax
  call construct_scary
  ret

• Memory allocation: Done via sub rsp, size or on the heap/stack.
• Address passing: The address is passed in rdi before the call.
• MEMORY classification: Used by System V ABI for types >16 bytes or non-INTEGER types.

Return Value Classification Rules

System V ABI defines three categories:

• INTEGER: Up to 16 bytes, scalar or aggregate fields ≤2×64 bits → returned in rax/rdx.
• MEMORY: >16 bytes, SSE-packed types >2×128 bits, or non-INTEGER → hidden rdi argument.
• SSE: Floating-point values up to 128 bits → returned in xmm0 (not covered here).

Compilers like GCC and Clang generate code based on these rules. Use -O0 during debugging to see intermediate steps clearly.

Key Takeaways

• Use rax/rdx for integer returns up to 16 bytes.
• For MEMORY-classified types, the caller provides memory via rdi.
• System V ABI ensures consistent behavior across Linux and macOS.
• Optimizations (-O2+) eliminate unnecessary mov instructions.
• Verify generated code using objdump -d or Godbolt Compiler Explorer.

— Editorial Team