Assembler x86-64 GUI X11 fasmg

Minimal GUI in x86-64 Assembly with X11: Optimization for fasmg

Developers writing low-level code often face the challenge of minimizing executable file size while maintaining functionality. This article explores creating a minimal GUI in x86-64 assembly using the X11 protocol. We use fasmg to generate a compact binary without a linker, mmap for dynamic memory allocation, and macros to simplify system calls.

The goal is a GUI window with text, smaller than original implementations. Focus is on code readability for mid-level to senior developers: X11 request structures, ABI compatibility, and data copy optimization.

Setting Up the Development Environment

Build in VSCode with tasks for release and debug modes. Release: DEBUG=0, debug: DEBUG=1 with DWARF symbol generation.

{
    "label": "Make release executable",
    "type": "shell",
    "command": "chmod",
    "args": ["+x", "${fileBasenameNoExtension}"],
    "dependsOn": "Release build with fasmg"
}

Debugging in Cutter with launch.json:

{
    "name": "Launch Cutter",
    "type": "node",
    "request": "launch",
    "preLaunchTask": "Debug build with fasmg",
    "runtimeExecutable": "Cutter.AppImage",
    "runtimeArgs": [
        "${fileDirname}/${fileBasenameNoExtension}",
        "--analysis", "2",
        "--arch", "x86",
        "--bits", "64"
    ]
}

Press Ctrl+Shift+T for debug builds (.asm files).

Macros for Debugging and Logging

Dynamically link libc only in debug mode:

include 'dynamic/import64.inc'
if DEBUG
    interpreter '/lib64/ld-linux-x86-64.so.2'
    needed 'libc.so.6'
    import printf
end if

Macros for vararg printf with support for %d, %f, %e, %g, %a. COUNT_ARGS_TO counts arguments, GET_VARARG_TO retrieves by index.

Convert float/double to IEEE-754 via fasm virtual memory:

macro MOVQ_IMM reg, val
    local bits
    virtual at 0
        dq val
        load bits qword from 0
    end virtual
    mov rax, bits
    movq reg, rax
end macro

The main VARG_FUNC macro parses the format string, distributing arguments to registers according to System V ABI (xmm0-xmm7 for float, rsi/rdx/... for integer). DEBUG_MSG is a wrapper for logs.

X11 Request Structures

A unified FULL_LAYOUT structure contains all necessary requests:

struct FULL_LAYOUT
    ridBase             dd  0
    ridMask             dd  0
    windowRootId        dd  0
    rootVisualId        dd  0
    gcId                dd  0
    exposed             db  0
    create_window_req   X11_CREATE_WINDOW
    draw_string_req     STRING_TO_DRAW
    handshake           X11_HANDSHAKE
    gc_request          X11_CREATE_GC
    addr_un             SOCKADDR_UN
    map_window_req      X11_MAP_WINDOW
    union
        free    rb     HEAP_SIZE - ($ - create_window_req)
        setup   X11_SETUP
        pollfd  POLLFD
    endu
ends

Fields are ordered to minimize copy code. Layout in r12, sockets in r13+.

Universal System Call Macro

MAKE_SYSCALL automates setting registers (rdi, rsi, rdx, r10, r8, r9) with support for ptr [c], [c], 0:

macro MAKE_SYSCALL scall, args&
    iterate arg, args
        if % = 1
            match =ptr [c], arg
                lea    rdi, [c]
            else match [c], arg
                mov     rdi, [c]
            else match =0, arg
                xor    rdi, rdi
            else
                mov    rdi, arg
            end match
        ; similarly for rsi, rdx, r10...
    end iterate
    ; syscall scall
end macro

Size and Performance Optimizations

• mmap instead of static data: 4KB page for all X11 structures.
• movsb for copying: grouping fields minimizes calls.
• Registers for constants: r12-r15 remain unchanged.
• Default values in binary: macros generate only used fields.
• Fixed font: X11 automatically selects "fixed".

Advantages of fasmg: compact code, macros, no linker.

Key Takeaways

• X11 request structures are ordered for optimal movsb copying.
• Vararg printf macro with IEEE-754 conversion via virtual memory.
• Universal syscall macro reduces boilerplate by 70%.
• mmap 4KB covers all structures without static data in the binary.
• Debug/release builds in VSCode + Cutter for reverse engineering.

— Editorial Team