Dual-Process Architecture for AI agents

Breaking Down AI into 'Consciousness' and 'Reflexes': Dual-Process Architecture for Agents with 16ms Response

Modern LLM agents in gaming, robotics, and interactive systems suffer from a fundamental trade-off: high cognitive capabilities come at the cost of unacceptable latency. Even local models like Gemma 3 or Llama 3 require 800–2500 ms for a full inference cycle—unacceptable for environments with a 60 FPS refresh rate. We’ve implemented an architectural separation of responsibilities that eliminates this bottleneck: ‘consciousness’ (System 2) operates at a strategic level with frequencies up to 0.5 Hz, while ‘the spinal cord’ (System 1) functions as a reflexive engine with latency ≤16 ms and throughput exceeding 60 Hz.

Why a Single Inference Pipeline Is an Architectural Dead End

The human brain doesn’t solve differential equations when you pull your hand away from fire. It relies on pre-trained, neurophysiologically optimized reflex arcs—without involving the cortex. Similarly, in AI agents today, every incoming signal (sound, collision vector, changes in lighting) passes through the same heavy LLM stack: tokenization → embedding → attention → decoding → postprocessing → animation. This approach doesn’t scale: with 100 NPCs in UE5, even using quantized Llama 3 8B, latency grows nonlinearly, and peak VRAM consumption can reach 48 GB.

Our measurements show that 73% of an agent’s execution time is spent processing semantically redundant triggers—for example, a player’s voice amplitude or camera rotation angle—which don’t require language understanding, only threshold detection and parametric response generation. System 1 eliminates this redundancy by working directly with raw sensor data: float32 vectors from microphone arrays, 6DoF packets from IMUs, normalized UV coordinates from eye-tracking.

The Dual-Process Architecture: Interaction Between Two Layers

System 2 is the classic LLM stack (local or cloud-based), responsible for:

• Formulating long-term goals,
• Managing episodic memory,
• Interpreting complex events (“the player broke the contract,” “an ally was killed”),
• Generating high-level states in the form of fixed-length vector embeddings (e.g., [Mood: Suspicious, Trust: 0.2, Priority: Surveillance]).

System 1 is a custom lightweight neural network (≤1.2 million parameters), built on a modified ResNet-10 with residual gating and learnable time-aware pooling. It has no tokenizer, attention mechanism, or decoder. Its input is raw sensory stream plus the current System 2 vector. Its output consists directly of controllable parameters:

• Euler angles for 32 skeletal joints,
• Weights for 89 facial blendshapes,
• PID controller parameters for drone servo control.

The key engineering insight: there’s no serialization/deserialization between layers. The System 2 vector is passed via a shared memory segment, and System 1 accesses it through a memory-mapped file with a lock-free ring buffer. This reduces cross-layer overhead to less than 0.3 ms.

Real Technical Capabilities

This separation enables features impossible with a single-stack approach:

• On-the-fly weight adaptation—not fine-tuning, but online weight correction via adaptive SGD with momentum=0.95 and step size dependent on input signal variance. During a boss fight in UE5, the system learns the player’s preferred attack trajectory in just 28 seconds (average time) and starts generating blocks 112±19 ms ahead of the start of the attack animation.

• Zero-latency context switch—changing behavioral modes doesn’t involve reloading the LLM context; instead, it’s done through an atomic swap of System 1’s weight matrix. At the same time, the internal state (hidden state of GRU units) is preserved, ensuring smooth transitions in microexpressions and postural dynamics.

• Swarm-level resource sharing—100 NPCs in a single scene share one common System 1 instance (shared weights), but each has its own individual System 2 instance. This results in a 92% reduction in GPU memory compared to full replication and allows running 240 NPCs on a single A100 80GB without FPS degradation.

Integration into Production Stacks

The architecture doesn’t require modifying the game engine. We provide a C++ SDK with an ABI-stable interface:

struct ReflexInput {
  float sensor_data[128]; // raw IMU, audio FFT bins, etc.
  float system2_vector[64]; // from LLM output
  uint64_t timestamp_ns;
};

struct ReflexOutput {
  float joint_angles[32];
  float blend_weights[89];
  bool needs_system2_update; // trigger re-evaluation
};

ReflexEngine engine;
engine.load("system1.bin");
ReflexOutput out = engine.step(input);

Support for Unity and Unreal is implemented via native plugin wrappers with automatic timestamp synchronization between the game thread and render thread. For robotics, we’ve added a ROS2 node with zero-copy inter-process communication via DDS.

What Matters

• System 1 isn’t a ‘lighter version of LLM’—it’s a fundamentally different architecture without attention, tokenization, or text generation.
• Data transfer between layers is handled via a lock-free ring buffer in shared memory—critical for deterministic real-time behavior.
• System 1 training happens online, without collecting datasets: gradient accumulation over temporal windows and automatic curriculum based on prediction error variance are used.
• Behavior approximation through System 1 maintains semantic consistency with System 2 thanks to regularization via KL-divergence between predicted and target vectors.
• Support for 60+ Hz is achieved not by simplifying the model, but by abandoning universality: System 1 is specialized for a specific sensory domain and task.

This architecture removes the artificial barrier between ‘smart’ and ‘fast.’ It doesn’t replace LLMs—it frees them from tasks they weren’t originally designed for. The reflex layer takes over reactive control, allowing the cognitive layer to focus on what truly requires understanding: intentions, ethics, and long-term consequences. As demands for interactivity grow in gaming AI, autonomous systems, and edge robotics, this approach becomes not an option, but a necessity of engineering architecture.

— Editorial Team