Hybrid AI Robotics: Safe Integration of LLM and TinyML

# Hybrid AI and TinyML Architecture: Safe Integration in Robotics

Traditional approaches to integrating LLMs into robotics face critical challenges: model non-determinism, processing delays, and security risks. This article breaks down the OpenGrall hybrid architecture, where the LLM handles strategic planning, and TinyML ensures safe real-time action execution.

Safety as an Architectural Principle

LLMs are inherently non-deterministic—one and the same prompt can generate different commands. Directly passing such commands to motors leads to unpredictable behavior. OpenGrall's solution is separation of responsibilities: the LLM sets strategic goals, but the onboard controller based on TinyML makes the final decision on action execution.

The TinyML model, running on a microcontroller (ESP32/STM32), processes sensor data with latency under 10 ms. It:

• Has absolute veto power over LLM commands
• Monitors the platform's physical limits (inertia, payload capacity)
• Ensures emergency stops upon detecting obstacles
• Manages active suspension and complex mechanical systems

TinyML training happens in a simulator with an accurate physical model of the platform. This lets the neural network "feel the hardware": calculate acceleration/braking, compensate for uneven surfaces, optimize movement trajectories. Input data is a minimal set: rangefinder data, odometry, IMU.

Modularity over Monoliths

Popular VLA models (Vision-Language-Action) are often used as black boxes, creating issues:

• Inability to add new sensors without retraining
• Massive computational resource demands
• Lack of transparency in decision-making

OpenGrall applies separation of concerns:

• LLM Agent (Vikhr/Gemma/YandexGPT)

- Sets strategic goals

- Analyzes compressed sensor data

- Generates JSON commands with parameters

- Explains action reasoning

• Execution Layer (TinyML/VLA/hardcoded)

- Converts commands into physical actions

- Enforces safety

- Adapts to specific platforms

This approach delivers key benefits:

• Safety becomes deterministic, not probabilistic
• The LLM can be swapped without rewriting the entire system
• The platform easily adapts to new robot types (hexapods, quadcopters)
• Parallel development of physical and logical layers

Lock-Free Architecture

A critical flaw in most implementations is synchronizing LLMs with fast sensors. OpenGrall uses an asynchronous WebSocket architecture:

• Server (runs on any device)—central hub
• Clients register with roles (operator/agent/esp/lidar)
• Standardized JSON protocol instead of .msg files
• Dynamic data weighting via WeightCalculator

The SensorMemory system automatically reduces a sensor's weight during anomalies. For example, if a lidar is dirty (all sectors show the same distance), its data is ignored until cleaned. This eliminates the need for manual data fusion—the LLM itself decides which sources to trust by analyzing data weight and age.

Token and Prompt Optimization

The main performance killer is inefficient data transmission to the LLM. OpenGrall uses two key methods:

1. Sensor Data Compression

Raw lidar point clouds (thousands of values/sec) are converted into 8 sectors with key parameters. Example processing:

• Objects closer than 80 cm: angle, distance, size, velocity
• Dynamic updates: data age, sensor weight

2. System Prompt

Critical instructions (response format, available commands) are embedded in the model via Ollama. This saves up to 300 tokens per request:

You robot Graal. Dimensions: 51×32×37sm.
0°=↑=directly
Reply JSON: {"action":"move_forward","params":{"speed":300},"reasoning":"..."}
...

The actual LLM request contains only current data:

CURRENT SITUATION:
Time: 1712345720.45
Current intent: inspect the corridor

DIALOG:
Human: Graal, posmotri that tam aheadi
Robot: Understood, investigating

SENSOR DATA (by importantsti):
  • lidar: front=0.4m, front_left=1.2m...

Key Takeaways

• Absolute TinyML Veto turns safety from a probabilistic parameter into a deterministic guarantee
• Modular Architecture allows independent changes to LLM, sensors, and platform
• Dynamic Data Weighting replaces complex sensor fusion algorithms
• System Prompts cut tokens by 65%, enabling use of local models <1B
• WebSocket Protocol ensures compatibility with any devices, including microcontrollers

Even experimental sensors (e.g., ultrasonic sonar on three microphones) integrate without changing agent code. The LLM independently interprets raw data, cross-referencing it with other sensors. This creates an evolving system where each new module boosts the robot's overall intelligence.

— Editorial Team