Precise LLM Arithmetic via Code

Exact Arithmetic in LLMs: Generating Executable Code

LLMs aren't built for precise calculations—the transformer architecture predicts tokens based on probabilities, not mathematical operations. A query like "18 × 38.76" might return 697.68 or an error like 680 due to pattern approximation from training data. This is a systemic limitation, not model degradation.

Solution: Keep the LLM out of direct computation. The model generates a Python script for the task, which runs in an isolated environment. Python ensures accuracy: 18 * 38.76 always returns exactly 697.68.

System Architecture

Processing flow: user → LLM → Python script → Docker container → result (text + Excel).

Example: calculating utility bills (Tomsk):

• User input: «CW 320, HW 229, electricity 7422, prev: CW 302, HW 222, electricity 7133».
• LLM with system prompt (rates from config) generates a script.
• Script computes usage, applies tariffs, and builds a table.
• Execution in sandbox, file returned.

Simplified generated code:

# Meter readings
cold_current, cold_prev = 320, 302
hot_current, hot_prev = 229, 222
elec_current, elec_prev = 7422, 7133

# Usage
cold_usage = cold_current - cold_prev   # 18 m³
hot_usage = hot_current - hot_prev      # 7 m³
elec_usage = elec_current - elec_prev   # 289 kWh

# Tomsk 2025 rates (from config, not model)
tariffs = {
    'cold_water': 38.76,
    'hot_water': 142.63,
    'electricity': 4.94,
    'drainage': 27.04,
}

# Calculation
cold_cost = cold_usage * tariffs['cold_water']       # 697.68
hot_cost = hot_usage * tariffs['hot_water']           # 998.41
elec_cost = elec_usage * tariffs['electricity']       # 1427.66
drain_cost = (cold_usage + hot_usage) * tariffs['drainage']  # 676.00

total = cold_cost + hot_cost + elec_cost + drain_cost  # 3799.75

# Generate Excel
import openpyxl
wb = openpyxl.Workbook()
ws = wb.active
ws.append(['Service', 'Usage', 'Rate, ₽', 'Amount, ₽'])
ws.append(['Cold Water', f'{cold_usage} m³', tariffs['cold_water'], cold_cost])
ws.append(['Hot Water', f'{hot_usage} m³', tariffs['hot_water'], hot_cost])
ws.append(['Electricity', f'{elec_usage} kWh', tariffs['electricity'], elec_cost])
ws.append(['Sewage', f'{cold_usage + hot_usage} m³', tariffs['drainage'], drain_cost])
ws.append(['TOTAL', '', '', total])
wb.save('communal.xlsx')

Rates are injected from an external config file updated via official sources—no hallucinations from the model.

Choosing the LLM and Sandbox

Models:

• Qwen (Alibaba): Free API, reliable code generation for simple tasks.
• DeepSeek V3: For complex scenarios (document analysis), supports prompt caching.

GPT-4/Claude are overkill for scripts under 20 lines.

Sandbox: Docker container on demand.

• User isolation.
• Base libraries: openpyxl, math, datetime.
• 30-second timeout.
• Pandas/numpy blocked in prompts.

Common Issues and Fixes

| Problem | Solution |

|--------|----------|

| Rate hallucination | Inject rates from config into prompt |

| Direct calculation without code | Validation + retry with instruction: "only code" |

| Garbled text in Excel | Use encoding='utf-8', post-validation |

| Missing imports | Restrict libraries in image and prompt |

| UX: incorrect token count | Backend-based counting |

| Code errors | Return stderr to model for iterative fixes |

Returning stderr reduces iterations by 5x—models analyze tracebacks and fix issues faster.

Extensions: Budget Analysis

For checking budgets (Excel file, city): model generates 150–200 lines of code.

• Parses arbitrary structures.
• Compares against market prices from config.
• Detects arithmetic errors.
• Generates reports with color coding (openpyxl).

Example: bathroom renovation budget (Tomsk)—overcharge of 25.9%, 8 items >50% over, discrepancies of 1–4 rubles.

Key Takeaways

• Arithmetic is 100% accurate: executed via Python.
• Rates come from external data, never from the model.
• Sandbox is mandatory for security.
• Iterations using stderr speed up debugging.
• Scalable: update config without retraining.

— Editorial Team