Physics of LLM-graphs: metrics F and rho

The Physics of LLM Agent Coordination: Degradation Metrics in Multi-Agent Graphs

Multi-agent systems based on LLMs often exhibit instability: adding agents leads to context loss and hallucinations. Research using the Llm coordination harness benchmark revealed key metrics—Fidelity (F), error correlation (rho), propagation balance (B), and context pressure (C). These indicators explain why hierarchical topologies disrupt information flow.

Testing was conducted on Qwen 3.5 Plus and Gemini 3.1 Flash Lite models with tasks from CRAFT-mini and AgentsNet-mini. Communication budgets were limited to 0, 32, or 96 tokens per message. Topologies: Star and Balanced Tree.

Metrics for Communication Analysis

The benchmark extracts four variables from events.jsonl logs, measuring the graph's physics:

• F (Fidelity): percentage of critical facts surviving at each hop from leaf to root.
• rho (error correlation): tendency of agents to make identical errors without communication (vote_local).
• B (Propagation Balance): Gini coefficient for the distribution of surviving facts across edges.
• C (Fan-in pressure): ratio of incoming tokens to a node's context window.

Synthetic tests (test_replay.py) confirmed metric independence from final score: with an incorrect answer (Score=0), the extractor recorded F=0.66 and B<1.0.

Dataset—144 cycles, 2000 API calls in a sterile environment without fallbacks.

Cheating of Baseline Predictors

Baseline ML models (Heuristic RF) assess success based on mean_billed_tokens (48% importance), ignoring the essence. The Core RF model, with access to F, rho, B, C, prioritizes rho (36%), F—15%, B—8%.

This shows: tokens do not correlate with effectiveness. Predictors without the graph's physical metrics are blind to real coordination issues.

Topological Penalty: Why Hierarchy Breaks Down

In the Star topology, score reaches 1.00. Switching to Balanced Tree at 96 tokens reduces it to 0.75 for Gemini (-0.25 delta).

Reasons in metrics:

• F drops by 25%: facts are lost at intermediate nodes due to compression.
• B decreases: propagation imbalance amplifies the "telephone game."

Hierarchical structures consume context under token limits. Flat graphs preserve information better.

The Natural Quarantine Paradox

Introducing a saboteur (leaf agent with false answers) revealed a trade-off.

• Star: the virus spreads instantly (1 hop to root), score collapses.
• Tree: degradation in F filters misinformation at intermediate nodes, ensuring resilience.

Efficient communication increases vulnerability. Deep graphs sacrifice speed for quarantine.

Key Takeaways

• Metrics F, rho, B, C predict collapse independently of final score.
• Hierarchies (Tree) reduce Fidelity by 25% with a 96-token budget.
• Star is optimal for coordination but vulnerable to sabotage.
• Tree provides a natural filter for misinformation at the cost of fact loss.
• Benchmark v0.1.0 is open for analyzing LLM graphs without alchemy.

Future Directions

Current LLMs are too stable for organic collapse. v0.2.0 will expand metrics, add attacks, and search for an order parameter for mathematical swarm design.

The benchmark captures negative results: multi-agent systems do not always improve. Use it for ablation studies in production.

— Editorial Team