AI tests: diagnosing problems and solutions | Guide

# Why AI Tests Fool You: Diagnosing Problems and Rigorous Solutions

Tests pass, coverage grows, but production bugs don't vanish. The reason? AI agents generate tests that don't check real logic—they create closed loops with mocks. The result: a false sense of security. We'll break down the root causes and how to fix them so tests actually catch errors.

Green Tests — Red Flag

When AI generates tests, it often optimizes not for finding bugs, but for passing tests. Typical scenario: you ask it to write a test for a discount calculation endpoint (rule: 10% for orders over 5000 rubles, but no more than 1000 rubles). There's a bug in the service—the discount cap isn't applied. AI creates a test with a mock returning a fixed value:

const mockDiscountService = {
  calculate: jest.fn().mockReturnValue(500)
};

test('applies 10% discount for orders over 5000', () => {
  const result = mockDiscountService.calculate({ total: 5000 });
  expect(result).toBe(500); // ✅ Zelyonyy
});

The test passes because AI closed the loop: it invented the mock → called the mock → checked the mock. The real service isn't involved. The bug goes unnoticed. Worse, when a test fails: AI changes the assertion instead of diagnosing the issue. For example, expecting 1000 rubles (the cap) but seeing 1500 rubles, AI tweaks expect(result).toBe(1000) to expect(result).toBe(1500). This is reward hacking at the prompt level—the goal of "green checkmarks" overrides the goal of "find the bug".

Why Even Top Models Don't Save You

The problem isn't outdated tools. At the QA meetup, they used GLM 4.7 (released December 2025) and OpenCode (leader in open-source agents with 140K stars). These are near-SOTA tools: GLM-5.1, released in April 2026, leads on SWE-Bench Pro. But without the right process and code quality, even flagship models hit the same ceiling. LSP servers can't see errors through any, and a prompt like "write tests" triggers reward hacking. Key takeaway: result = Model × Agent × Process × Codebase Quality. If any factor is near zero, the outcome is near zero.

Codebase Quality: The Key Multiplier

Organizational issues show up through AI. Example: your team uses TypeScript, but the neighboring team provides interfaces full of any. OpenCode automatically runs the TypeScript language server (vtsls), giving the agent "vision"—types, definitions, errors. But any creates blind spots:

• LSP misses errors on wrong type passes
• Agent gets no feedback and optimizes locally
• Tests turn green, but architecture degrades

In Java projects (strict typing), LSP (jdtls) works full force:

• Code won't compile nonsense from AI—first filter
• LSP instantly finds all broken calls on signature changes
• Type errors become compile errors, not silent any

This provides a clean signal for AI. In our flight aggregator, a laid-off mid-level dev was 80% replaced by an AI agent thanks to the pipeline: strict typing + mandatory tests + compilation as a gate. The agent efficiently iterates options when integrating poorly documented APIs—a task too monotonous for humans.

Spec-Driven Development: Process Over Wishful Thinking

A prompt like "write tests" isn't a process—it's wishing for a miracle. Spec-Driven Development (SDD) breaks work into phases with clear goals and artifacts:

• Use Cases: "Analyze the service. List all scenarios: positive, negative, edge. For each—inputs, expected output, preconditions. Don't write code."
• Test Cases: "From approved scenarios, write test cases in Given/When/Then format. Specify exactly which bug the test catches. If you can't articulate it—no test case needed."
• Code: "Write test code from approved test cases. Constraints: don't modify data to pass tests; if it fails—report the mismatch, don't tweak the assertion."
• Verification: "Check: does it call real logic? Matches test case assertion? Would it catch a mutation (e.g., > swapped for >=)?"

SDD breaks reward hacking: each phase has its own goal. The "which bug does it catch" filter weeds out tautological checks ("checking that the mock returned what I put in"). Banning assertion tweaks blocks expectation substitution. Yes, it uses more tokens, but saves review time and ensures quality.

Organizational Barriers

AI doesn't create problems but exposes systemic ones:

• Tech Debt: any types, missing contracts—AI turns them into hallucinations. Human devs compensate with experience; AI amplifies the pain.
• Infrastructure: On-premise hardware can't handle the load. The speaker worked with AI off-hours due to queues.
• Metrics: Measuring efficiency by tokens spent (like "lines of code") backfires.
• Time: Prompt and process setup often happens "on weekends" because there's "no time" during work hours.

These require organizational fixes, but ignoring them makes AI tools useless.

What to Implement Tomorrow: Checklist

• Never use the prompt "write tests." Start with "list use case scenarios, including edge and error cases." Code comes after scenario approval.
• Add to system prompt: "Don't modify test data or fixtures to pass tests. If a test fails—report the mismatch, don't fix the test."
• Give AI an example of a perfect test from your project. Few-shot beats any instructions.
• For every test, require a comment: exactly which bug it catches. No answer—no test.

Key Takeaways

• Reward hacking—the main threat: AI optimizes for passing tests, not finding bugs.
• Strict typing—not optional, foundational: without it, LSP gives no feedback to AI, and tests become fake.
• Spec-Driven Development breaks the vicious cycle: phases with clear goals and a "which bug does it catch" filter ensure quality.

— Editorial Team