Testing Pyramid for AI Assistants in QA

Testing Pyramid as an Architectural Constraint for AI Assistants in QA

When a language model becomes a test designer, the classic testing pyramid stops being just a recommendation—it turns into a strict technical requirement. The reason is simple: the LLM's context window physically can't accommodate a full description of a complex Feature with dozens of User Stories and Tasks. Trying to "feed it everything at once" leads to a drop in quality or complete failure to generate.

Why the Pyramid Solves the Context Problem

The testing pyramid offers a natural way to decompose tasks. Instead of loading the model with a description of the entire IT solution, you work with levels:

• Task Level (Component) — only the API contract and business rules for one microservice.
• User Story Level (System) — front-end and back-end integration within a single information system.
• Feature Level (E2E) — interactions between systems and end-to-end business processes.

Each level requires its own context. And each fits within LLM limits when provided in isolation. This isn't theory—it's a practical necessity.

Example: The "Promo Codes During Order Checkout" Feature consists of three User Stories and nine Tasks. The full description, including mockups, API specifications, and test data, takes ~150K tokens. Claude 3 Opus with its 200K token window can technically handle it—but test quality on lower levels drops due to "noise" from higher levels. Breaking it down by levels reduces the load to 8–15K tokens per request—and quality improves dramatically.

How the AI Assistant Uses Pyramid Levels

In the QA Assist system, each testing level is handled by a dedicated agent. The architecture looks like this:

• Unit/API Agent — receives the specification for one endpoint or component. Generates unit tests and API tests covering boundary values, negative scenarios, and backward compatibility checks.
• System Agent — receives the User Story description and interfaces of the involved components. Generates system tests that verify integration and user scenarios.
• E2E Agent — receives the Feature map and User Story statuses. Generates end-to-end tests that check data consistency across systems.

The key point is that each agent's context is strictly limited to its level. The Unit agent doesn't know about the admin panel. The System agent doesn't see order history. This isn't a limitation—it's an optimization tailored to LLM capabilities.

Concrete Examples: What AI Generates at Each Level

Take a Task: Back-end API for applying a promo code (POST /orders/apply-promo).

What the Unit/API Agent Receives:

• Endpoint specification (input parameters, output fields, error codes).
• Business rules for discount calculation (fixed amount, percentage, combined discounts).
• Constraints (usage limits, expiration dates, geographic restrictions).

What the Agent Generates:

# Example sgenerirovannogo test
@pytest.mark.parametrize("promo_code, expected_discount", [
    ("SALE10", 10.0),      # Protsentnaya skidka
    ("FIXED50", 50.0),     # Fiksirovannaya sum
    ("COMBO", 60.0),       # Kombinirovannaya (10% + 50 rubles)
])
def test_valid_promo_codes(api_client, promo_code, expected_discount):
    response = api_client.post("/orders/apply-promo", json={"code": promo_code})
    assert response.status_code == 200
    assert response.json()["discount"] == expected_discount

@pytest.mark.parametrize("invalid_code", ["EXPIRED", "OVERUSED", "INVALID_FORMAT"])
def test_invalid_promo_codes(api_client, invalid_code):
    response = api_client.post("/orders/apply-promo", json={"code": invalid_code})
    assert response.status_code == 400
    assert "error" in response.json()

Now take a User Story: "User applies promo code during order checkout".

What the System Agent Receives:

• Scenario description (user adds item, enters promo code, sees discount, completes order).
• Front-end and back-end interfaces (input form, API calls, total display).
• UX requirements (client-side validation, error messages, loading states).

What the Agent Generates:

// Example Playwright-test
test('User applies valid promo code during checkout', async ({ page }) => {
  await page.goto('/cart');
  await page.fill('#promo-code', 'SALE10');
  await page.click('#apply-promo');
  
  // Checking otobrazhenie skidki
  await expect(page.locator('.discount-amount')).toHaveText('-10%');
  
  // Checking pereschyot itogovoy summy
  const originalTotal = await page.locator('.original-total').innerText();
  const discountedTotal = await page.locator('.final-total').innerText();
  expect(parseFloat(discountedTotal)).toBeLessThan(parseFloat(originalTotal));
  
  // Oformlyaem zakaz
  await page.click('#checkout-button');
  await expect(page).toHaveURL('/order-confirmation');
});

Why LLMs Struggle More with Upper Levels

Language models show a predictable decline in quality when dealing with high-level abstractions:

• Feature/E2E Level — requires understanding business logic that's often not explicitly formalized. The model must make assumptions that could be incorrect.
• System/User Story Level — better, but still demands integrating knowledge from multiple sources (front-end, back-end, UX).
• Task/Unit Level — ideally suited to LLM strengths: clear specifications, formal rules, predictable inputs/outputs.

Empirical rule: the lower the pyramid level, the higher the accuracy of test generation by the AI assistant. In practice, this means:

• For Unit/API tests—trust AI almost completely (90–95% coverage).
• For System tests—conduct reviews, especially for complex integrations (70–80% coverage).
• For E2E tests—use AI only for generating drafts that need significant human refinement (50–60% coverage).

Architectural Takeaways for AI Assistants

If you're building an AI system for test generation, ignoring the testing pyramid is a recipe for failure. Here are the key principles we built into QA Assist:

• Decomposition by Levels — mandatory. No prompts dumping the entire Feature at once.
• Isolated Context — each agent works solely with its level. Information passed between levels via structured data, not textual context.
• Human in the Loop — especially at upper levels. AI generates a draft; humans refine and approve.
• Feedback Loop — test execution results are used to fine-tune agents. If AI misses a boundary case, it's added to the training set.

Without these principles, even the most powerful LLM will produce tests that seem plausible but fail to catch real bugs.

Key Takeaways

• The testing pyramid isn't just a methodology—it's a technical requirement for effective AI assistants.
• The LLM context window physically prevents processing complex Features in full without quality loss.
• Decomposition by levels (Task → User Story → Feature) is the only way to produce quality tests across all levels.
• AI excels at lower pyramid levels (Unit/API) and struggles more with upper ones (E2E).
• Humans remain essential, particularly for reviewing and refining System and E2E tests.

— Editorial Team