AI-agent for technical specification verification: development, architecture, practice

Building an AI Agent for Technical Spec Audits: Real-World Lessons

Developing a machine learning tool for automated technical specification (spec) reviews demands a clear strategy and insight into common pitfalls. Our hands-on experiment revealed that success hinges on high-quality training data and a hybrid architecture blending large language models (LLMs) with traditional algorithms.

Building the Training Dataset

The first and most critical step was gathering and labeling data. We built a representative dataset of 90 technical specs, split into three categories:

• High-Quality Specs (30 examples) that sailed through development with minimal changes.
• Problematic Specs (30 examples) that sparked conflicts, rework, and missed deadlines.
• Average Specs (30 examples) that worked but had flaws uncovered during development.

Each spec came with a detailed analyst review highlighting strengths and weaknesses. To boost diversity, we included specs from various authors and projects across industries like retail, manufacturing, and services.

Classifying Common Spec Errors

Analyzing problematic specs uncovered systemic issues our agent needed to detect. Key categories:

• Positive Scenarios Only. The requirement outlines the happy path but skips error handling, edge cases, and exceptions.
• Vague Wording. Subjective or fuzzy terms like "fast," "user-friendly," or "flexible" without measurable metrics.
• Untestable Requirements. No clear way to build a definitive test case for verification.
• Internal Conflicts. Inconsistencies between sections or clauses within the same spec.
• Missing Edge Cases. No guidance on system behavior with extreme or unusual inputs.

Examples of Turning Untestable Requirements into Testable Ones

Typical vague statements and how to make them concrete:

• Original: "Users should flexibly customize reports."

Concrete: "Reports offer customizable settings: date range (any interval), counterparty (multi-select). Settings save per user. Include grouping options by counterparty and warehouse."

• Original: "The system must handle large data volumes correctly."

Concrete: "Loading 10,000 documents totaling 500,000 rows must complete in under 15 minutes. Memory usage capped at 2 GB."

• Original: "All screen forms must follow a consistent style."

Concrete: "In all document forms, 'Save' and 'Post' buttons are in the top-right corner. Button color: green (R=0, G=128, B=0). Font: Arial, 10pt."

Quality Criteria and Agent Architecture

We defined a checklist for spec evaluation covering business goals, scenario completeness, clarity, testability, consistency, and feasibility. These formed the basis for the agent's prompts and logic.

The solution uses a hybrid model combining Retrieval-Augmented Generation (RAG) with decision tree classification.

• RAG Component handles contextual search. When reviewing a new requirement, the agent pulls semantically similar snippets from the knowledge base (90 labeled specs), recalling common errors and their contexts.
• Decision Trees categorize errors using structured features. The LLM agent first parses raw text into structured features (e.g., JSON), which feeds into the classifier.

Technical Implementation of Analysis

Checking a single requirement follows these steps:

• Feature Extraction: An LLM-based analyzer converts the text into a feature vector. For example:

* has_numbers: 1 (includes numeric specifics) or 0.

* has_negative_keywords: 1 (mentions "if," "else," "error") or 0.

* boundary_conditions_mentioned: 1 (covers edge cases) or 0.

* stopword_score: weight of subjective markers ("flexible," "intuitive").

• Classification: The structured feature vector goes to a decision tree model (e.g., from Scikit-learn), assigning it to an error category or marking it as solid.
• Report Generation: The agent produces a detailed comment flagging the issue type, its location in the text, and fix recommendations drawn from knowledge base analogies.

This division—LLM for semantic understanding and data extraction, classifier for rigid logic—ensures transparency and control.

Key Takeaways

• Data Quality Is Everything. Without a representative, well-labeled dataset (at least dozens of examples), effective training is impossible.
• Hybrid Beats Pure LLM. Pairing RAG for context with decision trees for precise classification delivers more reliable, explainable results.
• Prioritize Testability. A great spec allows unambiguous test cases. All requirements must be measurable and specific.
• Automate the Drudgery. The agent doesn't replace analysts but acts as a fresh set of eyes, systematically spotting common errors to save time and cut human error.
• Keep Learning. Regularly update the knowledge base with new specs and reviews to boost accuracy and adapt to fresh project types.

— Editorial Team