AI assistant for calls: from prototype to production

Building a Real-Time Voice AI Assistant: Backend Team's Journey

The voice AI assistant 'Suffler' supports sales managers during live calls. The system classifies services, detects objections, uses RAG to retrieve relevant data, and generates real-time text prompts. Input comes as text segments from the Voximplant API transcription service. A mandatory latency window of 1.5–2 seconds is enforced. Tech stack: Python, FastAPI, PostgreSQL, fine-tuned BERT models, and local Qwen 8B. On-premise deployment was required for security compliance.

A team of 12 backend developers with no prior ML experience mastered the fundamentals in six months and successfully launched a pilot version.

Proof-of-Concept in Three Weeks

Within the first three weeks, the team delivered a working prototype:

• Processing call transcripts as input data.
• Two BERT classifiers: one for 15+ services and another for 15+ objection types, trained on 1,500 annotated examples.
• FAISS vector databases containing service descriptions and scripts.
• Integration via prompts using GPT.
• A Django-based interface to display real-time suggestions.

The prototype showed a 10–15 second delay but convinced stakeholders of the idea’s viability.

Architectural Evolution: Cutting Complexity

The initial monolithic design included a custom Whisper-based audio pipeline, multiple classifiers, LLM fine-tuning, a vector database with feedback loops, and estimated development time of 12–18 months with high risk.

Abandoning fine-tuning. Instead of retraining Qwen or Llama on sales scripts, the team adopted RAG: the model pulls relevant context directly from a knowledge base on demand. This reduces hallucinations and eliminates extensive annotation efforts.

Using a ready-made transcriber. Voximplant delivers text segments via REST, saving two months of building an internal audio pipeline.

Simplifying classifiers. A multi-class objection detector was replaced with a binary classifier (objection present/absent). The LLM handles detailed analysis during generation. Running inference on a GPU server significantly improved speed.

JSON search over vector DBs. Knowledge bases (services, scripts) are stored as structured JSON in memory. Faster than FAISS or Weaviate, with no network latency.

Local LLM instead of cloud. Cloud APIs like GPT or DeepSeek introduced delays of 7–20 seconds and were blocked by security policies. Running Qwen 8B locally on a GPU server achieved ~2-second response times with concise outputs.

Lightweight interface. A simple Django page displaying prompts replaced complex CRM integration.

Final Real-Time Processing Pipeline

The system works as follows:

• Receive text from Voximplant.
• Binary BERT classification (service + objection).
• RAG lookup in an in-memory JSON database.
• Generate prompt for Qwen 8B.
• Display suggestion in the UI.

Total latency stays under 2 seconds.

Data Labeling: Embracing Pragmatism

Labeling 200 dialogues took weeks due to the need for contextual understanding. Sales managers labeled faster but weren’t available at scale.

Solution: Focus on binary detection. The LLM handles objection details during generation. Iteration cycles were drastically shortened.

Latency as a Hard Constraint

Live conversation demands sub-2-second response times. Cloud APIs proved unreliable. The local Qwen 8B (vs. 32B) sacrifices some detail for speed — a necessary trade-off.

Security and Isolation

Full on-premise workflow:

• Voximplant transcription (only external dependency).
• Internal processing and classification.
• Local LLM-generated responses.

No PII masking: avoided 200–500 ms overhead, reduced risk of data distortion, and saved a month of dev work.

Key Takeaways

• RAG over fine-tuning accelerates launch and minimizes labeling effort.
• Binary classification + LLM simplifies objection detection without quality loss.
• Local Qwen 8B strikes the right balance between speed and accuracy for real-time use.
• JSON search outperforms vector databases for small, structured knowledge bases.
• On-premise deployment solves both security and latency challenges simultaneously.

— Editorial Team