Fine-tuning Mistral 7B for Russian: From LoRA to Full Fine-tuning for Dialogue Summarization
Mistral 7B performs poorly in Russian: errors in declensions, ignoring prompts, and loss of meaning in long dialogues. The AiGenda team decided to fine-tune the model for automating meeting minutes. The task involves speech recognition, speaker identification, topic detection, and summarization using an LLM with up to 7B parameters and a context of 8192 tokens.
The architecture is designed for scalability, security, and fault tolerance. The backend routes requests via RabbitMQ to the ML service. The ML Controller manages models and interacts with a vector database for embeddings. Data is isolated, with S3 storage and meeting recording via the OpenStack API.
ML System Architecture
Processing pipeline: interface → Backend → RabbitMQ → ML → response. The ML Controller distributes tasks between Whisper (speech), pyannote/ResNet34 (speakers), embeddings for topics, and Mistral for text.
Key components:
- RabbitMQ queue: Batch processing, reducing latency.
- ML Controller: Routing, adding/replacing models.
- Vector database: Speaker and text embeddings.
- Meeting Controller: Dynamic resources for recording.
- Monitoring: Grafana, Sentry.
This ensures flexibility and security without direct access to models.
LoRA Failure and Shift to Full Fine-tuning
LoRA smoothed errors but did not fix weak sentence logic due to a shortage of Russian data in the base model. We switched to full fine-tuning.
We compiled a dataset of 5.35 billion tokens: texts, dialogues, instructions, machine translations. Training on transformers (Hugging Face) with optimizations:
- Mixed Precision, bfloat16.
- Padding-free sampler (10x speedup).
- Flash Attention 2 (saving VRAM).
- ZeRO-2 with offloading to CPU.
Two Nvidia RTX A6000 (48 GB) fit the budget. Logs in wandb.ai.
Evaluating the Language Model on MERA
Tested on Ubuntu 20.04, RTX 4090, 16 Ice Lake cores, 32 GB RAM. The fine-tuned model outperformed base Mistral 7B, GigaChat, and MTS Chat on:
- LCS.
- PARus.
- SimpleAR.
- RuHumanEval.
- ruHH.
In practice: reduced errors in dialogues, information extraction within 8192 tokens with low hallucinations.
Key takeaways:
- Full fine-tuning is critical for languages with limited data in pre-trained models.
- Optimizations (Flash Attention, ZeRO-2) enable training 7B models on consumer hardware.
- MERA is a reliable benchmark for Russian.
- A context of 8192 tokens is sufficient for meeting dialogues.
- Integration with the architecture ensures production readiness.
Task-specific Fine-tuning and Whisper
Final LoRA on a dialogue dataset (summarization, action items, topics). Annotations: manual + GPT-4 API. BertScore F1 = 0.93 for summarization.
Whisper fine-tuned on 50%+ Russian: lectures, conferences, interviews. Pseudo-labeling + dictionary checks (e.g., "ETMO" → "ITMO"). Reduced WER on noisy meeting audio.
Future Development
Plans: context up to 128k tokens, new architectures. Collecting feedback for iterations.
— Editorial Team
No comments yet.