Restoration of ruGPT-3 XL for transformers

Converting and Restoring ruGPT-3 XL for Modern Frameworks

Developers faced challenges running the outdated language model ai-forever/rugpt3xl (1.3B parameters) from 2021. Originally trained from scratch on a Russian corpus as a GPT-2 variant, it's stored in Megatron-LM checkpoint format requiring PyTorch 1.7 and transformers 3.5. Converting it to Hugging Face format unlocks compatibility with modern inference and fine-tuning tools.

The process involves extracting weights from mp_rank_00_model_states.pt, mapping them to the GPT2Model structure, and creating custom classes for transformers.

Weight Structure: Megatron-LM vs HuggingFace

Original weights use a fused QKV projection [6144, 2048], which must be split into separate Q, K, V matrices of size [2048, 2048]. Full mapping:

| Megatron-LM | HuggingFace |

|-------------|-------------|

| word_embeddings.weight | model.embed_tokens.weight |

| position_embeddings.weight | model.embed_positions.weight |

| transformer.layers.{i}.input_layernorm. | model.layers.{i}.input_layernorm. |

| transformer.layers.{i}.attention.query_key_value.weight | model.layers.{i}.self_attn.{q,k,v}_proj.weight |

| transformer.layers.{i}.attention.query_key_value.bias | model.layers.{i}.self_attn.{q,k,v}_proj.bias |

| transformer.layers.{i}.attention.dense. | model.layers.{i}.self_attn.o_proj. |

| transformer.layers.{i}.post_attention_layernorm. | model.layers.{i}.post_attention_layernorm. |

| transformer.layers.{i}.mlp.dense_h_to_4h. | model.layers.{i}.mlp.up_proj. |

| transformer.layers.{i}.mlp.dense_4h_to_h. | model.layers.{i}.mlp.down_proj. |

| transformer.final_layernorm. | model.norm. |

| - | lm_head.weight (copy of embed_tokens) |

The convert.py script transforms the checkpoint into safetensors compatible with transformers.

Custom Model Classes

New classes were built without dependencies on Megatron-LM or DeepSpeed:

• RuGPT3XLConfig: inherits PretrainedConfig with parameters (vocab_size=50264, hidden_size=2048, num_layers=24).
• RuGPT3XLAttention: multi-head attention with separate Q/K/V and DynamicCache.
• RuGPT3XLMLP: MLP block (up_proj → GELU → down_proj).
• RuGPT3XLDecoderLayer: decoder layer (pre-LN → attention → post-LN → MLP).
• RuGPT3XLModel: embeddings + 24 layers + final LN.
• RuGPT3XLForCausalLM: includes lm_head.

Key improvements:

• Standard forward() for SFTTrainer/LoRA.
• KV-cache via DynamicCache.
• Gradient checkpointing.
• Runs on CPU/GPU using device_map="auto".

The model is now available as evilfreelancer/ruGPT3XL on Hugging Face.

Testing Results

Generation on RTX 4090 (float16):

• Average speed: 66.7 tokens/sec (batch_size=1).
• Maintains context on long prompts.
• Generates accurate text (recipes, historical narratives).

MERA benchmark (overall score: 0.198):

• PARus (common sense): 0.500
• ruHateSpeech: 0.558
• BPS (code/math): 0.528
• RWSD (reasoning): 0.488
• ruTiE (dialogue): 0.502
• ruMMLU: 0.252

Math and code performance is close to baseline—expected for a base model without instruction tuning.

Converting to GGUF for llama.cpp

After HF format conversion, the convert_hf_to_gguf.py script was adapted for ruGPT-3 XL. The model runs in llama.cpp without Megatron dependencies, supporting KV-cache and modern inference.

This enables fine-tuning on custom datasets via LoRA or SFT, integration into pipelines, and testing on edge devices.

Key Takeaways

• Compatibility: Works with transformers ≥4.x, no legacy stack required.
• Performance: 66.7 t/s on RTX 4090, KV-cache support for generation.
• Trainability: Ready for LoRA/SFTTrainer with gradient checkpointing.
• Benchmarks: Strong in common sense and hate speech detection (>0.5), weaker in math/code.
• Formats: HF + GGUF for broad ecosystem support.

— Editorial Team