Articles by tag: rugpt3xl

ruGPT3XL with 8k context: sparse attention

Expansion of ruGPT3XL to 8k tokens via sparse attention and fine-tuning. PPL 13.00 on gazeta, Triton/SDPA optimization ×1.85. Instructions for developers.

Restoration of ruGPT-3 XL for transformers

Learn how to convert legacy ruGPT-3 XL to HF format for fine-tuning and inference. Weight structure, custom classes, MERA benchmarks. Instructions for developers.