Why was ruGPT3XL PPL 50.1?

Dense attention replaced sparse. Weights optimized for sparse mask, dense causes degradation after 128 tokens.

How did they expand context to 8k?

Tiling Learned APE, mixed dataset (60% long), stepwise fine-tuning 2k→4k→8k. PPL 13.00 on gazeta.

How to speed up sparse attention?

SDPA + torch.compile with Inductor: ×1.85 to eager mode (~11600 tok/s on RTX 4090).

ruGPT3XL with 8k context: sparse attention

The article describes the restoration of sparse attention in ruGPT3XL and expansion of context to 8k tokens. Tiling positional embeddings used, mixed gazeta dataset and stepwise fine-tuning. Achieved PPL 13.00 at 8192 tokens with minimal regression on base context.

By Sudonull Editorial Team • Published 2026-04-14 • Updated 2026-04-14

Extending ruGPT3XL Context to 8k Tokens: Sparse Attention and Fine-Tuning

Developers successfully adapted ruGPT3XL from Megatron-LM to HuggingFace format with GGUF support in llama.cpp. Testing revealed a degradation in Perplexity (PPL) to 50.1 on the gazeta dataset instead of the expected 12.05. The cause was the replacement of sparse attention with dense nn.MultiheadAttention from GPT-2.

Restoring the original mechanism from Megatron-LM reduced PPL to 11.68. Correlation with the claimed metrics of the original model reached R=0.93.

PPL Comparison on gazeta

| Model | PPL (gazeta) |

|----------------|--------------|

| ruGPT3Small | — |

| ruGPT3Medium | 25.2 |

| ruGPT3Large | 16.8 |

| ruGPT3XL (dense)| 50.1 |

| ruGPT3XL (sparse)| 11.68 |

The Sparse Attention Mechanism in ruGPT3XL

Sparse attention uses an alternating pattern from the paper "Generating Long Sequences with Sparse Transformers" (arXiv:1904.10509):

Even layers (0,2,4…): block-sparse attention. Each token sees a local window of 128 tokens + global blocks at regular intervals. Different heads use different global positions.
Odd layers (1,3,5…): standard causal dense attention.

This ensures nearly linear memory growth instead of quadratic. With a 4x context increase, memory for KV-cache and activations grows 3-4 times.

In llama.cpp, the LLM_ARCH_RUGPT3XL architecture (PR #21161) was added with full sparse attention support. A mask error for batches >1 during training was fixed.

Optimization via Triton and SDPA

Implementation via matmul + softmax + matmul (eager mode) yields ~6280 tok/s on an RTX 4090.

Speedup:

F.scaled_dot_product_attention (SDPA): +40% (~8800 tok/s).
SDPA with increased batch size (5×2048): another +25%.
torch.compile with Inductor: ×1.85 over baseline (~11600 tok/s), generates Triton kernels.

Strategy for Extending Context to 8k

Architectural limitations:

Learned Absolute Positional Embeddings (nn.Embedding(2048,2048)) do not extrapolate.
The sparse attention grid depends on max_position_embeddings // sparse_block_size.

Fine-Tuning Principles

Tiling positional embeddings: Cyclic duplication of 0-2047 for new positions. Maintains stability on short context.
Mixed dataset: 60% long examples (gazeta via EOS), 40% short chunks.
Stepwise extension: 2k→4k (2.6h), then 4k→8k (3.9h) on RTX 4090.

Parameters: gazeta train, 3 epochs, lr=5e-6 cosine decay, bfloat16, gradient checkpointing. OOM resolved via PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (38.5 GB peak).

Model: evilfreelancer/ruGPT3XL-8k.

PPL Results on gazeta test

| Model | 2048 | 4096 | 8192 |

|----------------|------|------|------|

| ruGPT3XL base | 11.68| — | — |

| ruGPT3XL-4k | 11.75| 12.04| — |

| ruGPT3XL-8k | 11.77| 11.99| 13.00 |

Minimal regression at 2k, model retains quality.

Key Takeaways

Sparse attention is critical: dense implementation yields PPL 50.1 instead of 11.68.
Tiling APE is effective: enables fine-tuning without degrading base context.
8k is feasible on RTX 4090: 38.5 GB with expandable_segments.
Stepwise approach minimizes overfitting: 2k→4k→8k in 6.5 hours.
Triton/SDPA speed up by 1.85x: torch.compile generates optimized kernels.

— Editorial Team

ruGPT3XL with 8k context: sparse attention