Articles by tag: sparse-attention

XML tags in prompts for AI models

Learn how XML tags control transformer attention in prompts. Comparisons, code examples, and patterns for middle/senior developers. Optimize sessions with long context.

ruGPT3XL with 8k context: sparse attention

Expansion of ruGPT3XL to 8k tokens via sparse attention and fine-tuning. PPL 13.00 on gazeta, Triton/SDPA optimization ×1.85. Instructions for developers.