Robots.txt and sitemap.xml: indexing setup

Configuring robots.txt and sitemap.xml for Optimal Crawling

A single character in robots.txt can block key sections of a website, leading to a traffic drop. On a recent project, a product catalog directory became inaccessible to search engine crawlers due to a file error—organic traffic decreased by a third in two weeks. Robots.txt manages crawler access, while sitemap.xml sets scanning priorities. These files determine indexing efficiency in Google and Yandex.

Directives and Syntax of robots.txt

Robots.txt is placed in the site's root and contains instructions for crawlers: Googlebot, YandexBot, and others. Key directives:

• User-agent: specifies the bot. User-agent: * — for all, User-agent: Googlebot — only Google.
• Disallow: prohibits a path. Disallow: /admin/ blocks a directory, Disallow: /search? — URLs with search parameters.
• Allow: permits within a blocked directory. Example:

User-agent: *
Disallow: /catalog/
Allow: /catalog/main-page/

• Sitemap: link to the sitemap, outside User-agent blocks.

A slash at the end of Disallow indicates a directory. Without a slash—any URL with the prefix. Robots.txt does not block indexing entirely: external links can add a page to the index with a warning. For noindex, use meta tags or authorization.

Basic robots.txt example:

User-agent: *
Disallow: /admin/
Disallow: /cart/
Disallow: /search?
Disallow: /*?sort=
Disallow: /*?filter=
Allow: /
Sitemap: https://site.ru/sitemap.xml

This template blocks service sections and filter parameters.

Differences in Directive Support

| Directive | Google | Yandex |

|---------------|--------|--------|

| User-agent | ✅ | ✅ |

| Disallow | ✅ | ✅ |

| Allow | ✅ | ✅ |

| Sitemap | ✅ | ✅ |

| Host | ❌ | ✅ |

| Crawl-delay | ❌ | ✅ |

| Clean-param | ❌ | ✅ |

The Host directive for Yandex sets the primary mirror: Host: https://site.ru. Google ignores it—use Search Console.

Placement and Verification of robots.txt

• Create the robots.txt file in a text editor (VS Code, Notepad++).
• Upload to the root: https://site.ru/robots.txt.
• For CMS:

- WordPress: Yoast SEO or FTP.

- Bitrix: admin panel file manager.

- Tilda/Wix: panel settings.

• Check accessibility: open the URL in a browser (expect text, not 404/500).
• Test: Google Search Console (Crawling), Yandex.Webmaster (Robots.txt Analysis).

Syntax errors lead to ignored lines or unpredictable behavior.

Structure of sitemap.xml

Sitemap.xml lists important URLs for crawl prioritization. Useful for sites with >100 pages, frequent updates, or orphan pages.

Basic structure:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://site.ru/</loc>
    <lastmod>2026-01-15</lastmod>
    <changefreq>daily</changefreq>
    <priority>1.0</priority>
  </url>
</urlset>

Tags:

• loc: mandatory URL.
• lastmod: actual modification date.
• changefreq: frequency (daily/weekly; Google ignores, Yandex considers).
• priority: 0.0–1.0 for order.

Limitations: 50,000 URLs, 50 MB. For more—sitemap-index:

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap>
    <loc>https://site.ru/sitemap-pages.xml</loc>
  </sitemap>
</sitemapindex>

Generating sitemap.xml

• CMS/plugins: WordPress (Yoast), Bitrix (Marketing). Auto-updates, but check for unnecessary URLs.
• Online generators: for static sites, manual updates.
• Scripts: for SPAs (Next.js + next-sitemap). Generates during build.

Place in the root or specify in robots.txt.

File Integration

• Add to robots.txt: Sitemap: https://site.ru/sitemap.xml.
• Submit to Google Search Console (Sitemaps) and Yandex.Webmaster (Indexing).
• Verify: URLs in sitemap should not be in Disallow, status 200 OK.

Advanced Sitemaps

For media:

Images:

<url>
  <loc>https://site.ru/article/</loc>
  <image:image>
    <image:loc>https://site.ru/images/photo.jpg</image:loc>
    <image:title>Description</image:title>
  </image:image>
</url>

Video (fragment):

<url>
  <loc>https://site.ru/video-page/</loc>
  <video:video>
    <video:thumbnail_loc>https://site.ru/thumb.jpg</video:thumbnail_loc>
  </video:video>
</url>

Improves indexing in Google Images and Yandex.Images.

Key Takeaways

• Robots.txt saves crawl budget but does not replace noindex.
• Sitemap speeds up discovery of new pages and structures the site.
• Always check conflicts: URLs in sitemap must be accessible.
• Consider Google/Yandex differences in directives (Host, Clean-param).
• For large sites, use sitemap-index and media extensions.

— Editorial Team