A Guide to Robots.txt: Best Practices for SEO

You are currently viewing A Guide to Robots.txt: Best Practices for SEO

Learn how to use robots.txt to block unnecessary URLs and boost your website's SEO performance.

Why Robots.txt Matters for SEO The robots.txt file controls what search engines can and cannot crawl on your website. Done correctly, it helps Google and other crawlers focus on important pages, improving efficiency and saving resources. Done wrong, it can cause pages to be missed in search results.

Google recommends using robots.txt to block unnecessary pages like login, checkout, or add-to-cart URLs.

What is Robots.txt?

  • Robots.txt is a text file located in the root directory of your site. It tells search engine crawlers what content to crawl or ignore. Basic directives include:
    • User-agent: Specifies which crawler the rule applies to.
    • Disallow: Blocks specific URLs from being crawled.
    • Allow: Allows crawling of certain URLs, even if their directory is blocked.
    • Sitemap: Indicates the location of your XML Sitemap for easier crawling.
  • Order of Precedence
    • Crawlers apply the most specific rule first, followed by the least restrictive rule. For example:
      • If “Disallow: /downloads/” and “Allow: /downloads/free/” both exist, only “/downloads/free/” will be crawled.

Why Robots.txt is Important for SEO

Blocking unnecessary pages helps Google save resources and focus on valuable content, ensuring efficient crawling.

When to Use Robots.txt

  • Use robots.txt to block:
    • Internal search pages (e.g., URLs with query parameters).
    • Faceted navigation pages (filtering options that create duplicate content).
    • Action pages (like add to cart or wishlist).
    • Private pages (login, account pages).
    • Irrelevant JavaScript files (like tracking scripts).

Examples of Robots.txt Use

Block Internal Search Pages: Example:

makefile
Copy code
User-agent: *

Disallow: *s=*

Block Faceted Navigation: Example:

makefile
Copy code
User-agent: *

Disallow: *sortby=*

Disallow: *color=*

Disallow: *price=*

Block PDFs: Example:

makefile
Copy code
User-agent: *

Disallow: /*.pdf$

Troubleshooting Robots.txt

Validate your robots.txt file using Google Search Console or their Robots.txt parser to avoid mistakes.

Centralised Management

Manage robots.txt centrally by redirecting subdomains’ robots.txt to the main domain.

Optimising robots.txt ensures search engines focus on important content, improving your crawl budget and reducing wasted resources. Regularly validate and update your robots.txt file to avoid issues.