Robots.txt

What is Robots.txt?

Robots.txt tells crawlers which paths they may request. It can reduce crawl load but does not reliably hide content; sensitive pages need authentication or noindex.

Example: Disallowing `/cart/` might reduce noise, but accidentally disallowing `/products/` blocks discovery of your entire catalog.

Why Robots.txt matters

Robots.txt matters because incorrect rules can accidentally block crawling of entire sections; it’s a high-impact, low-visibility file during launches.

How Robots.txt works in digital marketing

Audit rules after migrations, avoid blocking CSS/JS needed for rendering, and pair with meta robots for true index control.