#robots.txt
1 articles tagged robots.txt. At twelve articles this topic starts lifting the whole site in search.
Timeline
All articles
-
Tech · 3 min readWhat is robots.txt — blocking AI crawlers in one line, and why it can't be enforced
Definition — a root-level text file listing allowed and disallowed paths; standardized as RFC 9309 in 2022