#web crawling
1 articles tagged web crawling. At twelve articles this topic starts lifting the whole site in search.
Timeline
All articles
-
Tech · 3 min readWhat is robots.txt — blocking AI crawlers in one line, and why it can't be enforced
Definition — a root-level text file listing allowed and disallowed paths; standardized as RFC 9309 in 2022