Build a valid robots.txt with custom rules per crawler, then download it ready to drop in your site's root directory.
robots.txt works through user-agent groups, each listing Allow and Disallow paths that apply to the crawlers matching that group's name — and when a path matches more than one rule, the longest matching path wins regardless of whether it's an Allow or a Disallow, not whichever rule appears first. This generator builds those groups for you, includes presets for common setups like WordPress's admin paths or blocking specific AI crawlers by name, and appends a Sitemap declaration and optional Crawl-delay.
Not directly — it only tells well-behaved crawlers not to fetch a path. If a disallowed page is linked from elsewhere, a search engine can still know it exists and list it in results (usually without a description, since it never crawled the content). To actually prevent indexing, you need a noindex meta tag or header on the page itself, which requires the crawler to fetch the page to see it.
It asks a crawler to wait a given number of seconds between requests to your server, which can help if aggressive crawling is straining your hosting. Support for it is inconsistent — Google's crawler ignores it entirely and paces itself using its own algorithm — so it's mainly honored by smaller or more literal crawlers.
No — it's a voluntary convention, not an enforced restriction. Major search engines generally respect it, but nothing stops a scraper or bot from ignoring the file entirely, and since robots.txt is a publicly accessible file at your domain's root, it also can't be used to hide anything sensitive — it's just a set of instructions crawlers can choose to follow.