GrowthGadgetAI video walkthrough
How to Use the Robots.txt Tester (Free)
Paste a robots.txt and test whether a given path is allowed for a user-agent. A short video walkthrough of the free Robots.txt Tester.
Watch: how to use the Robots.txt Tester in 32 seconds
Video transcript
Robots.txt Tester. Paste a robots.txt and test whether a given path is allowed for a user-agent. First, paste your robots.txt, or enter its address to fetch it. Then, type the path you want to test, for example /admin/secret. You get: the contents of a site's robots.txt and whether a URL is allowed or blocked for a crawler. Free, and it runs in your browser. Try it free at growthgadgetai.com/tools.
What this video covers
The Robots.txt Tester is a free online tool you can use to paste a robots.txt and test whether a given path is allowed for a user-agent. Pasted content is processed in your browser; the optional fetch-from-URL mode sends that public URL to our server and is rate-limited.
In this video
- 0:00 Robots.txt Tester. Paste a robots.txt and test whether a given path is allowed for a…
- 0:08 First, paste your robots.txt, or enter its address to fetch it.
- 0:12 Then, type the path you want to test, for example /admin/secret.
- 0:18 You get: the contents of a site's robots.txt and whether a URL is allowed or blocked for…
- 0:24 Free, and it runs in your browser. Try it free at growthgadgetai.com/tools.
Steps shown
- Paste your robots.txt, or enter its address to fetch it
- Type the path you want to test, for example /admin/secret, and the user-agent (use * for the default group)
- Read Allowed or Blocked and the rule that matched, then edit the file and test again until the result is what you intend
Frequently asked questions
How do I test a robots.txt file?
Paste your robots.txt, enter a URL path and choose a user-agent such as Googlebot. The tester reports whether that path is allowed or blocked and which rule decided it.
Is my data stored?
The URL is sent to our server for a public-page request. Copy the result before leaving the page. Do not submit private URLs, passwords or access tokens.
Which rule wins when Allow and Disallow both match a path?
In this tester the longest matching pattern wins, which mirrors how major search engines choose the most specific rule. If two matching patterns are the same length, the one listed first is used here, so avoid writing conflicting rules of equal length.
More video walkthroughs for utilities
Open the related free tool