Skip to content
Request access
Crawler

RecrawlBot

Last updated: November 12, 2026

RecrawlBot is the web crawler of recrawl (recrawl.co), an SEO platform run by Crawl Compass. It uses one user agent for two jobs: site audits and link checks. This page explains what it does, how to recognise it, and how to control it on your site.

At a glance

  • User agent: Mozilla/5.0 (compatible; RecrawlBot/1.0; +https://recrawl.co/bot)
  • robots.txt token: RecrawlBot
  • Obeys robots.txt: yes, both Allow and Disallow rules
  • Obeys Crawl-delay: yes
  • Speed: it adapts to your site: it follows your Crawl-delay and slows down when your server is busy
  • Runs JavaScript: no
  • IP addresses: recrawl.co/bot/ips.json (JSON) and recrawl.co/bot/ips.txt (plain text). RecrawlBot only ever crawls from these addresses.
  • Reverse DNS: every address resolves to a name ending in .bot.recrawl.co (today crawl-01.bot.recrawl.co and crawl-02.bot.recrawl.co), and that name resolves back to the same address
  • Contact: bot@recrawl.co

What it does

Site audits

recrawl customers audit websites they own or manage. RecrawlBot visits the site, follows the links between its pages and checks them for technical SEO issues: status codes, redirects, broken links, titles and descriptions, headings, canonical tags and robots meta tags. To check that a page loads correctly, it may also request the page's images, scripts and stylesheets. An audit runs when a customer starts one or schedules it.

Link checks

When a page links to a recrawl customer's website, RecrawlBot revisits that one page regularly to confirm the link is still there and to read its anchor text and rel attributes (nofollow, sponsored, ugc). It requests only that page, not the rest of your site.

What it does not do

It does not log in, fill in forms, solve CAPTCHAs, add items to carts or click anything that changes a site. It does not run JavaScript, so it never loads ads or tracking scripts.

Why is RecrawlBot visiting my site?

Usually for one of two reasons:

  • A recrawl customer is auditing a website they own or manage.
  • A page on your site links to a recrawl customer's website, and RecrawlBot is checking that the link is still there.

How to check a request is really RecrawlBot

Anyone can copy a user agent, so check the address as well:

  1. 1.The request's IP address is in our IP list.
  2. 2.Or use reverse DNS: look up the IP address (for example host 157.245.51.125). The name must end in .bot.recrawl.co (here crawl-01.bot.recrawl.co). Then look up that name (host crawl-01.bot.recrawl.co); it must return the same IP address.

If either check fails, the request did not come from RecrawlBot. Please tell us about it at bot@recrawl.co, with the IP address and time.

IP addresses

RecrawlBot crawls only from the addresses in ips.json (the same format Google uses for its crawlers) and ips.txt (one address per line). Today:

157.245.51.125
167.71.215.78

These addresses are used only by recrawl and are never shared with anyone else. When we add an address, it appears in both lists before RecrawlBot uses it, and the creationTime in ips.json changes. If you allow RecrawlBot through a firewall or CDN, read the list from these files rather than copying it once.

How it reads robots.txt

  • It reads /robots.txt before crawling a site and follows the group for RecrawlBot (matched without regard to upper or lower case). If there is no such group, it follows the * group.
  • When an Allow and a Disallow rule both match a URL, the more specific (longer) rule wins, as in the robots.txt standard (RFC 9309).
  • It keeps a copy of your robots.txt for up to a day, so changes are picked up within a day.
  • If robots.txt does not exist (a 4xx answer such as 404), it may crawl the whole site.
  • If robots.txt cannot be read because your server returns an error (5xx), it does not crawl your site until it can be read.
  • Crawl-delay is read as the minimum number of seconds between two requests.
  • robots.txt applies to each host separately: www.example.com and shop.example.com each have their own.

How it behaves on your site

  • It adapts its speed to your site, follows your Crawl-delay and spreads its requests out rather than sending bursts.
  • It slows down when your site answers 429 (Too Many Requests) or 503 (Service Unavailable), and stops when your site keeps refusing.
  • If your site blocks it, it stops. It never retries from another address or with another user agent.
  • Every request comes from the addresses above with the user agent above. It never uses proxies, residential IP addresses or browser disguises.
  • noindex and nofollow robots meta tags are reported in audits but do not keep RecrawlBot from reading a page. To keep it out, use robots.txt.

How to slow it down or block it

To set a minimum wait between two requests (in seconds), add this to your robots.txt:

User-agent: RecrawlBot
Crawl-delay: 10

To keep it out of part of your site:

User-agent: RecrawlBot
Disallow: /private/

To block it from your whole site:

User-agent: RecrawlBot
Disallow: /

To slow it down for a short while, for example during maintenance, answer its requests with 429 or 503. It backs off by itself.

Common questions

  • Does blocking RecrawlBot affect my rankings in Google or other search engines? No. RecrawlBot is not a search engine crawler, and blocking it only means recrawl cannot check your pages.
  • Will it overload my server? It is built not to: it follows your Crawl-delay, slows down when your server answers 429 or 503, and stops when your site keeps refusing. If you still see too much traffic, set a Crawl-delay or write to us.
  • I blocked it. Why do I still see a request? It reads your robots.txt again within a day, and it needs to fetch robots.txt itself to see the change. If requests continue after a day, write to us with the IP address and time.
  • Can I see what it found on my site? If you own the site, you can audit it in recrawl.

What we keep

RecrawlBot keeps facts about the public pages it visits: the status code, title and headings, the links between pages, and technical details such as the page's server and software. For a limited time it also keeps a copy of each page, so changes can be compared. We do not look for personal information, we do not sell page content, and we do not use it to train generative AI models. recrawl uses these facts in its SEO tools and research. To have what we hold about your site deleted, write to us.

Site owners who use recrawl

If you verified your site in Google Search Console and connected it to recrawl, you can let RecrawlBot audit that site faster, or ignore robots.txt on that site only. This never applies to anyone else's site.

Contact

Questions, problems, suspicious traffic, or a request to stop crawling your site or to delete what we hold about it: bot@recrawl.co. The fastest way to slow or stop RecrawlBot is robots.txt, which it reads before crawling a site.