About the AiReadinessBot crawler

Who we are

We are the crawler of a site-checking service. We come for one of two reasons: the owner asked us to check the site, or someone asked for a report on a public address. We introduce ourselves like this:

AiReadinessBot/1.0 (+https://ai-readiness.local/bot)

What we ask for

• the home page — twice: once as a crawler, once as a browser, to see whether they get the same thing
• robots.txt, sitemap.xml, llms.txt and the favicon
• up to 8 inner pages found in your own links
• up to 15 links, to see which are broken
• up to 8 images and up to 5 addresses from the sitemap
• six well-known addresses that should not be public: .git/config, .env, phpinfo.php, backup.zip, dump.sql, database.sql

We only read. We do not submit forms, do not log in, do not change anything and do not try any password. Every request is a GET.

How often

• One check is at most about 60 requests, usually fewer.
• Your site is not checked more than 6 times an hour, no matter who asks or from how many addresses. If you subscribed and asked us to watch it, the ceiling is 30 — you are then the one loading your own server, deliberately.
• A site under monitoring is rechecked no more often than once every 20 hours.
• One visitor cannot start more than 5 checks a minute and 30 an hour.
• A repeated request within 5 minutes gets the existing report instead of a new visit to your server.

How to shut us out

Add this to your robots.txt:

User-agent: AiReadinessBot
Disallow: /

A rule that names us is binding for us always, without exception — we check robots.txt before every visit.

One honest exception: a blanket rule for all robots (User-agent: *) does not stop us on a site whose owner has an active subscription with us and asked us to watch it. A rule naming AiReadinessBot stops us there too.

If something went wrong

The address will appear together with the domain. Until then: if our crawler is causing you trouble, the robots.txt rule above stops it immediately.