SerpelBot

What is SerpelBot?

SerpelBot is the crawler of Serpel. It fetches pages when a Serpel user crawls a website, for example for a site audit of their project. It reads pages, follows internal links and reports what it finds to that Serpel user. It doesn’t submit forms and doesn’t sign in.

This is the user agent that you will find in your server logs:

Mozilla/5.0 (Linux; Android 14; Pixel 8) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0.0.0 Mobile Safari/537.36 (compatible; SerpelBot/0.1; +https://serpel.app/bot)

By default SerpelBot identifies as a mobile browser, the way search engines evaluate websites today. The token SerpelBot/0.1 and the address of the crawler page come last. It also sends an Accept-Language header that matches the language and country of the project, for example de-DE,de;q=0.9,en;q=0.5 for a German project, so websites that choose a language by header answer in the right one.

Serpel describes the crawler for website owners on a public page, serpel.app/bot. Signed-in users find a pointer to it under Settings → Crawler info. The product side of the crawl, from the start button to the results, is explained in the site audit guide, and the site audit feature page summarises what the crawl is for.

How does SerpelBot treat your robots.txt?

SerpelBot fetches /robots.txt before any page of a crawl and applies it to every page it crawls. It looks for a group that starts with User-agent: SerpelBot, matched without regard to case. If there is no such group, it uses the * group. The groups are never merged: as soon as a SerpelBot group exists, the * group no longer applies to SerpelBot, even if it only contains an empty Disallow: line. Checks of links and embedded resources, the sitemaps and /llms.txt are not filtered by your rules.

Inside the group, the longest matching rule wins, and when an Allow and a Disallow rule are equally long, Allow wins. The characters * and $ work as wildcards in paths. Which answer SerpelBot gets for the file itself matters as much as its content:

How fast does SerpelBot crawl?

For a site audit crawl, these are the current settings:

A Serpel user can’t raise these values. If your server needs more room, add a Crawl-delay line for SerpelBot, as shown below.

What does SerpelBot fetch?

• The HTML of your pages. For other responses, such as a PDF that you link to, it reads only the headers.
• Your robots.txt, your sitemaps and /llms.txt.
• The external links on your pages and the images, scripts and stylesheets they load, with HEAD requests.
• One request to a made-up address that starts with /serpel-pruefung-, followed by random characters. Serpel uses it to see how your website answers for a page that doesn’t exist, so it can spot soft 404 pages.
• Your homepage over the other protocol, to see whether HTTP redirects to HTTPS.
• If JavaScript rendering is on for the project, pages in a headless browser.

SerpelBot only connects to ports 80 and 443. It refuses addresses in private networks and hosts such as localhost, so a staging website that is reachable only inside your network can’t be crawled.

What happens when SerpelBot is blocked?

It depends on how the block happens. Serpel tells the three cases apart:

In the Pages list of the crawl, blocked URLs carry the tag robots.txt in the Status column, and the Status filter offers “Blocked by robots.txt”. The issue “Start URL blocked by robots.txt” means that the rule which applies to SerpelBot covers the start URL itself. If that rule is meant for all crawlers, search engines can’t see your content either, so remove or narrow it when your website should be audited and indexed.

How do you allow or block SerpelBot in robots.txt?

Put the lines into the robots.txt of the website that SerpelBot crawls. Serpel fetches the file again at the start of every crawl, so the next crawl follows your change. A rule for SerpelBot affects only SerpelBot. It doesn’t change how Googlebot, Bingbot or any AI crawler treats your website.

Block SerpelBot completely

User-agent: SerpelBot
Disallow: /

Block only some sections

User-agent: SerpelBot
Disallow: /internal/

To block a section but keep one page inside it open, add an Allow rule. The longer rule wins, so the press page stays crawlable:

User-agent: SerpelBot
Disallow: /private/
Allow: /private/press/

Allow SerpelBot while you block other crawlers

A User-agent: * group with Disallow: / blocks SerpelBot too. Give SerpelBot its own group to let it in. An empty Disallow: allows everything:

User-agent: SerpelBot
Disallow:

User-agent: *
Disallow: /

Slow SerpelBot down

User-agent: SerpelBot
Crawl-delay: 5

Wildcards work in paths: Disallow: /*.pdf$ blocks every URL that ends in .pdf. The free robots.txt checker shows which AI crawlers a robots.txt allows on your homepage, plus your llms.txt and sitemaps, so you can confirm that your rules do what you intend.

What does the AI crawler access check do?

With every crawl, Serpel tests your robots.txt against your start URL for Googlebot and Bingbot and for 13 AI crawlers. A crawler counts as allowed when the group that applies to it doesn’t disallow the start URL. Serpel also requests /llms.txt and records whether it answers with a successful status. This is a test of your rules, not a request with each user agent, so a firewall can still stop a crawler that robots.txt allows.

Only some results become issues. “Search engine crawler blocked by robots.txt” is an error and appears when Googlebot or Bingbot is blocked. “AI search crawler blocked by robots.txt” is a warning and appears when a crawler with the purpose “AI search” is blocked, because then your website can’t show up as a source in that assistant. “llms.txt is missing” is a notice. Blocked crawlers with the purpose “Training” or “User fetch” raise no issue, so you can block them on purpose. The audit checks explain each finding, and AI visibility measures what ChatGPT and Google actually answer.

The dashboard doesn’t show this list crawler by crawler. You get it from the CLI, where serpel ai status prints the table and serpel crawl show prints how many AI crawlers are allowed. The CLI guide covers both, and the same data is in the REST API response of GET /projects/{id}/ai-visibility.

serpel ai status --project <id>
serpel crawl show <crawlId>

What can go wrong?

Where to go next