The SerpelBot crawler
If you see a user agent with SerpelBot/0.1 in your server logs, someone has checked your website with Serpel. The crawler identifies as a mobile browser, the way search engines evaluate websites today. Here is what happens during a crawl.
Who is crawling
The crawler belongs to Serpel, a tool for technical SEO audits. It only visits a website if someone has added it as a project and started or scheduled a crawl.
What it fetches
The served HTML of individual pages, the robots.txt, sitemaps, the llms.txt and the favicon. It also checks embedded images, scripts and stylesheets as well as the targets of external links with HEAD requests, and requests one random, non-existent address per website to detect the error page. If JavaScript rendering is enabled for the project, a headless browser loads individual pages. Forms are not submitted and sign-ins are not attempted.
How gently
At most three concurrent requests to your website with at least 250 milliseconds between them, and a crawl-delay in the robots.txt is respected. Third-party hosts receive at most two concurrent requests. At most 1000 URLs are checked per crawl, usually far fewer.
What it respects
All robots.txt rules for the SerpelBot group, or for all crawlers if there is none. Blocked URLs are not fetched but are marked as blocked in the report.
Block the crawler
Add these two lines to the robots.txt of your domain. The next crawl will follow them.
User-agent: SerpelBot Disallow: /
To exclude only certain sections, use the respective path instead of “/”, for example “/internal/”.