Site audit

What does a site audit check?

A site audit is a crawl of your website followed by an analysis. SerpelBot, the Serpel crawler, starts at your start URL, follows your internal links and reads every page it reaches. Serpel then applies 94 checks to what it found: 11 errors, 43 warnings and 40 notices in 18 categories, among them “Redirects”, “Title and description”, “Linking”, “Structured data” and “Core Web Vitals”. The check reference explains every check and its fix, and the site audit feature page gives the short overview.

The crawler collects facts about each URL, such as status code, title, canonical tag, links and response time. The checks judge those facts. A finding is an issue type, for example “Internal URL returns a 4xx status code”, together with the URLs it affects.

How does a crawl run?

Every crawl runs through the same stages. The order matters for the page limit: links are followed first, and the sitemap and hreflang alternates are added afterwards.

When the results are saved, Serpel also re-checks tasks that you linked to an issue and queues a recalculation of your opportunities (at most once an hour).

How do you start a crawl?

1. 1

   Open Issues.

   In the sidebar, open Issues under Site audit. The Start crawl button sits at the top right of Issues and Crawls. On an empty page it reads Start first crawl. You need the editor or owner role in the project. Viewers can read results but can’t start crawls.

2. 2

   Choose the page limit.

   In the Crawl website dialog, set Maximum number of pages. It starts at the Pages per crawl value of your project, which is 100 by default, and it can’t exceed the limit of your plan. The dialog shows the estimated maximum price in credits.

3. 3

   Start and follow the crawl.

   Click Start crawl. A Website crawl is running banner shows the progress, and Cancel stops the crawl. Only one crawl runs per project at a time. If you start another one, Serpel tells you that a crawl is already running.

4. 4

   Read the result.

   When the crawl has finished, Issues shows the new result. The audit score appears under Crawls, in the comparison view and on the project overview.

The page limit counts every URL the crawler fetches, so redirects and error pages use it up as well. The limit per crawl depends on your plan:

Set the default for new crawls and for the weekly crawl under Settings → General → Pages per crawl. The other plan limits are described in credits and billing.

What does the crawler do on your website?

Which URLs does it visit?

The crawler stays on your domain. It treats your domain with and without www as internal. Other subdomains count as external links and are only checked, not crawled. It follows links from page to page unless a page carries a nofollow robots directive, so a page that no crawled page links to and that your sitemap doesn’t list is never found.

Once the links are used up, the crawler reads the sitemaps that your robots.txt names, or /sitemap.xml if it names none, and visits the URLs that only the sitemap lists. Hreflang alternates on your domain come last. If the page limit ends the crawl earlier, those URLs are never visited, so raise the limit when your sitemap lists pages you want checked. How the crawler paces its requests and what it does with your robots.txt is described on the SerpelBot page.

How does it treat redirects, errors and soft 404s?

A redirect response is saved as its own URL together with its target, and the crawler then visits the target. Error pages and unreachable URLs are saved with their error. The crawler also tests the start URL over HTTP and HTTPS to see whether HTTP redirects to HTTPS.

To catch pages that answer with status 200 but behave like error pages, the crawler requests one made-up address per host and compares your pages with the answer. The result is a probability. You see it in the Soft 404 check section of the page detail, and Serpel states there that it is not a definite finding.

When does it render JavaScript?

By default the crawler reads the HTML that your server delivers. Some websites build their text and links in the browser, so the delivered HTML is nearly empty. In the default mode, Serpel renders a page in a real browser when it has fewer than 50 words and shows a sign of a JavaScript application. You choose the mode under Settings → General → JavaScript rendering.

A crawl renders at most 200 pages. Pages has a Rendered column, hidden by default, and the field Checked as in the page detail shows Rendered with JavaScript for rendered pages. A notice on Issues tells you when the rendering service wasn’t available, when the render limit was reached or when single pages failed to render, and it names the reason.

How does it measure Core Web Vitals?

After the crawl, Serpel asks Google PageSpeed Insights to measure your start URL and the most linked indexable pages, 5 pages in total, on mobile and on desktop. It reports Largest Contentful Paint (good up to 2.5 s, poor above 4 s), Interaction to Next Paint (good up to 200 ms, poor above 500 ms) and Cumulative Layout Shift (good up to 0.1, poor above 0.25). Field data comes from real visitors over the last 28 days, lab data is a single measurement, and Interaction to Next Paint exists only as field data.

If PageSpeed Insights can’t be reached, the crawl still completes. The Core Web Vitals section on Issues then says that the measurement failed, and Serpel tries again with the next crawl.

How does it check external links?

The crawler checks the external links on your pages and the images, scripts and stylesheets they load. It sends HEAD requests and confirms a failure with a GET request. A crawl checks at most 400 targets and stops after 120 seconds, so on large websites some targets are skipped.

How do you read the audit results?

The sidebar section Site audit holds the result pages. They all show one crawl, by default the latest completed one.

Issues marked Based on a heuristic are hints, not certain errors. Check them before you change anything on your website.

How is the audit score calculated?

The audit score is Serpel’s own measure, not a Google metric. Every crawl starts with 100 points. For each issue type that the crawl finds, Serpel deducts a fixed base amount that depends on the severity, plus an amount that grows with the number of affected locations compared with the number of HTML pages. It deducts per issue type, not per URL. Site-wide issues, such as a missing sitemap, count as if they affected every page.

Take a crawl with 200 HTML pages. “Internal URL returns a 4xx status code” affects 20 pages and costs 7.4 points. “Meta description is missing” affects 100 pages and costs 5 points. “No XML sitemap found” is site-wide and costs 2 points. Together that is 14.4 points, so the score is 86. Serpel rounds the result and keeps it between 0 and 100.

The CLI prints these labels next to the score. In the dashboard, the ring beside the score is green from 90, amber from 70 and red below that. All checks count, including heuristic ones, and the score is not comparable between crawls with very different page limits.

How do you compare two crawls?

On Issues, click Compare to compare the shown crawl with the one before it. The button appears once a second crawl has finished. On the comparison page, Before and After let you pick other completed crawls.

Serpel matches issue types. New lists types that the older crawl didn’t find, Resolved lists types that the newer crawl no longer finds, and Changed count lists types with more or fewer affected locations. If the two crawls checked a different number of pages, the page says so, because part of a difference can come from that.

How does the weekly crawl work?

Turn on Weekly crawl under Settings → Monitoring. Serpel then starts a crawl every Monday and compares it with the previous one. The start time is the hour you set for monitoring in the same section, in Europe/Berlin time. The crawl uses your Pages per crawl value, capped by your plan. The monitoring and reports guide describes the schedule in detail.

A scheduled crawl needs the same credits as a manual one. If the workspace can’t cover the reserved maximum on that Monday, Serpel doesn’t start the crawl, emails the workspace owner once and checks again during that Monday. If the credits are still missing, the crawl is skipped until the next Monday. A project above the project limit of your plan is paused without an email.

What does a crawl cost?

A crawl costs 1 credit per started block of 20 pages. When the crawl starts, Serpel reserves the maximum for your page limit. When it ends, you are charged only for the pages it delivered, never more than the reserved amount. A crawl that fails, that you cancel or that delivers no pages costs nothing. Rendering, link checks and Core Web Vitals add no credits. Here is what Serpel reserves at different limits:

Serpel counts every URL in the result, including redirects, error pages and URLs that robots.txt blocks. A crawl of 21 pages therefore costs the same as a crawl of 40. The full price list and the rules for reservations are in credits and billing.

How long does Serpel keep crawl data, and how do you export it?

Serpel keeps the pages and links of the last 5 crawls of each project. For older crawls, Pages and the page detail are empty. Their score, issue counts and issues stay, so older results and comparisons keep working.

To export issues, press Ctrl+K or Cmd+K on Issues and choose Download issues as CSV or Download issues as JSON. On Pages, open Options and choose Export as CSV. The history and exports guide lists every export.

How do you run a site audit from the CLI, API or MCP?

The CLI and the REST API cover starting and reading crawls, comparing them and reading Core Web Vitals. The CLI is the quickest way to start a crawl in CI, and the API exposes the same data as JSON. Over MCP, an agent can only read the issues of the latest crawl. It can’t start a crawl.

The read endpoints need the scope audit:read, and get_crawl_issues needs it too. Set up an agent with the Claude Code guide. To act on a finding, serpel tasks create --project <id> --issue <code> turns an issue into a task, and Serpel re-checks the task after every crawl.

serpel crawl start --project <id> --max-pages 200 --wait
serpel audit issues --project <id> --severity error
serpel crawl compare --project <id>

serpel projects update <id> --crawl-schedule weekly --render-mode auto --max-pages 500

What can go wrong with a crawl?

Where to go next