Running your first crawl

Crawls are the heart of Semantyx. Each one fetches a sample of your pages, runs every SEO rule against them, and produces an issues list scoped to that crawl.

How to start a crawl

• Open the project (sidebar → All projects → click your site).
• Click any of the audit pages (Overview, Technical SEO, etc.).
• If no crawl has run yet, you'll see a Run crawl button.
• Click it. The crawl runs in the background and the page populates as soon as it's done.

How long it takes

Most crawls finish in 30 seconds to 3 minutes depending on your plan's page cap. The free tier crawls 100 pages, Growth crawls 5,000, Agency crawls 50,000.

How we crawl

• We fetch your sitemap.xml and use it as the seed list.
• We follow internal links breadth-first up to your plan's page cap.
• We respect robots.txt — pages you've disallowed are skipped.
• We send a polite request rate (one page every ~500ms by default).
• The crawler identifies as SemantyxBot/1.0.

Scheduled crawls

Growth and Agency plans can schedule recurring crawls (weekly is most common). The previous-vs-current diff highlights what changed between crawls so you spot regressions early.