SEO scaling: from a single page to a million
Horizontal scaling strategy for SEO across huge page counts and high concurrency.
Quick answer
How do I keep canonical and sitemap hostnames consistent?
Both must read site.origin (the Astro site config). Hardcoded hostnames break the moment you change domain or run a Preview, and the page gets flagged as duplicate content.
Machine-readable page data: /data/seo-scaling.json
In-depth analysis
SEO is fundamentally about making canonical signals (canonical link, sitemap, structured data) clear and traceable so search engines never classify your pages as duplicates. Scaling is data-driven templates plus edge delivery — full SSG pre-render hands peak traffic to cache.
# public/robots.txt
User-agent: *
Allow: /
Disallow: /links/
Disallow: /api/
Sitemap: https://aicode8.com/sitemap-index.xmlSEO Scaling · Key takeaways
- Pages driven by a single data source
- Build output fully static
- Structured data covers every page
- Edge cache load-tested
FAQ & pitfalls
How often does search engine cache refresh?
Bump lastmod to trigger a sitemap re-crawl, paired with an independent canonical. Do not rely on forcing a CDN purge — after the static output updates, crawlers return naturally.
Explore the hub
See all 5 guides in Traffic & SEOSame category as this article, sorted across the codebase.
AI Search scaling: from a single page to a million
Horizontal scaling strategy for AI Search across huge page counts and high concurrency.
Astro scaling: from a single page to a million
Horizontal scaling strategy for Astro across huge page counts and high concurrency.
Cloudflare Pages scaling: from a single page to a million
Horizontal scaling strategy for Cloudflare Pages across huge page counts and high concurrency.
GEO in practice: optimizing for AI search engines and LLMs
How to embed canonical Schema.org structured data so large language models and AI search engines can pick up your pages precisely.
Share & save
What readers are asking
Questions that keep coming back in reader feedback, already answered.
- 01
How do the canonical and sitemap domains stay in sync?
Both must read site.origin (the Astro site setting). A hardcoded domain breaks everywhere the moment you change domains or use a Preview URL, and the pages get treated as duplicates.
See the full guide → - 02
How do I stop generated pages being flagged as duplicate content?
Every page needs a unique canonical, its own title, and genuinely differentiated body copy. Pages that only swap a keyword while keeping an identical structure must gain real differences, or search engines fold them together.
See the full guide → - 03
Why is my Astro component's interactivity not working?
Components ship as zero-JS HTML by default, so an interactive one needs a client:* directive (client:load, client:visible, …). Without it the script is never bundled into the page.
See the full guide →