Site Map Crawl

Map same-origin pages via sitemap and BFS crawl.

Skill metadata

Source	Bundled (installed by default)
Path	skills/web-development/site-map-crawl
Version	1.0.0
Author	VibeOS
License	MIT
Platforms	linux, macos, windows
Tags	browser, crawl, sitemap, architecture, seo, web
Related skills	site-page-analyze, browser-e2e, dogfood, cinematic-web

Reference: full SKILL.md

信息

The following is the complete skill definition that VibeOS loads when this skill is triggered. This is what the agent sees as instructions when the skill is active.

Site Map Crawl

Build a same-origin site architecture map from sitemap.xml discovery plus a bounded BFS HTML crawl. Writes site-map.json and an optional graph summary markdown. No new core tools — prefer the CLI scripts under scripts/; use browser_navigate / browser_snapshot only for spot-checks, not as the crawler.

When to Use

• Inventory routes, internal links, and orphan candidates on one origin
• Diff two crawls after a redesign or CMS migration (diff_site_maps.py)
• Feed a shortlist of important URLs into site-page-analyze (cap ≤5 unless the user asks for more)
• Offline / CI: local fixtures via crawl_site.py --html-dir (no network)

Prerequisites

• Python 3 available; scripts live next to this skill under scripts/
• Network for live crawls, or local HTML fixtures / a local test server
• Write access for site-map.json (and optional summary .md) in the cwd
• Optional: browser toolset for verifying a few pages after the map exists

How to Run

Live (preferred):

1. Resolve seed URL (homepage or known entry). Confirm same-origin intent.
2. python scripts/fetch_sitemap.py --url <seed> — reads robots.txt, collects sitemap URL lists (live URL or local sitemap/robots files).
3. python scripts/crawl_site.py --seed <url> … — BFS over HTML, extracts <a href>, merges sitemap seeds, emits site-map.json (+ optional graph md).
4. Defaults (safety): same-origin only, max pages 30, rate limit between requests, honor robots.txt unless --ignore-robots.
5. Optional: python scripts/diff_site_maps.py a.json b.json → added / removed / changed pages.
6. Spot-check 1–2 key URLs with browser_navigate + browser_snapshot if the map looks thin or auth-walled.
7. Hand off: pick ≤5 important URLs → site-page-analyze per page.

Offline:

1. fetch_sitemap.py --robots-file … --sitemap-file … --offline and/or crawl_site.py --html-dir <fixture> --seed-path /index.html --base-url ….
2. Skip live browser tools unless verifying a served page.

If paths are unknown: search_files for site-map-crawl/.

Quick Reference

Script	Role
scripts/fetch_sitemap.py	robots.txt + sitemap URL list (live or local files)
scripts/crawl_site.py	BFS HTML crawl → pages + edges → site-map.json
scripts/diff_site_maps.py	Compare two site-map.json → added/removed/changed

Flag / default	Meaning
same-origin	Never follow off-origin links into the map
max pages = 30	Hard cap unless user overrides
rate limit	Pause between fetches (script default)
honor robots.txt	Skip disallowed paths unless --ignore-robots

site-map.json schema (v1 overview): schema_version (1), seed_url, origin, pages[], edges[], counts (e.g. page/edge totals). Pages carry normalized URL + crawl metadata; edges are {from, to} (or equivalent) link pairs discovered from HTML / sitemap seeds.

Procedure

1. Confirm seed URL, origin, and max-page budget (default 30).
2. Run fetch_sitemap.py for robots + sitemap URL seeds.
3. Run crawl_site.py with same-origin + rate limit; do not disable robots unless the user passes --ignore-robots intent.
4. Write site-map.json; optionally emit graph summary markdown.
5. Skim counts and sample pages/edges for empty crawl or auth walls.
6. If comparing releases: diff_site_maps.py old.json new.json.
7. Select ≤5 high-value URLs from the map; run site-page-analyze on each (raise the cap only if the user asks).
8. Report output paths, page/edge counts, and blockers (robots blocks, 4xx/5xx clusters, login walls).

Pitfalls

• Zero new core tools — do not invent a core “site_map” tool; use scripts.
• Prefer scripts (and optional browser_navigate / browser_snapshot / web_extract for spot-checks). Do not shell-grep/cat HTML as the primary crawl path.
• Off-origin and mailto:/javascript: links are noise — keep same-origin.
• Ignoring robots without explicit user intent is unsafe; keep the default.
• Huge sites: raise max pages deliberately; do not unbounded-crawl.
• SPAs with client-only routes may under-report without a real HTML link graph; note the gap, then spot-check with the browser toolset.

Verification

• site-map.json exists with schema_version: 1, seed_url, origin, non-empty pages (for a healthy seed), edges, and counts.
• Crawl stayed same-origin; page count ≤ configured max.
• Without --ignore-robots, disallowed paths from robots are absent.
• Diff run (if used) reports added/removed/changed without schema errors.
• Offline/fixture run exits 0 against local files or local HTTP fixtures.