Перевод: оболочка страницы на русском; тело навыка (SKILL.md) пока на английском — это исходные инструкции агента.

Site Map Crawl

Map same-origin pages via sitemap and BFS crawl.

Метаданные навыка

Источник	Встроенный (установлен по умолчанию)
Путь	skills/web-development/site-map-crawl
Версия	1.0.0
Автор	VibeOS
Лицензия	MIT
Платформы	linux, macos, windows
Теги	browser, crawl, sitemap, architecture, seo, web
Связанные навыки	site-page-analyze, browser-e2e, dogfood, cinematic-web

Справка: полный SKILL.md

к сведению

Ниже полное определение навыка, которое VibeOS загружает при активации. Это инструкции, которые видит агент.

Site Map Crawl

Build a same-origin site architecture map from sitemap.xml discovery plus a bounded BFS HTML crawl. Writes site-map.json and an optional graph summary markdown. No new core tools — prefer the CLI scripts under scripts/; use browser_navigate / browser_snapshot only for spot-checks, not as the crawler.

When to Use

• Inventory routes, internal links, and orphan candidates on one origin
• Diff two crawls after a redesign or CMS migration (diff_site_maps.py)
• Feed a shortlist of important URLs into site-page-analyze (cap ≤5 unless the user asks for more)
• Offline / CI: local fixtures via crawl_site.py --html-dir (no network)

Prerequisites

• Python 3 available; scripts live next to this skill under scripts/
• Network for live crawls, or local HTML fixtures / a local test server
• Write access for site-map.json (and optional summary .md) in the cwd
• Optional: browser toolset for verifying a few pages after the map exists

How to Run

Live (preferred):

1. Resolve seed URL (homepage or known entry). Confirm same-origin intent.
2. python scripts/fetch_sitemap.py --url <seed> — reads robots.txt, collects sitemap URL lists (live URL or local sitemap/robots files).
3. python scripts/crawl_site.py --seed <url> … — BFS over HTML, extracts <a href>, merges sitemap seeds, emits site-map.json (+ optional graph md).
4. Defaults (safety): same-origin only, max pages 30, rate limit between requests, honor robots.txt unless --ignore-robots.
5. Optional: python scripts/diff_site_maps.py a.json b.json → added / removed / changed pages.
6. Spot-check 1–2 key URLs with browser_navigate + browser_snapshot if the map looks thin or auth-walled.
7. Hand off: pick ≤5 important URLs → site-page-analyze per page.

Offline:

1. fetch_sitemap.py --robots-file … --sitemap-file … --offline and/or crawl_site.py --html-dir <fixture> --seed-path /index.html --base-url ….
2. Skip live browser tools unless verifying a served page.

If paths are unknown: search_files for site-map-crawl/.

Quick Reference

Script	Role
scripts/fetch_sitemap.py	robots.txt + sitemap URL list (live or local files)
scripts/crawl_site.py	BFS HTML crawl → pages + edges → site-map.json
scripts/diff_site_maps.py	Compare two site-map.json → added/removed/changed

Flag / default	Meaning
same-origin	Never follow off-origin links into the map
max pages = 30	Hard cap unless user overrides
rate limit	Pause between fetches (script default)
honor robots.txt	Skip disallowed paths unless --ignore-robots

site-map.json schema (v1 overview): schema_version (1), seed_url, origin, pages[], edges[], counts (e.g. page/edge totals). Pages carry normalized URL + crawl metadata; edges are {from, to} (or equivalent) link pairs discovered from HTML / sitemap seeds.

Procedure

1. Confirm seed URL, origin, and max-page budget (default 30).
2. Run fetch_sitemap.py for robots + sitemap URL seeds.
3. Run crawl_site.py with same-origin + rate limit; do not disable robots unless the user passes --ignore-robots intent.
4. Write site-map.json; optionally emit graph summary markdown.
5. Skim counts and sample pages/edges for empty crawl or auth walls.
6. If comparing releases: diff_site_maps.py old.json new.json.
7. Select ≤5 high-value URLs from the map; run site-page-analyze on each (raise the cap only if the user asks).
8. Report output paths, page/edge counts, and blockers (robots blocks, 4xx/5xx clusters, login walls).

Pitfalls

• Zero new core tools — do not invent a core “site_map” tool; use scripts.
• Prefer scripts (and optional browser_navigate / browser_snapshot / web_extract for spot-checks). Do not shell-grep/cat HTML as the primary crawl path.
• Off-origin and mailto:/javascript: links are noise — keep same-origin.
• Ignoring robots without explicit user intent is unsafe; keep the default.
• Huge sites: raise max pages deliberately; do not unbounded-crawl.
• SPAs with client-only routes may under-report without a real HTML link graph; note the gap, then spot-check with the browser toolset.

Verification

• site-map.json exists with schema_version: 1, seed_url, origin, non-empty pages (for a healthy seed), edges, and counts.
• Crawl stayed same-origin; page count ≤ configured max.
• Without --ignore-robots, disallowed paths from robots are absent.
• Diff run (if used) reports added/removed/changed without schema errors.
• Offline/fixture run exits 0 against local files or local HTTP fixtures.