Choose Interaction Surface

Pick browser, computer_use, MCP, or terminal for a task.

Skill metadata

Source	Bundled (installed by default)
Path	skills/desktop/choose-interaction-surface
Version	1.0.0
Author	VibeOS
License	MIT
Platforms	linux, macos, windows
Tags	desktop, browser, computer-use, mcp, routing
Related skills	computer-use, macos-desktop, windows-desktop, linux-desktop, macos-app-playbooks, windows-app-playbooks, linux-app-playbooks, browser-e2e, challenge-gate

Reference: full SKILL.md

info

The following is the complete skill definition that VibeOS loads when this skill is triggered. This is what the agent sees as instructions when the skill is active.

Choose Interaction Surface

Pick the cheapest reliable surface before acting. Wrong surface wastes turns and breaks GUIs. Prefer structured APIs over pixels.

When to Use

• Task could be done via browser, native desktop, shell, or an installed MCP.
• User says “open / click / fill / control” without specifying the path.
• After a failed attempt on one surface (escalate, don’t thrash).

Prerequisites

• Know which toolsets/MCP servers are loaded this session.
• Do not ask to enable new toolsets mid-conversation (prompt-cache). Use what is available; say what is missing for next session.

Decision table

Situation	Prefer	Avoid
Public web page / web app with DOM	browser_*	computer_use on the browser window
Native desktop app (Finder, Notes, Excel UI)	computer_use (capture → element click)	Blind coordinate clicks
Operate pack task (scorecard / ≤2 retries)	desktop-operate + OS playbook	endless GUI thrash
Creative still polish in Photoshop/Affinity	raster-edit then computer_use	inventing UXP scripts
Figma file already shared	Figma MCP (figma-edit)	pixel-driving Figma first
Service has MCP (Notion, Linear, GitHub, Gmail/Drive, …)	that MCP	scraping the same SaaS in a browser
Google Workspace multi-app	google-workspace MCP or skill	Gmail web UI via computer_use
CLI / files / git / scripts	terminal / file tools / execute_code	driving a terminal GUI
PDF / image on disk (read/OCR)	read_file / vision_analyze	screenshotting the previewer first
PDF / image needs GUI export annotate	Preview / editor playbook + desktop-operate	blind clicks without verify
Login wall / heavy CAPTCHA	challenge-gate (L0–L4): search/extract/MCP → one assist → clarify human → partial	endless browser retries; computer_use on Chrome first
Soft Cloudflare “Just a moment”	browser_* + ChallengeGate L1/L2 once	pixel spam; invent page text
Hard image CAPTCHA / Turnstile	L3 human (clarify → готово/done)	captcha farms; computer_use first
Native app challenge (not a browser tab)	computer_use only if no browser_*/MCP path	driving Chrome via CU for web CAPTCHA

CAPTCHA taxonomy (browser vs native)

Kind	Surface	Notes
Web bot-wall / CAPTCHA	browser_* + challenge-gate	Last resort: human; CU on Chrome is not first choice
Native WebView CAPTCHA (no browser tools)	computer_use after one failed browser path	Still human-first for hard puzzles
Login / OAuth SaaS	MCP with OAuth	Never paste passwords into chat

How to Run

1. Name the goal in one sentence.
2. Classify with the table above.
3. Announce the chosen surface briefly, then act.
4. If it fails once for a good reason (no a11y tree, tool missing), escalate:

browser → computer_use (same app) → ask user / MCP / terminal
computer_use → browser (if it’s really a webview) → terminal scripts
MCP missing → browser/CU only if necessary; note install for next session

Quick Reference

Surface	Entry tools
Browser	browser_navigate, browser_snapshot, browser_click, browser_type
Desktop	computer_use (capture/som → click by element)
Apps API	MCP tools (notion-search, Linear, …)
Shell	terminal, read_file, patch, execute_code

Procedure

Browser path

1. browser_navigate → browser_snapshot.
2. Click/type via refs from the snapshot.
3. Use browser_vision only when the a11y tree is useless.

Computer-use path

1. computer_use(action="capture", mode="som", app="...").
2. Click by element index, not pixels.
3. Re-capture after state changes (capture_after=True when available).

MCP path

1. Prefer search/fetch tools first (read-only).
2. Mutations only when the user asked and the tool is enabled.

Pitfalls

• Do not drive Chrome via computer_use when browser_* works.
• Do not open a SaaS UI in the browser when its MCP is already connected.
• Do not invent MCP tools that are not in the session schema.
• On messaging platforms, computer_use may be disabled — use browser/MCP/terminal.
• On CAPTCHA/bot-wall: load challenge-gate; never thrash retries or fabricate page text.

Verification

• Chosen surface matches the table.
• First action is observe (snapshot/capture/list), not a blind click.
• On failure, escalate once with a stated reason.