Desktop Operate

Run app-control tasks with self-heal and scorecard.

Skill metadata

Source	Bundled (installed by default)
Path	skills/desktop/desktop-operate
Version	1.0.0
Author	VibeOS
License	MIT
Platforms	macos, windows, linux
Tags	operate, computer-use, desktop, self-heal, scorecard
Related skills	choose-interaction-surface, computer-use, macos-app-playbooks, windows-app-playbooks, linux-app-playbooks, challenge-gate, raster-edit, video-nle, figma-edit

Reference: full SKILL.md

info

The following is the complete skill definition that VibeOS loads when this skill is triggered. This is what the agent sees as instructions when the skill is active.

Desktop Operate Skill

Orchestrate Operate mode: pick a surface, drive a native app (or browser when appropriate), self-heal with re-capture, and stop cleanly within ≤2 retries before human escalate. Write an honest scorecard.

This skill does not replace OS playbooks or creative editor skills — it routes to them and enforces reliability rules.

When to Use

• User wants the agent to complete a concrete GUI task in a desktop app.
• Creative editor handoffs (raster-edit, video-nle, figma-edit GUI path).
• Office / Finder / Preview style tasks via OS app playbooks.

Do not use for deep web research (Investigate) or greenfield media generation without an editor (Create via creative-studio tools first).

Prerequisites

• vibeos computer-use doctor green enough for the OS (permissions granted)
• Load choose-interaction-surface before acting
• Purchases, Send mail, delete, overwrite masters → human gate

How to Run

1. Write operate-brief.md from templates/operate-brief.md.
2. Choose surface; announce it; open the target app.
3. Loop: capture → act by element → verify (re-capture).
4. On failure: one alternate-surface attempt, then escalate (≤2 tries total).
5. Fill scorecard.md + qa.md; run scripts/eval_operate.py.

Quick Reference

Step	Rule
Surface	MCP/API/terminal > browser_* > computer_use
Action	Element index from SoM/AX — no blind pixels
Retry budget	2 stuck attempts then human
CAPTCHA	challenge-gate (never CU-first on Chrome puzzles)
Destructive	Stop and ask

Procedure

1. Brief

Copy templates/operate-brief.md. One goal, one primary app, success check.

2. Self-heal policy

capture → act → verify (capture_after / re-capture)
  ├─ success → next step
  ├─ empty tree / wrong window → refocus app, re-capture (counts as retry)
  ├─ wrong surface → switch once (browser↔CU / MCP note)
  └─ still stuck after 2 tries → clarify / ask user; partial scorecard

Never burn the budget on the same blind click.

3. Scorecard

Update scorecard.md (template in templates/scorecard.md):

• Goal met? (yes / partial / no)
• Attempts used (1–2)
• Escalate reason (if any)
• Artifacts paths

4. Creative editors

Still polish → raster-edit. NLE → video-nle. Figma → figma-edit (MCP first).

Pitfalls

• Do not thrash permissions dialogs — point the user at vibeos computer-use permissions grant.
• Do not claim success without a verify capture or on-disk artifact.
• Messaging platforms may disable computer_use — say so and stop.

Verification

• operate-brief.md filled; scorecard.md Status is honest.
• Artifact or screenshot proof when the brief requires it.
• Offline eval: python scripts/eval_operate.py --workspace <dir>.
• CLI: vibeos operate run|doctor|preset desktop-operate.
• Smoke: scripts/smoke-desktop-operate.sh.