Talking Head

Render talking-photo clips from portrait and audio.

Skill metadata

Source	Optional — install with vibeos skills install official/creative/talking-head
Path	optional-skills/creative/talking-head
Version	1.0.0
Author	VibeOS
License	MIT
Platforms	linux, macos, windows
Tags	creative, avatar, video, talking-head, tts
Related skills	creative-studio, hyperframes, video-nle

Reference: full SKILL.md

信息

The following is the complete skill definition that VibeOS loads when this skill is triggered. This is what the agent sees as instructions when the skill is active.

Talking-Head Skill

Optional render layer for Avatar dialogue. Turns a still portrait + spoken reply into a short clip. Does not add core model tools — it orchestrates video_generate, TTS files, and optional HyperFrames.

Vendor talking-head SaaS (HeyGen, D-ID, …) must ship as a standalone plugin under ~/.vibeos/plugins/ — never as an in-tree product plugin. Authoring guide + scaffold:

• references/standalone-vendor-plugin.md
• bash scripts/scaffold-talking-head-vendor-plugin.sh

When to Use

• User wants a talking-head / talking-photo clip for an AvatarProfile.
• display.avatar.talking_head.mode is skill and a portrait is configured.
• Creative Studio motion mode needs a portrait-driven short.

Do not use for live webcam conferencing or real-time streaming.

Prerequisites

• Avatar with portrait_path (local image under workspace / avatars/).
• Local ffmpeg for the default talking-photo mux (scripts/render_local.py).
• Optional: video_generate or HyperFrames for true motion / captions.
• TTS audio already produced, or text that can be spoken via text_to_speech.

Install this optional skill:

vibeos skills install official/creative/talking-head

How to Run

1. Write a job JSON from templates/talking-head-job.json.
2. Validate: python scripts/validate_job.py --job job.json.
3. Default (offline): mux still + audio with ffmpeg:

python scripts/render_local.py \
  --portrait avatars/pilot-portrait.png \
  --audio /tmp/line.mp3 \
  --output exports/pilot-talking-head.mp4 \
  --json

Desktop skill mode calls the same path via avatar.talking_head.render (TTS → ffmpeg → MP4 data URL above the pet).

4. Escalation: video_generate with image_url = portrait for motion.
5. Captions / overlays → hyperframes.
6. Paid lip-sync: only via a user-installed vendor plugin (talking_head_vendor_render after scaffold) — see references/standalone-vendor-plugin.md.
7. Tell the user the default clip is a talking-photo (still + audio), not vendor lip-sync.

Quick Reference

Input	Source
portrait	AvatarProfile.portrait_path or job portrait_path
speech	TTS file (audio_path) or job text
mode	display.avatar.talking_head.mode (skill)
wave-only UX	desktop pet celebrate during TTS (no this skill)

Procedure

Decision tree (wave vs local vs vendor)

Situation	Action
Short reply, pet visible, mode=wave	Desktop celebrate + TTS only — no this skill
mode=skill, no vendor / prefer_vendor=false	Local ffmpeg talking-photo (render_local.py or avatar.talking_head.render)
prefer_vendor=true and talking_head_vendor_render available	Call vendor tool first; on failure fall back to ffmpeg
User asks for paid lip-sync and plugin missing	Point at scaffold; do not invent a core tool
Offline dry-run of vendor wiring	TALKING_HEAD_VENDOR_MOCK=1 on the user plugin (still+audio, labeled mock)

Honesty labels: default clips are talking-photo (backend: ffmpeg-still). Vendor success should report backend: vendor (or mock-vendor). Never call a mux “lip-sync” unless the vendor path actually ran.

1. Resolve inputs

Load the active avatar (avatar.get / YAML under <VIBEOS_HOME>/avatars/). Require a local portrait_path. Prefer existing audio_path; else speak text with the avatar voice_preset. Check display.avatar.talking_head.prefer_vendor.

2. Generate

• If prefer-vendor and the tool exists: talking_head_vendor_render → MP4.
• Else primary local: scripts/render_local.py / gateway avatar.talking_head.render.
• Escalation: video_generate with image_url = portrait for motion.
• Advanced: HyperFrames composition with the portrait + audio track.

3. Verify

Confirm output_path exists and is a non-empty video. State the backend label (ffmpeg-still / vendor / mock-vendor). Do not claim lip-sync accuracy the provider cannot deliver.

Pitfalls

• Missing portrait → stop; do not invent a face.
• Do not call unpaid vendor APIs without user-configured credentials.
• Do not add new core tools for this pipeline.

Verification

python scripts/validate_job.py --job templates/talking-head-job.json