Sanad_Package_2/PLAN.md

# Sanad Package 2 — Premium Communication — PLAN (locked scope)

> **Status: IMPLEMENTED (self-contained) — built 2026-06-22, pending Docker build
> + on-robot test.** Vendored from SanadV3, mirrors P1. Dashboard port **:8012**.
> Structural validation passed (compile, import resolution, shim-symbol coverage,
> namespace bootstrap, `license_check P2` entitled, YAML).
>
> **Refinement (2026-06):** the engine is now vendored from **SanadV3**, not
> plain Sanad. SanadV3 already implements the mask/face subsystem and the
> evolved voice/audio/arm engine P2 needs — so P2 *vendors and wires* it rather
> than building the mask from scratch. See "Why vendor SanadV3".

P2 is a **superset of P1**: everything P1 does, plus multilingual auto-detect,
voice-commanded **arm gestures**, gestures while speaking, and a lip-syncing LED
face on the BLE "Shining Mask".

## Locked scope (decisions taken)
- **Motion = arm gestures only (first pass).** Voice-command **locomotion**
  (robot walking/turning) is **DEFERRED** to a later pass — `voice/movement_dispatch.py`
  + `G1_Controller/loco_controller.py` and SanadV3's `dashboard/routes/_arbiter.py`
  (loco/nav leg arbitration) are intentionally **out of scope here**.
- **Mask = included, BLE driven from INSIDE the P2 container** (vendor the Mask
  controller; `bleak` + host BlueZ/D-Bus). No separate `sanad-mask` side-car.
- **Single self-contained container** (no `Sanad_Core`, no `sanad-base`) — like P1.
- **Keyless** ship; customer supplies their own Gemini key.

## Why vendor SanadV3 (not plain Sanad)
SanadV3 is **not a fork** of Sanad — it is plain Sanad **plus exactly the
subsystems P2 needs**, reachable through the identical `Project.Sanad.*` import
surface the dashboard routes already use. Verified in the tree:

- `face/mask_face.py` → `FaceController` (BLE asyncio-loop thread + reconnect
  supervisor + `set_speaking`/`set_mouth` lip-sync inputs). Imports only
  `Project.Sanad.config.BASE_DIR`, `core.config_loader`, `core.logger`, then the
  flat Mask lib via `sys.path.insert(mask_dir)`.
- `face/face_motion.py` → `LifelikeFace` (saccades, varied blinks,
  idle/listening/thinking/speaking states, timed reactions, smooth lip-sync),
  with automatic fallback to the flat lib's `FaceAnimator`.
- `config/mask_config.json` → already env-driven
  (`SANAD_MASK_DIR`/`ADDRESS`/`NAME_PREFIX`/`ADAPTER`), `brightness`/`fps`/
  `lifelike`/`autostart`, persisted face colors.
- `dashboard/routes/mask.py` → a **failure-safe** `/api/mask` router; every
  handler is `asyncio.to_thread`-wrapped and maps errors to 503/409/500 so a
  missing mask never crashes the dashboard.
- Lip-sync chain → `gemini/script.py` emits `[[MOUTH:n]]` (0–3) RMS markers;
  `gemini/subprocess.py` exposes `register_mouth_callback`; `core/brain.py`
  `set_gestural_speaking` emits `brain.gestural_speaking_changed`.
- Evolved voice/audio → `voice/live_voice_loop.py` (position-based dedup),
  `voice/text_utils.py` (Arabic normalize + `maybe_trigger_arm`),
  `voice/audio_manager.py` (per-instance + TTL throttle) — supersets of plain
  Sanad's race-prone equivalents.

Building the mask from plain Sanad (the prior plan) would mean **re-deriving all
of the above by hand** and inheriting plain Sanad's known bugs (content-based
dedup, no `[[MOUTH:n]]`, non-atomic `index.json` writes). The one thing SanadV3
does **not** solve is BLE-inside-a-container (it ran the mask on the host
`g1_env`) — but that is a Docker capability/mount problem P2 owns regardless of
base engine.

**Correction to the prior plan:** the lip-sync source of truth is **Gemini's
`[[MOUTH:n]]` markers over the event bus**, *not* a raw audio-amplitude tap, and
the driver is **`FaceController` + `LifelikeFace`**, *not* `Mask/talking.py
TalkingFace`. P2 adopts the marker path.

## Architecture
One self-contained container that **owns all the hardware it needs directly**:
the G1 DDS link, `rt/arm_sdk` (arm), chest **or** USB/Anker audio, and the BLE
mask. It runs the premium slice of the vendored **SanadV3** engine.

P2 is a **containerization wrapper** (like P1), not a fork: it
1. bootstraps the `Project.Sanad` namespace (deployed layout, P1's mechanism),
2. constructs ONLY the P2 superset subsystems,
3. injects a P2-scoped `Project.Sanad.main` shim exposing the singletons the
   routers import lazily (`mask_face`, `brain`, `live_voice`, …),
4. mounts the P1 + premium routers + the logs websocket,
5. serves the real SanadV3 SPA with non-P2 tabs hidden,
6. runs uvicorn on **:8012**.

Why no `Sanad_Core`/`hwbroker`/`sanad-mask` split: that split exists only to stop
**multiple** package containers fighting over the one device set. A customer who
buys **just P2** has a single owner — no contention — so everything folds into one
container (exactly like P1 standalone, and like the original Sanad monolith which
did comms + arm + mask in one process). The ZMQ bus seam stays available for a
future P1+P2+P3 fleet SKU but is **not** part of standalone P2.

## Namespace bootstrap (reuse P1's, not SanadV3's self-alias)
P2 reuses **P1's exact bootstrap** (`app_p1.py` lines 30–46): synthesize a
`Project` namespace package and alias `Project.Sanad` → the vendored `Sanad`
module, then inject a `Project.Sanad.main` shim holding the P2 singletons. The
mask + voice routes resolve their singletons via lazy `from Project.Sanad.main
import mask_face` inside handlers — so the shim must define them. (SanadV3's own
self-alias-by-folder-name is skipped automatically — the vendored tree is named
`Sanad`, so `main.py`'s `if _THIS_DIR.name != 'Sanad'` branch never fires — and
the wrapper-with-shim is what P1 ships and what the routes expect.)

## Features / capabilities

### Inherited from P1 (superset — same code, premium flags on)
Hands-free Gemini conversation · persona editor (who/tone/language) · **keyless**
Gemini key (customer adds own) · chest **or** USB/Anker audio (selectable,
hot-swap) · typed-replay / "say a line" · live logs + download · offline license
gate. **Same audio (mic+speaker) mechanism as P1**, via the evolved per-instance
`voice/audio_manager.py` + `voice/audio_devices.py`.

### New in P2 (this pass)
1. **Multilingual auto-detect** — Gemini natively detects the visitor's language
   (Arabic Gulf/English) and replies in kind, via the bilingual system prompt in
   `gemini/script.py` / `voice/sanad_voice.py`. No per-user flag.
2. **Voice-command arm gestures** — `voice/live_voice_loop.py`: USER speech →
   arm actions via `scripts/sanad_arm.txt` (**23 actions**, non-contiguous ids
   `{0–15, 23–28, 30}`; the file's nominal range is 0–28; hundreds of Arabic/EN
   phrase variants) → `sanad_arm_controller.ARM.trigger_action_by_id()`.
   **Instant** or **deferred** mode (0.65 s fallback so a silent user still
   fires). Master **trigger-enabled** gate (default **OFF**). Position-based
   dedup (`_last_snapshot` + `_trigger_lock`).
3. **Gestures while speaking** — `core/brain.set_gestural_speaking` →
   `brain.gestural_speaking_changed` → `mask_face.set_speaking(True)` (mouth +
   any gestural motion animate together while Gemini talks).
4. **Wake-phrase management** — phrase→action CRUD
   (`voice/wake_phrase_manager.py`, persisted to `data/wake_phrases.json`),
   folded into the live loop at runtime via `_merge_wake_phrases`.
5. **Skills registry** — skill CRUD, execute, upload-audio
   (`dashboard/routes/skills.py`).
6. **Lip-sync on the LED "Shining Mask"** — vendored `face/mask_face.py`
   `FaceController` + `face/face_motion.py` `LifelikeFace`:
   - mouth driven by Gemini `[[MOUTH:n]]` (0–3) markers →
     `gemini/subprocess.register_mouth_callback` → `mask_face.set_mouth(level)`;
   - `set_speaking(on)` from the gestural-speaking event for auto-talk;
   - state-aware idle/listening/thinking + timed reactions (smile/surprised/sad);
   - falls back to the flat lib's `FaceAnimator` if `LifelikeFace`/Pillow/bleak
     unavailable (`lifelike=false` in `mask_config.json`).

### Deferred (NOT in this pass)
- **Voice-command locomotion** (`voice/movement_dispatch.py` + `loco_controller`):
  Gemini's spoken confirmation → discrete bounded steps, `movement_enabled` gate,
  "stop" = E-STOP. Adds a walking-on-voice safety surface + on-robot calibration —
  staged separately. SanadV3's `dashboard/routes/_arbiter.py` (Nav2 ↔ LocoController
  leg arbitration) belongs to this pass, **not** P2's arm-only pass.
- Multi-package fleet via `Sanad_Core` (hwbroker/busd/shared `sanad-mask`).

## Dashboard (:8012) = all P1 tabs **+** (all routes VENDORED from SanadV3)
- **Voice** — adds the multilingual auto-detect toggle + per-language voice config.
- **Live-voice (commands)** — `dashboard/routes/live_voice.py`: start/stop,
  deferred-mode toggle, **trigger-enabled** master gate, status, trigger history.
- **Wake-phrases** — phrase→action CRUD (AR dialects + EN).
- **Motion / Gestural** — `dashboard/routes/motion.py`: gestural-speaking toggle,
  trigger / cancel arm actions. *(Arm only — loco controls present in the route
  but unwired this pass.)*
- **Skills** — `dashboard/routes/skills.py`.
- **Mask / Lip-sync** — the existing SanadV3 SPA **Mask Face** tab + the vendored
  `dashboard/routes/mask.py` (`/api/mask/*`): connect/disconnect, brightness,
  face start/stop/return/color, speaking toggle, mouth slider, expressions,
  text/image/animation overrides, status. **Mounted as-is — not authored.**
- **Logs**.

Non-P2 tabs (recognition, temp/3D, controller, navigation, terminal) are hidden
the same way P1 hides its non-P1 set.

## What it vendors / reuses (self-contained, like P1)
- `vendor/Sanad` — the **SanadV3** engine tree (rsync-excluding
  `data/`, `Logs/`, `__pycache__/`, `tests/`, `static/temp3d/`). Includes
  `face/`, evolved `voice/`, evolved `gemini/`, `motion/`, the mask + live-voice
  + motion + skills routes, and the SPA. *(Locomotion modules vendored but left
  unwired this pass.)*
- `vendor/sanad_pkg` — IPC bus shim + offline license verification lib (P1's set).
- `vendor/mask` — the flat `shiningmask` library copied from `Project/Mask`
  (`mask.py`, `faceanim.py`, `colorface.py`, `constants.py`, `protocol.py`,
  `transport.py`, `bitmap.py`, `NotoSans-Regular.ttf`, …). `Project/Mask` uses
  **flat imports** (`import faces`, `import mask`), so it goes on its **own**
  path (`SANAD_MASK_DIR=/app/mask`, also on `PYTHONPATH`) — **NOT** under
  `Sanad/`, to avoid collisions. `face/mask_face.py` and `face/face_motion.py`
  both `sys.path.insert(mask_dir)` and `import mask / faceanim / colorface`.
- A `sync_vendor.sh` that refreshes **both** `vendor/Sanad` (from SanadV3) and
  `vendor/mask` (from `Project/Mask`), and blanks any baked Gemini key.

## Wiring the lip-sync + gestures (in `app_p2.py` / the `Project.Sanad.main` shim)
1. Construct P1 comms singletons (brain, audio_mgr, voice_client,
   `GeminiSubprocess`) exactly as P1 does.
2. Construct premium singletons: `FaceController()` (mask_face),
   `LiveVoiceLoop(...)`, `WakePhraseManager()`, arm controllers, skills.
3. `brain.attach_live_voice(live_voice)`; **wire the arm⇄locomotion motion-block
   predicate exactly as SanadV3's `main.py` does** (`arm_controller.set_motion_block(...)`).
   This is load-bearing safety, not optional: **two `rt/arm_sdk` publishers
   coexist in-process** — `motion/arm_controller.py` (publisher ~line 237) and
   `motion/sanad_arm_controller.py` (~line 176). They stay collision-free ONLY via
   (a) that `set_motion_block` interlock on `ArmController` and (b)
   `sanad_arm_controller`'s `_is_busy`/`_busy_lock` atomic guard. A boot-time
   'sole writer' assertion only covers cross-*container* contention — it does
   **not** replace this intra-process interlock, which `app_p2.py` must reproduce.
4. **Lip-sync:** `gemini_subprocess.register_mouth_callback(mask_face.set_mouth)`.
5. **Gestures-while-speaking:** subscribe `brain.gestural_speaking_changed` →
   `mask_face.set_speaking(on)`. The event bus is **synchronous**
   (`core/event_bus` `.on`/`.emit_sync`), so `set_speaking` runs on the caller's
   thread — keep it non-blocking (it only flips a flag the BLE loop reads).
6. **Lifelike state:** wire voice events (connected→`set_listening`,
   user_said→`set_thinking`, disconnected→`set_idle`,
   `voice.error`/`motion.action_error`→`react('sad')`,
   `skill.finished`→`react('smile')`). **SanadV3's `main.py` lines ~360–427 are
   the concrete reference implementation** — copy that wiring (incl.
   `register_mouth_callback` ~383–391 and `mask_face.shutdown()` ~587–591) rather
   than re-deriving it.
7. Expose `mask_face`, `brain`, `live_voice` on the `Project.Sanad.main` shim so
   the lazy route accessors resolve them.
8. On shutdown: `mask_face.shutdown()` (BLE disconnect + stop loop) — handle
   SIGTERM so the container exits cleanly.

## Container & hardware
- `FROM python:3.10-slim`, `WITH_UNITREE_SDK=1` (builds — the cyclonedds `idlc`
  fix gives arm + chest audio).
- **System deps (added over P1):** `bluez`, `libdbus-1-3`/`libdbus-1-dev`,
  `libglib2.0-0` for BlueZ/D-Bus; Pillow needs no extra apt on slim.
- **Python deps:** P1's set **+** `bleak==0.22.3` (**pinned** — bleak 3.x throws
  `KeyError 'Roles'` on the Jetson's BlueZ 5.53 and every connect fails) **+**
  `Pillow` (LifelikeFace frame rendering).
- **BLE for the mask (in-container):** mount `/var/run/dbus`,
  `--cap-add NET_ADMIN`, `/dev/bus/usb`; `network_mode: host`. Free the mask
  from the phone app before connecting. Set `SANAD_MASK_DIR=/app/mask`.
- `/dev/snd` (audio), license mount, **writable** `./data` + `./config` mounts
  (mask color persistence), `restart: unless-stopped`.
- Port **:8012**. Ships **keyless** (`strip_key.py` blanks any baked key).
- **License features:** `multilingual`, `voice_command_motion` (arm gestures),
  `lipsync`, `mask`. Entrypoint checks entitlement **P2**. *(A future
  `voice_command_locomotion`/`navigation` feature gates the deferred walking.)*

## Package layout (to build later — mirrors P1)
```
Sanad_Package_2/
  app_p2.py  routes_p2.py  entrypoint.sh  strip_key.py  p2ctl.sh
  config/p2_config.json  static/  Dockerfile  docker-compose.yml  requirements.txt
  vendor/Sanad   (from SanadV3)
  vendor/sanad_pkg
  vendor/mask    (from Project/Mask — own PYTHONPATH)
  license/(pubkey + example)  data/(seed incl. wake_phrases.json)  sync_vendor.sh
  README.md  PLAN.md  NEW_ROBOT_SETUP.md
```

## Build sequence (when implemented)
1. Vendor the **SanadV3** engine + `sanad_pkg` + the flat `Project/Mask` →
   `vendor/mask`; merge requirements (P1 deps + `bleak==0.22.3` + `Pillow`).
2. Self-contained Dockerfile (P1's + BlueZ/D-Bus system deps; `COPY vendor/mask
   /app/mask`; `ENV SANAD_MASK_DIR=/app/mask`).
3. `app_p2.py` — P1's namespace bootstrap + `Project.Sanad.main` shim; construct
   P1 comms **+** `FaceController` + `LiveVoiceLoop` + wake-phrase mgr + skills;
   wire `register_mouth_callback`→`set_mouth` and
   `gestural_speaking_changed`→`set_speaking` + lifelike state hooks; mount P1 +
   premium routers (incl. the vendored `mask.py`); serve :8012.
4. Mask: `FaceController` already runs its own BLE asyncio loop +
   reconnect supervisor; with `autostart=true` it connects + uploads frames
   (~30–90 s one-time, persists on flash) in the background — never blocks boot.
5. Multilingual: ship SanadV3's bilingual system prompt (rename persona as
   decided); enable per-language voice config in the Voice tab.
6. License gate `P2`; keyless; smoke test (P1 endpoints + live-voice + wake +
   skills + `/api/mask/status` → connect → face/start → mouth slider); validate
   on the robot.

## Risks / mitigations
- **BLE-from-container is fiddly** — prototype the mask connect early on the
  target Jetson (BlueZ 5.53) with `bleak==0.22.3`; if in-container BLE
  misbehaves, fall back to the mask controller on the **host** with a tiny socket
  shim (contingency, not the plan). `mask.py` route already degrades to 503 if
  the subsystem is down, so the dashboard never crashes.
- **Mask flat imports** — vendor onto `/app/mask` via `SANAD_MASK_DIR`; never
  place under `Sanad/`.
- **LifelikeFace deps** — needs Pillow + bleak; if either is missing the
  subsystem auto-falls-back to `FaceAnimator` (`lifelike=false`) and reports the
  reason in the Mask tab. The rest of P2 is unaffected.
- **Lip-sync chain spans the voice subprocess** — `gemini/subprocess.py` spawns
  `voice/sanad_voice.py` (the child, `SANAD_VOICE_BRAIN=gemini`), which is an
  **orchestrator** that in turn runs `gemini/script.py` — the actual `GeminiBrain`.
  The `[[MOUTH:n]]` markers are emitted in **`gemini/script.py` (~lines 563/578)**,
  not `sanad_voice.py`. Vendor the full **`script.py → sanad_voice.py →
  subprocess.py`** chain intact and register the parent callback
  (`GeminiSubprocess.register_mouth_callback`) in the shim. P1 already vendors
  `gemini/subprocess.py`, so this is additive wiring, not a new vendor.
- **Arm safety (two publishers + interlock)** — `trigger_enabled` defaults **OFF**;
  arm actions are bounded. The single container is the sole *container* writing
  `rt/arm_sdk`, but **two in-process publishers** (`arm_controller.py` +
  `sanad_arm_controller.py`) coexist — collision-free ONLY via the
  `set_motion_block` interlock + `_is_busy` busy-lock that `app_p2.py` must wire
  (Wiring step 3). Assert sole-container-writer at boot **and** reproduce the interlock.
- **Arm⇄locomotion interlock is a no-op in this arm-only build** —
  `sanad_arm_controller.trigger_action_by_id()` calls `_blocked()` (refuses arm
  motion while the robot may be walking). With locomotion **deferred**, nothing
  publishes the locomotion-state signals `_blocked()` reads, so it always permits
  arm motion — safe by omission **now**, but it **must be re-armed before the
  deferred voice-command-locomotion pass ships**, else voice-walking would enable
  with a silently-disabled arm interlock.
- **Mask hardware presence** — lip-sync needs the physical BLE mask paired/in
  range and freed from the phone app.
- **Locomotion creep** — `movement_dispatch.py` + `_arbiter.py` are present in
  the vendored tree; keep them **unwired** this pass to avoid accidentally
  shipping voice-command walking without the safety surface + calibration.

## Open decisions (resolve before/while building)
1. **In-container BLE vs host side-car** — confirm `bleak`+BlueZ/D-Bus actually
   connects from inside `python:3.10-slim` with `/var/run/dbus` + `NET_ADMIN` +
   `network_mode: host` on the target Jetson (BlueZ 5.53). Pin **`bleak==0.22.3`**
   (3.x throws `KeyError 'Roles'` on BlueZ 5.53 — every connect fails). If flaky,
   fall back to a host-side mask controller + tiny socket shim (contingency).
2. **LifelikeFace default** — SanadV3 defaults `lifelike=true` (needs
   `face/face_motion.py` + Pillow + bleak). Confirm Pillow+bleak install cleanly
   in the slim image; otherwise `mask_config.json lifelike=false` auto-falls back
   to `FaceAnimator`.
3. **Gemini lip-sync chain vendoring** — vendor the evolved `gemini/script.py`
   (emits `[[MOUTH:n]]`) **and** `gemini/subprocess.py` (`register_mouth_callback`),
   and wire `register_mouth_callback → mask_face.set_mouth` in the shim. P1 already
   vendors `subprocess.py`, so this is additive.
4. **Persona / robot name** — SanadV3's fallback prompt identifies the robot as
   **"Marcus"** (bilingual Gulf-Arabic/English auto-detect — this prompt *is* the
   multilingual engine). Decide P2's shipped persona name/dialect in
   `scripts/sanad_script.txt` and whether to keep the bilingual prompt verbatim.
5. **First-boot frame-upload latency** — the one-time DIY frame upload (~30–90 s,
   persists on mask flash). Decide `autostart=true` (background, non-blocking) vs
   gating `face/start` behind an explicit dashboard action with a progress indicator.
6. **Vendor `movement_dispatch.py` now (unwired) or omit** — keep for forward-compat
   with the deferred locomotion pass, or drop it to keep the arm-only image lean.
7. **Writable mounts for persisted face colors** — `/face/color` persists eye/
   mouth/sclera colors to `mask_config.json`; ensure `config/` (or the `data/`
   seed) is on a writable volume so colors survive container restarts.

> **Carried-forward safety gate:** before the deferred *voice-command-locomotion*
> pass ships, re-arm the `_blocked()` arm⇄locomotion interlock (a no-op today) and
> revisit `_arbiter.py` (Nav2 ↔ LocoController leg arbitration).

## Container-runtime audit (2026-06-23)
A 3-reader audit of the vendored runtime paths (voice/audio, mask/face, arm+dashboard)
for in-container failure modes. **Verdict: container-safe on the G1 after the
`iproute2` fix** — no remaining crash-level landmine on the builtin+gemini+host-net path.
- **FIXED (crash, P1+P2):** chest-mic `voice/audio_io.py:_find_g1_local_ip()` shells
  out to `ip` — added **`iproute2`** to both Dockerfiles. (This was the live-voice
  crash we hit.)
- **Mitigated (crash, off-robot only):** `BuiltinMic.start()` calls
  `_find_g1_local_ip()` **unguarded**; off a G1 (no `192.168.123.x` net) it raises
  and kills the voice subprocess. On the G1 (network_mode host) the interface exists →
  fine. Package-level safety valve: **`SANAD_AUDIO_PROFILE=plugged` on any non-G1 host**
  (documented in compose + `.env.example`). Not fixed in the engine (shared with SanadV3).
- **FIXED (degraded, persistence):** mask face colors are written to
  `config/mask_config.json` (a baked layer, lost on recreate) → added a pre-seeded
  single-file mount `./config/mask_config.json:/app/Sanad/config/mask_config.json`.
- **Deploy-side (degraded, P2):** mask BLE cold-start can stall ~45 s (3×15 s scan) if
  host `bluetoothd` is down / adapter missing — retry-bounded, supervised, background
  thread; does **not** crash. Ensure `bluetoothd` is up; leave `SANAD_MASK_ADDRESS`
  empty for auto-detect.

Verified-safe (NO action, to avoid over-fixing): `~/logs` FileHandler (root-writable,
no crash); `NotoSans-Regular.ttf` present + COPYed; `local_tts` torch never imported
(gemini brain); `pactl` with no server (guarded, and not on the builtin voice path);
`parec` capture (gated); arm/DDS init (degrades, catches); config path resolution
(robust fallback chain); `teaching.py` tempfile (writes under the `./data` mount).