Compare commits

...

10 Commits

Author SHA1 Message Date
699e89f336 Update 2026-07-04 23:29:09 2026-07-04 23:29:10 +04:00
4210c4cc61 Update 2026-06-08 12:59:00 2026-06-08 12:59:01 +04:00
ca0de44401 Update 2026-06-08 11:03:52 2026-06-08 11:03:53 +04:00
811a391932 Update 2026-05-15 09:39:52 2026-05-15 09:39:53 +04:00
edddb7e0c3 Update 2026-05-13 14:42:31 2026-05-13 14:42:34 +04:00
54b1e745ca Update 2026-05-01 13:40:59 2026-05-01 13:41:00 +04:00
f7da15da1b Update 2026-04-21 12:10:35 2026-04-21 12:10:37 +04:00
1693776f3f Update 2026-04-21 12:00:42 2026-04-21 12:00:43 +04:00
cf5e916120 Update 2026-04-21 11:47:43 2026-04-21 11:47:44 +04:00
94e4a9c4cb Update 2026-04-20 17:59:46 2026-04-20 17:59:47 +04:00
191 changed files with 24696 additions and 1588 deletions

7
.claude/settings.json Normal file
View File

@ -0,0 +1,7 @@
{
"permissions": {
"allow": [
"Bash(node -e ' *)"
]
}
}

136
FEATURES.md Normal file
View File

@ -0,0 +1,136 @@
# SanadV3 — Feature Catalog
Sanad is a bilingual (Arabic/English) humanoid receptionist/assistant on a
Unitree G1 (Jetson Orin NX, ROS 2 Foxy, Livox MID-360). This catalogs
**what's built today** (Part A) and **what's on the roadmap** (Part B).
---
# Part A — Current features (built & running)
Verified from the live subsystem registry (19 subsystems), dashboard tabs
(12), and API routers (22).
## 1. Voice & Conversation
- **Gemini live voice** — real-time bilingual AR/EN spoken conversation (native-audio model)
- **Offline brain** — local pipeline via `ollama` (`SANAD_VOICE_BRAIN=local`), no cloud
- **Wake phrases** — configurable wake-word manager
- **Typed replay** — type text, robot speaks it (with speaker-monitor capture)
- **Local TTS** — on-device text-to-speech engine
- **Prompt management** — edit the system prompt from the dashboard
- **Lip-sync** — mask mouth driven by TTS `MOUTH` markers
- **Barge-in** — interrupt speech (volume-scaled threshold)
## 2. Motion & Arm
- **35 arm actions** — 16 SDK built-ins + 19 custom JSONL motions
- **Macro record / playback** — capture and replay motion sequences
- **Teaching mode** — kinesthetic teach-and-repeat
- **Skills** — composed higher-level behaviors (skills.json)
- **Movement dispatch** — voice → motion (53 fixed + 10 parametric phrases, cooldown-gated)
- **Arm motion-block** — auto-inhibits arm moves while locomotion is active (safety interlock)
## 3. Locomotion
- **LocoClient + MotionSwitcher** — walk / pose control via Unitree SDK (eth0)
- **E-STOP** — dashboard kill button
- **Single Ctrl+C teardown** — one signal cleanly stops every subsystem (~2s)
## 4. LED Face Mask
- **Animated expressions** — neutral, smile, blink, look L/R, talk13, surprised, sad
- **Gestural-speaking events** — face reacts while speaking
- **Lip-sync** — mouth animates to speech
## 5. Vision & Recognition
- **Face recognition** — identify people via camera
- **Face gallery** — enroll/manage known faces
- **Zone gallery / zones** — visual zone recognition
- **Camera feed** — attached to the live voice subprocess (vision-in-the-loop)
## 6. Navigation (web_nav3 integration)
- **Live Map tab** — full embedded web_nav3 dashboard (set-pose, goals, bringup)
- **Navigation tab** — native canvas viewer (saved/live map, places, missions)
- **map_relay** — re-publishes the latched `/map` @1Hz so the map renders even when stationary
- **Saved maps** — load & view a pre-built `.db` (localize mode)
- **Places** — save named poses, one-click "Go"
- **Missions** — multi-waypoint routes (defined in web_nav3)
- **Cancel goal** — stop an active goal without tearing down bringup
- **SLAM** — RTABMap LiDAR-ICP, drift-corrected mapping/localization
## 7. Audio
- **Device manager** — sink/source selection, live refresh
- **Audio profiles** — builtin / anker / hollyland_builtin (auto-switch on plug/unplug)
## 8. Operations, System & Diagnostics
- **System control** — start/stop subsystems, status
- **Temperature monitor** — motor temps (live websocket stream)
- **Controller** — gamepad/teleop input
- **Web terminal** — shell in the browser (websocket)
- **Logs** — live log stream
- **Recordings & replay** — record/playback sessions
- **Scripts** — run saved scripts
## Dashboard infrastructure
- 12 tabs, **fault-isolated routers** (one broken module never breaks the dashboard)
- WebSocket streams: log_stream, motor_temps, terminal
- No-store HTML (no stale-cache 404s after deploy)
- Lazy subsystem imports (missing dep → that subsystem unavailable, rest runs)
---
# Part B — Roadmap (to add)
Tiers = priority. 🏗️ = load-bearing · ⚠️ = Foxy constraint.
## Tier 1 — Autonomous behaviors (the product)
1. **Voice-driven navigation** — "Sanad, go to the lobby" → nav goal
2. **Greeter mission** — recognized face → navigate → greet → express
3. **Named-person greeting** — identity → personalized line
4. **Patrol / guided tours** — ordered places, speech at each stop
5. **Return-to-base / dock-on-idle** — auto-home on idle/low battery
## Tier 2 — Navigation & map (harden + edit)
6. 🏗️ **Map republish relay** — ✅ DONE (map_relay)
7. **Click-to-goal on Nav tab canvas**
8. **Live nav telemetry** — distance/ETA/waypoint, "arrived" toast
9. **Battery + nav-state status bar**
10. **Geofence zones on the map**
11. **Cancel-goal button** — ✅ DONE
### Map editing & annotation (all build on #6)
12. **Erase tool** — paint cells free; wipe ghost obstacles + the SLAM "spokes"
13. **Obstacle paint ("black points" / virtual walls)** — ⚠️ Foxy-safe KeepoutFilter substitute
14. **Shape tools + brush size** — line/rectangle/polygon
15. **Non-destructive overlay + undo/redo**
16. **Persist & auto-reload edits per map**
17. **Crop / trim map bounds**
## Tier 3 — Voice & interaction
18. **Barge-in from dashboard**
19. **Quick-phrase soundboard**
20. **Conversation memory / visitor log**
21. **Per-speaker AR/EN auto-detect**
22. **Scheduled announcements**
23. **Bake edited map → PGM/YAML** (static map_server deploy)
## Tier 4 — Face & presence
24. **Gaze / head-track recognized face**
25. **Emotion-from-context** (sentiment → expression)
26. **Idle breathing / look-around**
27. **Lip-sync to TTS amplitude** (enhance existing markers)
## Tier 5 — Operator, fleet & reliability
28. 🏗️ **Global E-STOP button** — ✅ exists; surface consistently
29. **Health watchdog** — auto-restart dead subsystem + alert
30. **Per-subsystem enable/disable toggles**
31. **Behavior recorder → replay** (nav+voice timelines)
32. **Mission editor UI** (visual sequence builder)
33. **Remote access / tunnel**
34. **Reverse-proxy web_nav3 through :8001** — one origin, no iframe cross-port issues
## Tier 6 — Future / blocked
35. **Speed / caution zones** — needs Galactic SpeedFilter or custom layer
36. **Multi-robot fleet** (SanadV3 ↔ BotBrain) — needs LocoClient arbitration + coordinator
---
### Recommended next build order
**#1 voice→nav** → **#2 greeter mission** (the product), then **#12/#13 map editing**
(clean the spokes + virtual walls). #6 republish relay and #11 cancel are already done.

12
G1_Controller/__init__.py Normal file
View File

@ -0,0 +1,12 @@
"""G1_Controller — manual dashboard locomotion control (N2 Phase 1).
`LocoController` wraps the Unitree `LocoClient` + `MotionSwitcherClient` for
operator-driven walking, postures and a discrete step pad. It reuses the arm
controller's single process-wide DDS init (one `ChannelFactoryInitialize`) and
is gated behind an in-memory "Enable movement" arm flag that defaults OFF every
boot. See dashboard/routes/controller.py for the REST surface.
"""
from Project.Sanad.G1_Controller.loco_controller import LocoController
__all__ = ["LocoController"]

View File

@ -0,0 +1,593 @@
"""LocoController — manual G1 locomotion via the Unitree LocoClient (N2 Phase 1).
Ported from the proven scripts in G1_Lootah/Controller (g1_mode_controller.py,
keyboard_controller.py, hanger_boot_sequence.py). Design notes:
* **One DDS init per process.** The arm controller owns the single
`ChannelFactoryInitialize(0, nic)` (motion/arm_controller.py). This class
NEVER initialises DDS it lazily builds its `LocoClient` /
`MotionSwitcherClient` only after `arm._initialized` is True.
* **Default DISARMED.** `_armed` starts False every boot and gates every WRITE
method. Reads (status / fsm / joints), E-STOP and disarm are ALWAYS allowed.
* **StopMove watchdog.** Continuous `Move(..., True)` never self-terminates, so a
daemon thread StopMoves if no `move()` refresh arrives within
`watchdog_timeout_sec`. The frontend re-sends setpoints at ~10 Hz, so a tab
close / network drop trips the watchdog within the timeout.
* **Velocity caps.** Symmetric clamp on vx/vy/vyaw Walk 0.6, Run 1.2.
* **Allow-anytime-warn.** move/step never hard-block on FSM; if not walk-ready
they still execute but return a `warning`.
* **Sim fallback.** When `unitree_sdk2py` is absent (workstation), every write
returns `{"simulated": True}` (never raises) so the whole UI is testable.
SDK facts confirmed from source do not "fix" them:
* `LocoClient.Move(vx, vy, vyaw, True)` the continuous-mode kwarg is misspelled
`continous_move` (one n); we pass it POSITIONALLY to avoid a TypeError.
* `LocoClient` has NO StandUp()/Squat() use SetFsmId(4)/SetFsmId(2).
* FSM id / mode are read via the private RPC `bot._Call(7001/7002, "{}")`.
"""
from __future__ import annotations
import json
import threading
import time
from typing import Any, Optional
from Project.Sanad.core.config_loader import section as _cfg_section
from Project.Sanad.core.logger import get_logger
log = get_logger("loco_controller")
# -- SDK import (optional) -----------------------------------------------------
try:
from unitree_sdk2py.g1.loco.g1_loco_client import LocoClient
from unitree_sdk2py.comm.motion_switcher.motion_switcher_client import (
MotionSwitcherClient,
)
_HAS_SDK = True
except ImportError:
LocoClient = None
MotionSwitcherClient = None
_HAS_SDK = False
log.warning("Unitree SDK not available — LocoController in simulation mode")
# LocoClient general RPC api-ids for FSM read-back (stable across SDK builds).
ROBOT_API_ID_LOCO_GET_FSM_ID = 7001
ROBOT_API_ID_LOCO_GET_FSM_MODE = 7002
# G1 29-DoF joint names for indices 12-28 (0-11 legs, 12-14 waist, 15-21 left
# arm, 22-28 right arm). Used by the Diagnostics joint read-out.
JOINT_NAMES = {
12: "WAIST_YAW", 13: "WAIST_ROLL", 14: "WAIST_PITCH",
15: "L_SHOULDER_PITCH", 16: "L_SHOULDER_ROLL", 17: "L_SHOULDER_YAW",
18: "L_ELBOW", 19: "L_WRIST_ROLL", 20: "L_WRIST_PITCH", 21: "L_WRIST_YAW",
22: "R_SHOULDER_PITCH", 23: "R_SHOULDER_ROLL", 24: "R_SHOULDER_YAW",
25: "R_ELBOW", 26: "R_WRIST_ROLL", 27: "R_WRIST_PITCH", 28: "R_WRIST_YAW",
}
# Discrete step pad — (vx, vy, vyaw) sign per direction; magnitude is
# step_speed_frac * cap_walk (a gentle single step).
_STEP_DIRS = {
"forward": (1.0, 0.0, 0.0),
"backward": (-1.0, 0.0, 0.0),
"slide_left": (0.0, 1.0, 0.0),
"slide_right": (0.0, -1.0, 0.0),
"rotate_left": (0.0, 0.0, 1.0),
"rotate_right": (0.0, 0.0, -1.0),
}
_POSTURES = (
"zero_torque", "damp", "stand_up", "squat", "sit",
"low_stand", "high_stand", "lie_to_stand",
)
class LocoController:
"""Thread-safe manual locomotion control with a simulation fallback."""
def __init__(self, arm=None):
self._arm = arm # shared ArmController (owns the ONE DDS init)
self._bot = None # LocoClient (lazy)
self._msc = None # MotionSwitcherClient (lazy)
self._lc_ready = False
self._lock = threading.RLock() # serialise all loco client WRITE calls
self._armed = False # in-memory MANUAL gate — OFF every boot
self._cur_v = (0.0, 0.0, 0.0) # last commanded (vx, vy, vyaw)
self._teleop_active = False
self._last_msc_mode: Optional[str] = None
# watchdog
self._last_move_ts = 0.0
self._wd_thread: Optional[threading.Thread] = None
self._wd_stop = threading.Event()
self._wd_stop.set() # not running until armed
# Monotonic stop-generation counter, bumped under _lock by
# estop/stop/disarm. move()/step()/prep_mode() capture it at start and
# bail the instant it changes — so E-STOP preempts an in-flight motion
# immediately AND can never be silently "un-cancelled" by a concurrent
# command (a lock-free Event clear() could; an int compare under the
# lock cannot).
self._stop_gen = 0
# Serializes the discrete blocking operations (step/prep_mode) so two
# can't overlap and interleave Move commands. Continuous teleop move()
# is intentionally NOT guarded by this.
self._discrete_busy = False
cfg = _cfg_section("motion", "loco_controller")
self._cap_walk = float(cfg.get("cap_walk", 0.6))
self._cap_run = float(cfg.get("cap_run", 1.2))
self._lin_step = float(cfg.get("lin_step", 0.05))
self._ang_step = float(cfg.get("ang_step", 0.2))
self._wd_timeout = float(cfg.get("watchdog_timeout_sec", 0.5))
self._block_window = float(cfg.get("arm_block_window_sec", 1.5))
self._step_dur = float(cfg.get("step_duration_sec", 0.6))
self._step_frac = float(cfg.get("step_speed_frac", 0.5))
self._loco_timeout = float(cfg.get("loco_timeout_sec", 10.0))
self._msc_timeout = float(cfg.get("msc_timeout_sec", 5.0))
# ── client lifecycle ─────────────────────────────────────────────────────
def _ensure_client(self) -> bool:
"""Lazily build LocoClient + MotionSwitcherClient. Returns readiness.
Never initialises DDS requires the shared arm to have already run the
single ChannelFactoryInitialize.
"""
if not _HAS_SDK:
return False
if self._lc_ready:
return True
if self._arm is None or not getattr(self._arm, "_initialized", False):
return False
with self._lock:
if self._lc_ready:
return True
try:
bot = LocoClient()
bot.SetTimeout(self._loco_timeout)
bot.Init()
msc = MotionSwitcherClient()
msc.SetTimeout(self._msc_timeout)
msc.Init()
self._bot = bot
self._msc = msc
self._lc_ready = True
log.info("LocoClient + MotionSwitcherClient ready")
except Exception as exc:
log.error("LocoClient init failed: %s", exc)
self._lc_ready = False
return self._lc_ready
def _safe_call(self, name: str, fn, *a, **kw):
try:
return True, fn(*a, **kw)
except Exception as exc:
log.error("%s failed: %s", name, exc)
return False, None
def _rpc_get_int(self, api_id: int):
bot = self._bot
if bot is None:
return None
try:
code, data = bot._Call(api_id, "{}")
if code == 0 and data:
return json.loads(data).get("data")
except Exception:
pass
return None
@staticmethod
def _clamp(v: float, cap: float) -> float:
return max(-cap, min(cap, float(v)))
# ── FSM / readiness ──────────────────────────────────────────────────────
def fsm_id(self):
return self._rpc_get_int(ROBOT_API_ID_LOCO_GET_FSM_ID)
def fsm_mode(self):
return self._rpc_get_int(ROBOT_API_ID_LOCO_GET_FSM_MODE)
def _walk_ready_warning(self) -> Optional[str]:
"""allow-anytime-warn: None when ready, else a human message."""
if not self._lc_ready:
return None
fid = self.fsm_id()
fmode = self.fsm_mode()
if fid == 200 and fmode not in (None, 2):
return None
return (f"Robot not in walk-ready FSM (id={fid}, mode={fmode}). "
f"Command sent anyway.")
# ── arm flag + watchdog ──────────────────────────────────────────────────
def is_armed(self) -> bool:
return self._armed
def movement_active(self) -> bool:
"""True when the robot may be walking: manual armed, teleop active, OR a
move/step issued within the block window. Used as the arm's motion-block
predicate so the arm never replays while the robot is (or just was)
moving regardless of whether the MANUAL gate or the GEMINI gate
(Phase 3 voice dispatch, which calls move/step directly) triggered it."""
if self._armed or self._teleop_active:
return True
return (time.monotonic() - self._last_move_ts) < self._block_window
def arm_movement(self) -> dict:
"""Unlock manual control. Cancels any in-flight arm motion first so the
arm and locomotion are never active simultaneously (movement wins)."""
try:
if self._arm is not None and getattr(self._arm, "is_busy", False):
log.info("arming movement — cancelling in-flight arm motion")
self._arm.cancel()
except Exception:
log.exception("arm.cancel() on arm_movement failed")
with self._lock:
self._armed = True
self._start_watchdog()
log.info("movement ARMED")
return {"ok": True, "armed": True}
def disarm_movement(self) -> dict:
with self._lock:
self._stop_gen += 1 # break any in-flight step/prep/move
self._armed = False
self._teleop_active = False
self._wd_stop.set()
try:
self._raw_stop()
except Exception:
log.exception("StopMove on disarm failed")
log.info("movement DISARMED")
return {"ok": True, "armed": False}
def _start_watchdog(self):
self._wd_stop.clear()
if self._wd_thread is None or not self._wd_thread.is_alive():
self._wd_thread = threading.Thread(
target=self._watchdog_loop, daemon=True, name="loco-watchdog")
self._wd_thread.start()
def _watchdog_loop(self):
period = max(0.02, min(0.1, self._wd_timeout / 2.0))
while not self._wd_stop.is_set():
fire = False
park = False
# Read-and-decide under the lock (atomic check-then-act); the actual
# StopMove runs after release so the critical section stays tiny.
with self._lock:
if self._teleop_active and (time.monotonic() - self._last_move_ts) > self._wd_timeout:
self._teleop_active = False
fire = True
# Self-park once there's nothing left to guard. The Gemini
# dispatch path uses step() directly and never calls
# disarm_movement(), so without this the watchdog would spin for
# the rest of the process lifetime after the first voice step.
if not self._armed and not self._teleop_active and not self._discrete_busy:
park = True
if fire:
log.warning("watchdog: teleop setpoint stale (>%.2fs) — StopMove",
self._wd_timeout)
try:
self._raw_stop()
except Exception:
log.exception("watchdog StopMove failed")
if park:
# Nothing left to guard — stop the thread (a later move/step
# re-arms it via _start_watchdog()). Done AFTER any stale-stop
# above so we never skip a pending StopMove.
self._wd_stop.set()
break
self._wd_stop.wait(period)
def _raw_stop(self) -> bool:
"""Issue StopMove if the client is up; no-op in sim. Lock-light."""
if not self._lc_ready or self._bot is None:
return False
with self._lock:
ok, _ = self._safe_call("StopMove", self._bot.StopMove)
return ok
# ── movement ─────────────────────────────────────────────────────────────
def move(self, vx: float, vy: float, vyaw: float, run: bool = False) -> dict:
cap = self._cap_run if run else self._cap_walk
cvx, cvy, cvyaw = self._clamp(vx, cap), self._clamp(vy, cap), self._clamp(vyaw, cap)
capped = (cvx, cvy, cvyaw) != (float(vx), float(vy), float(vyaw))
warning = self._walk_ready_warning()
sent = {"vx": cvx, "vy": cvy, "vyaw": cvyaw}
with self._lock:
my_gen = self._stop_gen # capture under lock
if not self._ensure_client():
with self._lock: # sim: record intent for UI/watchdog
self._cur_v = (cvx, cvy, cvyaw)
self._last_move_ts = time.monotonic()
self._teleop_active = True
self._start_watchdog()
return {"ok": True, "sent": sent, "capped": capped,
"warning": warning, "simulated": True}
with self._lock:
# If an E-STOP / stop / disarm landed since we captured my_gen, do NOT
# (re)command velocity — and do NOT stamp the motion flags (so a
# cancelled tick doesn't extend the arm-block window).
if self._stop_gen != my_gen:
return {"ok": False, "cancelled": True, "sent": sent,
"capped": capped, "warning": warning, "simulated": False}
self._cur_v = (cvx, cvy, cvyaw)
self._last_move_ts = time.monotonic()
self._teleop_active = True
self._safe_call("SetBalanceMode", self._bot.SetBalanceMode, 1)
ok, _ = self._safe_call("Move", self._bot.Move, cvx, cvy, cvyaw, True)
self._start_watchdog()
return {"ok": bool(ok), "sent": sent, "capped": capped,
"warning": warning, "simulated": False}
def stop_move(self) -> dict:
"""Halt translation/rotation. Allowed even when disarmed."""
with self._lock:
self._stop_gen += 1
self._teleop_active = False
if not self._ensure_client():
return {"ok": True, "simulated": True}
ok = self._raw_stop()
return {"ok": bool(ok), "simulated": False}
def estop(self) -> dict:
"""Emergency stop = StopMove only (no Damp / FSM change → keeps posture).
ALWAYS allowed, even disarmed and in sim. Bumps the stop generation so any
in-flight move()/step()/prep_mode() bails immediately (no lock wait)."""
with self._lock:
self._stop_gen += 1
self._teleop_active = False
self._cur_v = (0.0, 0.0, 0.0)
if not self._ensure_client():
log.warning("E-STOP (sim)")
return {"ok": True, "simulated": True}
ok = self._raw_stop()
log.warning("E-STOP — StopMove issued")
return {"ok": bool(ok), "simulated": False}
def step(self, direction: str) -> dict:
"""Discrete one-step pad: Move for step_duration then StopMove.
Blocking (~step_duration); call via asyncio.to_thread from the route.
The sleep loop does NOT hold self._lock, so E-STOP / StopMove (which take
the lock briefly) preempt it immediately; the loop also bails the moment
the stop generation changes."""
if direction not in _STEP_DIRS:
return {"ok": False, "reason": f"unknown direction: {direction}"}
sx, sy, syaw = _STEP_DIRS[direction]
k = self._cap_walk * self._step_frac
vx, vy, vyaw = sx * k, sy * k, syaw * k
warning = self._walk_ready_warning()
with self._lock:
if self._discrete_busy:
return {"ok": False, "dir": direction, "reason": "busy",
"warning": warning, "simulated": not self._lc_ready}
self._discrete_busy = True
my_gen = self._stop_gen
self._last_move_ts = time.monotonic()
self._teleop_active = True
self._start_watchdog()
if not self._ensure_client():
with self._lock:
self._teleop_active = False
self._discrete_busy = False
return {"ok": True, "dir": direction, "warning": warning, "simulated": True}
try:
with self._lock:
if self._stop_gen != my_gen: # stopped before we began
return {"ok": False, "dir": direction, "cancelled": True,
"warning": warning, "simulated": False}
self._safe_call("SetBalanceMode", self._bot.SetBalanceMode, 1)
self._safe_call("Move", self._bot.Move, vx, vy, vyaw, True)
t_end = time.monotonic() + self._step_dur
while time.monotonic() < t_end:
if self._stop_gen != my_gen:
break
with self._lock:
self._last_move_ts = time.monotonic() # keep watchdog fed
time.sleep(0.05)
finally:
with self._lock:
self._safe_call("StopMove", self._bot.StopMove)
self._teleop_active = False
self._discrete_busy = False
return {"ok": True, "dir": direction, "warning": warning, "simulated": False}
# ── postures / modes ─────────────────────────────────────────────────────
def prep_mode(self) -> dict:
"""PREP — StopMove → Damp → StandUp(FSM4) → height ramp → BalanceStand(0).
Exact order from g1_mode_controller.prep_mode, minus the blocking input().
Blocking (~1s); call via asyncio.to_thread."""
if not self._ensure_client():
return {"ok": True, "mode": "prep", "simulated": True}
with self._lock:
if self._discrete_busy:
return {"ok": False, "mode": "prep", "reason": "busy", "simulated": False}
self._discrete_busy = True
my_gen = self._stop_gen
self._safe_call("StopMove", self._bot.StopMove)
self._safe_call("Damp", self._bot.Damp)
self._safe_call("SetFsmId(4)", self._bot.SetFsmId, 4)
try:
# Height ramp OUTSIDE the lock so E-STOP can preempt at any time.
h = 0.02
while h <= 0.5 + 1e-9:
if self._stop_gen != my_gen:
log.warning("PREP cancelled (E-STOP)")
return {"ok": False, "mode": "prep", "cancelled": True, "simulated": False}
with self._lock:
self._safe_call("SetStandHeight", self._bot.SetStandHeight, round(h, 3))
time.sleep(0.03)
h += 0.02
with self._lock:
self._safe_call("BalanceStand", self._bot.BalanceStand, 0)
self._safe_call("SetStandHeight", self._bot.SetStandHeight, 0.22)
finally:
with self._lock:
self._discrete_busy = False
log.info("PREP complete")
return {"ok": True, "mode": "prep", "simulated": False}
def ready_start_mode(self) -> dict:
"""READY = PREP then Start (FSM 200 / balance engaged)."""
self.prep_mode()
if not self._ensure_client():
return {"ok": True, "mode": "ready", "simulated": True}
with self._lock:
if hasattr(self._bot, "Start"):
ok, _ = self._safe_call("Start", self._bot.Start)
else:
ok, _ = self._safe_call("SetFsmId(200)", self._bot.SetFsmId, 200)
log.info("READY/START complete")
return {"ok": bool(ok), "mode": "ready", "simulated": False}
def posture(self, name: str) -> dict:
if name not in _POSTURES:
return {"ok": False, "reason": f"unknown posture: {name}"}
if not self._ensure_client():
return {"ok": True, "posture": name, "simulated": True}
bot = self._bot
with self._lock:
if name == "zero_torque":
ok, _ = self._safe_call("ZeroTorque", bot.ZeroTorque)
elif name == "damp":
ok, _ = self._safe_call("Damp", bot.Damp)
elif name == "stand_up":
ok, _ = self._safe_call("SetFsmId(4)", bot.SetFsmId, 4)
elif name == "squat":
ok, _ = self._safe_call("SetFsmId(2)", bot.SetFsmId, 2)
elif name == "sit":
ok, _ = self._safe_call("Sit", bot.Sit)
elif name == "low_stand":
ok, _ = self._safe_call("LowStand", bot.LowStand)
elif name == "high_stand":
ok, _ = self._safe_call("HighStand", bot.HighStand)
elif name == "lie_to_stand":
if hasattr(bot, "Lie2StandUp"):
ok, _ = self._safe_call("Lie2StandUp", bot.Lie2StandUp)
else:
ok, _ = self._safe_call("SetFsmId(702)", bot.SetFsmId, 702)
else: # unreachable (guarded above)
ok = False
return {"ok": bool(ok), "posture": name, "simulated": False}
def set_balance_mode(self, mode: int) -> dict:
if not self._ensure_client():
return {"ok": True, "balance_mode": int(mode), "simulated": True}
with self._lock:
ok, _ = self._safe_call("SetBalanceMode", self._bot.SetBalanceMode, int(mode))
return {"ok": bool(ok), "balance_mode": int(mode), "simulated": False}
def set_stand_height(self, h: float) -> dict:
if not self._ensure_client():
return {"ok": True, "height": float(h), "simulated": True}
with self._lock:
ok, _ = self._safe_call("SetStandHeight", self._bot.SetStandHeight, float(h))
return {"ok": bool(ok), "height": float(h), "simulated": False}
# ── MotionSwitcher ───────────────────────────────────────────────────────
def msc_check(self) -> dict:
if not self._ensure_client() or self._msc is None:
return {"mode_name": None, "simulated": not self._lc_ready}
try:
ret = self._msc.CheckMode()
name = None
if isinstance(ret, tuple) and len(ret) >= 2 and isinstance(ret[1], dict):
name = ret[1].get("name")
elif isinstance(ret, dict):
name = ret.get("name")
self._last_msc_mode = name
return {"mode_name": name}
except Exception as exc:
log.error("msc_check failed: %s", exc)
return {"mode_name": None}
def msc_select_ai(self) -> dict:
if not self._ensure_client() or self._msc is None:
return {"ok": True, "simulated": True}
with self._lock:
ok, _ = self._safe_call("SelectMode(ai)", self._msc.SelectMode, "ai")
return {"ok": bool(ok), "simulated": False}
def msc_release(self) -> dict:
if not self._ensure_client() or self._msc is None:
return {"ok": True, "simulated": True}
with self._lock:
ok, _ = self._safe_call("ReleaseMode", self._msc.ReleaseMode)
return {"ok": bool(ok), "simulated": False}
def reconnect(self) -> dict:
"""Drop and rebuild Loco + MSC clients (does NOT re-init the DDS factory)."""
with self._lock:
self._bot = None
self._msc = None
self._lc_ready = False
ok = self._ensure_client()
return {"ok": bool(ok), "lc_ready": self._lc_ready}
# ── reads ────────────────────────────────────────────────────────────────
def joints(self) -> dict:
q: list = []
try:
if self._arm is not None:
q = self._arm.get_current_q()
except Exception:
q = []
out = []
for idx in range(12, 29):
val = q[idx] if idx < len(q) else 0.0
out.append({"idx": idx, "name": JOINT_NAMES.get(idx, f"motor_{idx}"),
"q": float(val)})
return {"joints": out}
def status(self) -> dict:
# Polling /status lazily brings up the client once arm DDS is ready.
self._ensure_client()
fid = self.fsm_id() if self._lc_ready else None
fmode = self.fsm_mode() if self._lc_ready else None
walk_ready = bool(self._lc_ready and fid == 200 and fmode not in (None, 2))
return {
"sdk_available": _HAS_SDK,
"lc_ready": self._lc_ready,
"armed": self._armed,
"fsm_id": fid,
"fsm_mode": fmode,
"walk_ready": walk_ready,
"msc_mode": self._last_msc_mode,
"teleop_active": self._teleop_active,
"last_velocity": {"vx": self._cur_v[0], "vy": self._cur_v[1], "vyaw": self._cur_v[2]},
"caps": {"walk": self._cap_walk, "run": self._cap_run},
"arm_initialized": bool(self._arm is not None and getattr(self._arm, "_initialized", False)),
}
# ── shutdown helper ──────────────────────────────────────────────────────
def shutdown(self):
"""Best-effort StopMove + disarm for process shutdown.
Uses _raw_stop() (NOT estop()) so teardown never builds a brand-new
LocoClient: estop() _ensure_client() would lazily construct a client
and run bot.Init() (a DDS RPC) during interpreter teardown when we were
armed-but-never-built (Enable movement clicked, never moved, then
Ctrl+C). _raw_stop() no-ops when no client was ever created. Bump the
stop generation so any in-flight motion bails immediately."""
with self._lock:
self._stop_gen += 1
self._teleop_active = False
self._cur_v = (0.0, 0.0, 0.0)
try:
self._raw_stop() # no-op when _bot is None — never re-inits
except Exception:
log.exception("StopMove on shutdown failed")
finally:
self.disarm_movement()

412
README.md Normal file
View File

@ -0,0 +1,412 @@
# Sanad
Voice + motion assistant for the Unitree G1 humanoid. **Gemini Live** (or a
fully-offline pipeline) handles bilingual Arabic/English conversation; an arm
controller plays built-in SDK poses and recorded JSONL macros; a locomotion
controller walks/turns the robot; an optional camera feeds **Gemini-side face &
place recognition**; everything is orchestrated through a fault-isolated
**FastAPI dashboard** on `http://<robot>:8000`.
```
┌──────────────────────────────────────────────────────────────────────┐
│ Dashboard (FastAPI) ── http://<robot>:8000 │
│ ├─ Operations Quick-fire arm actions + gestural-speaking │
│ ├─ Voice & Audio Live Gemini, Typed Replay, Wake Phrases, Audio │
│ ├─ Motion & Replay SDK actions, JSONL replays, macros, teaching │
│ ├─ Controller Locomotion teleop, postures, FSM modes, E-STOP │
│ ├─ Recognition Camera vision + face gallery + zones/places │
│ ├─ Recordings Skill registry, saved Gemini turns │
│ ├─ Temperature Live 3D motor-temperature heatmap (three.js) │
│ ├─ Terminal In-browser shell (PTY) to the robot │
│ └─ Settings & Logs System info, tail/stream live logs │
└──────────────────────────────────────────────────────────────────────┘
├─ voice/sanad_voice.py (subprocess — model-agnostic voice loop)
│ ├─ gemini/script.py (Gemini Live brain — audio+video+state)
│ └─ local/script.py (offline brain — VAD→STT→LLM→TTS)
├─ gemini/client.py (short-session client for Typed Replay)
├─ gemini/subprocess.py (spawns+supervises sanad_voice.py;
│ pushes camera frames + motion state
│ to the child over its stdin)
├─ voice/movement_dispatch.py(Gemini spoken phrase → locomotion)
├─ vision/camera.py (RealSense/USB capture daemon)
├─ vision/face_gallery.py (data/faces/ CRUD for the primer turn)
├─ vision/zone_gallery.py (data/zones/ places + "go here" targets)
├─ motion/arm_controller.py (G1 arm DDS publisher — owns DDS init)
├─ G1_Controller/loco_controller.py (G1 locomotion via LocoClient)
├─ voice/audio_io.py (mic + speaker abstraction — 3 profiles)
└─ core/brain.py (skill dispatcher, event bus)
```
### Camera + face/place recognition data flow
```
CameraDaemon (parent, in-memory JPEG+b64 cache)
├─→ dashboard /api/recognition/frame.jpg ── snapshot_jpeg()
└─→ GeminiSubprocess._frame_forwarder ── get_frame_b64()
│ "frame:<b64>\n" over stdin
ArmController ─emit→ event bus ─→ main.py ─→ live_sub.send_state()
│ "state:<json>\n" over stdin
gemini/script.py _stdin_watcher thread
├─ frame: → _LATEST_FRAME → _send_frame_loop →
│ session.send_realtime_input(video=Blob)
└─ state: → _STATE_PENDING → _send_state_loop →
session.send_realtime_input(text=…)
Recognition toggles (vision / face-rec / zone-rec / movement) are written by the
dashboard to data/.recognition_state.json and POLLED by the Gemini child at 1 Hz
— so flipping a toggle takes effect mid-session with NO restart.
```
## Quick start (on the robot)
```bash
conda activate gemini_sdk
cd ~/Sanad
python3 main.py
```
Then open `http://<robot-ip>:8000` in a browser. (The dashboard binds to the
`wlan0` IP by default — see *Runtime selection* to override.)
Fully-offline brain (no cloud): `SANAD_VOICE_BRAIN=local python3 main.py`
(requires `ollama serve` + the local model env — see *Voice brains*).
> **Gemini API key — required, none ships with the repo.** The `api_key`
> fields in `config/core_config.json` (`gemini_defaults`) and
> `data/motions/config.json` (`gemini`) are intentionally empty (`""`).
> The voice loop cannot connect until you supply one, by any of:
> - **Dashboard***Voice & Audio → Gemini API Key* — paste + save, hot-swaps live (no restart). Persists to `data/motions/config.json`.
> - **Env var**`export SANAD_GEMINI_API_KEY=AIza...` before `python3 main.py`.
> - **Config file** — set `gemini_defaults.api_key` in `config/core_config.json`.
>
> Precedence (highest first): `data/motions/config.json``SANAD_GEMINI_API_KEY``config/core_config.json`. Get a key at <https://aistudio.google.com/apikey>.
## Dashboard features
### Operations
Quick-fire SDK + JSONL arm actions (chip buttons), gestural-speaking toggle.
### Voice & Audio
- **Live Voice Commands** — fire arm gestures from the *user's* transcript
(wake-phrase → arm action). Master gate + Deferred-trigger toggle.
- **Live Gemini Process** — start/stop the voice conversation subprocess, tail
its log. Choose the Gemini cloud brain or the offline brain via
`SANAD_VOICE_BRAIN`.
- **Typed Replay** — Gemini reads typed text aloud (wrapped with a
"repeat verbatim" prompt); optionally records the clip.
- **Gemini API Key** — hot-swap the key without restart.
- **Wake Phrase Manager** — add/remove phrase → action bindings.
- **Audio Controls** — mic/speaker mute, G1 chest-speaker volume (DDS), device
profile selection, PulseAudio soft-reset and Anker USB hard-reset.
### Motion & Replay
- **Motion Control** — list SDK (built-in) + JSONL (recorded) actions, select +
play. Cancel smoothly returns to `arm_home.jsonl`.
- **Replay Manager** — upload `.jsonl` files, test-play with speed, Teaching
Mode (kinesthetic record — limp the arm and hand-guide it).
- **Macro Recorder** — record a new audio+motion pair, OR pick any WAV + any
motion (SDK or JSONL) and play them in parallel.
### Controller *(locomotion)*
Manual teleoperation of the G1's **legs** via the Unitree `LocoClient`.
**Disarmed every boot**; all motion writes require Arm first.
- **Move / Step** — continuous teleop (vx/vy/vyaw) or discrete one-shot steps.
- **Postures & FSM modes** — zero-torque, damp, squat, sit, stand, balance,
stand-height; prep/ready sequences; MotionSwitcher select-AI/release.
- **Gemini Movement** — toggle voice-driven walking: the `MovementDispatcher`
parses Gemini's *own spoken confirmation phrases* ("Turning right." /
"أستدير يميناً.") and drives the legs (gated on this toggle + an E-STOP latch).
- **E-STOP** — always available; `StopMove` + disarm + latch the dispatcher.
> **Safety:** the arm and locomotion are **mutually exclusive**
> `arm.set_motion_block(loco.movement_active)` makes every arm
> replay/gesture refuse while the robot is (or just was, within ~1.5 s) walking.
### Recognition
Camera vision + Gemini-side **face** and **zone/place** recognition. All are
**off by default**; each is a **hot toggle** (≈1 s to take effect, no restart).
- **Camera Vision**`CameraDaemon` captures from a RealSense (preferred) or
USB camera; the supervisor streams JPEG frames to Gemini Live so it can answer
"what do you see?". Live preview panel. Auto-reconnects on USB unplug/stall
and warns if a RealSense negotiated USB 2.0 (Marcus-ported resilience).
- **Face Recognition** — manage `data/faces/face_{id}/` galleries: enroll from
the live camera or upload photos, rename, describe, download (per-photo or
ZIP), delete. On session start (and on any gallery change) the child sends a
**primer turn** carrying every enrolled face + a Khaleeji greeting
instruction — **Gemini matches in-context, so there is no local
face-recognition model**. Recognition needs vision on.
- **Zones & Places**`data/zones/zone_{zid}/place_{pid}/` two-level gallery:
reference photos per place, optional linked face_ids, and a **"go here"** nav
target (`nav_target_zone/place_id` in the recognition-state file) for
place-aware navigation.
- **Sync Gallery** — force-resend the face/zone primer to the live session.
### Recordings
Skill Registry (predefined audio+motion+callback skills from `skills.json`) +
Saved Records (captured Gemini turn recordings; play/pause/stop/rename/delete).
### Temperature
Live **3D motor-temperature heatmap** — a standalone three.js viewer
(`dashboard/static/temp3d/`) loads the G1 29-DoF URDF + STL meshes and colors
each joint blue→red from the arm controller's throttled `rt/lowstate` snapshot,
streamed over `/ws/motor-temps` at ~8 fps. No second DDS subscriber.
### Terminal
In-browser **PTY shell** to the robot (`/ws/terminal`, xterm.js) — a `bash -i`
as the dashboard's user, with resize + backpressure, bounded to 4 sessions.
(See *Security* — this is full shell access to whoever reaches the URL.)
### Settings & Logs
System info (host, network interfaces, DDS interface, bound dashboard host/port,
per-subsystem status, audio devices), live log stream (`/ws/logs`), per-file
tail, snapshot, and a one-blob "Copy All Logs" bundle.
## Directory layout
| Path | Contents |
|---|---|
| `main.py` | Entry point — fault-isolated boot of all subsystems + the dashboard. Doubles as the service container (route handlers `import` its module globals). |
| `config.py` | Runtime constants + layout-agnostic path resolution; layers `data/motions/config.json` over the JSON config at import. |
| `config/` | Per-subsystem JSON: `core`, `voice`, `gemini`, `local`, `motion`, `dashboard`. |
| `core/` | `brain.py` (skill dispatcher), `event_bus.py`, `skill_registry.py`, `config_loader.py`, `logger.py` (rotating + WS push), `asyncio_compat.py` (3.8 `to_thread` shim). |
| `gemini/` | Gemini Live — `client.py` (one-shot), `script.py` (live brain: audio + video + motion-state), `subprocess.py` (supervisor + stdin frame/state push). |
| `local/` | Fully-offline brain — `vad.py` (Silero), `stt.py` (faster-whisper), `llm.py` (Qwen via Ollama/llama.cpp), `tts.py` (CosyVoice2), `script.py` (the brain), `subprocess.py` (supervisor). Opt-in via `SANAD_VOICE_BRAIN=local`. |
| `voice/` | `sanad_voice.py` (subprocess entry, model-agnostic), `audio_io.py` / `audio_manager.py` / `audio_devices.py` (mic/speaker), `local_tts.py` (SpeechT5 Arabic TTS), `live_voice_loop.py` (user-transcript → arm gesture), `movement_dispatch.py` (Gemini-phrase → locomotion), `typed_replay.py`, `wake_phrase_manager.py`, `text_utils.py` (Arabic normalization + phrase matching), `model_script.py` / `model_subprocess.py` (brain templates). |
| `motion/` | `arm_controller.py` (production 5-phase JSONL replay engine, owns the single DDS init), `macro_player.py`, `macro_recorder.py`, `teaching.py`. (`sanad_arm_controller.py` is a legacy alternate — not wired by `main.py`.) |
| `G1_Controller/` | `loco_controller.py` — locomotion via Unitree `LocoClient` (move/step/postures/FSM/E-STOP); reuses the arm's DDS participant. |
| `vision/` | `camera.py` (RealSense/USB daemon, auto-reconnect), `face_gallery.py`, `zone_gallery.py`, `recognition_state.py` (atomic-JSON toggle IPC). |
| `dashboard/` | `app.py` (FastAPI factory + fault-isolated router registration), `routes/*.py` (20 REST routers), `websockets/*.py` (logs, motor-temps, terminal), `static/index.html` (single-page UI), `static/temp3d/` (3D viewer). |
| `scripts/` | Persona files — `sanad_script.txt` (voice persona "Bousandah"), `sanad_rule.txt`, `sanad_arm.txt` (voice→arm phrases). |
| `data/` | Runtime state — `motions/*.jsonl` (arm trajectories) + `instruction.json` (locomotion phrase map) + `skills.json` + `config.json` (dashboard-editable), `recordings/` (captured turns + macros), `faces/face_{id}/` + `zones/zone_{zid}/place_{pid}/` (galleries), `audio/` (typed-replay WAVs + records index), `.recognition_state.json` (toggle IPC). |
| `model/` | Local SpeechT5 / Whisper / CosyVoice2 weights when using the offline pipeline. |
| `logs/` | Per-module rotating logs. |
## Voice brains
The child `voice/sanad_voice.py` is model-agnostic and selects a brain via
`SANAD_VOICE_BRAIN`. Every brain implements the same contract
(`__init__(audio_io, recorder, voice, system_prompt)`, `async run()`, `stop()`)
and ships a sibling supervisor that spawns the child and parses its
`USER:` / `BOT:` / state log markers.
| Value | Brain | Pipeline |
|---|---|---|
| `gemini` *(default)* | `gemini/script.py` | Gemini Live native-audio (full-duplex speech-to-speech, server-side VAD, vision frames, face/zone primers, voice→movement). Cloud. |
| `local` | `local/script.py` | Silero VAD → faster-whisper (large-v3-turbo, CUDA int8) → Qwen2.5 (Ollama/llama.cpp) → CosyVoice2 streaming TTS. Fully on-device. |
| `model` | `voice/model_script.py` | Template/stub for adding a new provider (OpenAI Realtime, Claude Voice, …). |
To add a brain: drop a file in `voice/` or a new `<brand>/` folder and add a
branch to `voice/sanad_voice.py:_build_brain()`; ship a supervisor modeled on
`voice/model_subprocess.py`.
## Runtime selection (env vars)
| Var | Values | Default | Effect |
|---|---|---|---|
| `SANAD_VOICE_BRAIN` | `gemini`, `local`, `model` | `gemini` | Which brain the subprocess loads (see `voice/sanad_voice.py:_build_brain`). |
| `SANAD_AUDIO_PROFILE` | `builtin`, `anker`, `hollyland_builtin` | `builtin` | Mic + speaker pair. `builtin` = G1 UDP mic + G1 chest speaker via DDS. |
| `SANAD_DDS_INTERFACE` | network iface | `eth0` | DDS network for G1 low-level comms (arm + locomotion + speaker). |
| `SANAD_DASHBOARD_HOST` / `_INTERFACE` | IP / iface | `wlan0` IP | Dashboard bind address. |
| `SANAD_GEMINI_API_KEY` | string | `""` (empty) | Gemini API key. No key ships in the repo — set this, paste one in the dashboard (**Voice & Audio → Gemini API Key**), or fill `gemini_defaults.api_key` in `config/core_config.json`. See [Quick start](#quick-start-on-the-robot). |
| `SANAD_GEMINI_MODEL` / `_VOICE` | string | reads config | Override the Gemini model id / prebuilt voice. |
| `SANAD_G1_VOLUME` | `0``100` | `100` | G1 chest-speaker volume; also scales the barge-in threshold. |
| `SANAD_LIVE_SCRIPT` | path | auto | Override the subprocess entry script path. |
| `SANAD_RECORD` | `0` or `1` | `1` | Record every Gemini turn to `data/recordings/`. |
| `SANAD_AEC_ENABLE` | `0` or `1` | `1` | Enable WebRTC AEC3 (if the Python binding is installed). |
| `SANAD_VISION_ENABLE` | `0` or `1` | `0` | Boot default for camera vision. **Runtime truth is the Recognition-tab toggle**`data/.recognition_state.json`, hot-applied without a restart. |
| `SANAD_FACE_RECOGNITION_ENABLE` | `0` or `1` | `0` | Boot default for Gemini-side face recognition. Also a hot toggle. |
| `SANAD_VISION_SEND_HZ` | float | `2` | Frames/sec the Gemini child relays to Live. |
| `SANAD_CAMERA_WIDTH` / `_HEIGHT` / `_FPS` | int | `424` / `240` / `15` | Capture profile. Also settable per-deploy in `config/core_config.json > camera`. |
| `SANAD_CAMERA_USB_INDEX` | int | auto | Pin a `/dev/videoN` node (avoids picking a RealSense IR stream). |
| `SANAD_FACES_MAX_SAMPLES` | int | `3` | Max photos per person fed into the gallery primer turn (token budget). |
| `SANAD_PROJECT_ROOT` | path | auto | Override the project root (see *Dynamic paths*). |
> All `SANAD_VISION_*` / `SANAD_CAMERA_*` / `SANAD_FACE_*` vars are **boot
> defaults** forwarded to the Gemini child via `LIVE_TUNE`. Once running, the
> Recognition tab's toggles (vision / face-rec / zone-rec / movement) are the
> live source of truth in `data/.recognition_state.json`, polled at 1 Hz.
CLI flags: `python3 main.py --host <ip> --port 8000 --network <dds_iface>`;
`--check-env` prints a subsystem/environment diagnostic and exits.
## API surface
All routes are registered defensively — a router whose import fails is recorded
(`GET /api/_dashboard_status`) and the server still boots without it.
**REST** (prefix → controls): `/api` health · `/api/system` info ·
`/api/voice` Gemini/local generate+connect+key · `/api/motion` arm actions ·
`/api/skills` skill registry · `/api/macros` record/play · `/api/replay` JSONL
CRUD + teaching · `/api/audio` mute/volume/devices/reset · `/api/scripts`
persona files · `/api/records` saved WAVs · `/api/prompt` system prompt ·
`/api/wake-phrases` bindings · `/api/live-voice` arm-phrase dispatcher ·
`/api/live-subprocess` Gemini child · `/api/typed-replay` TTS · `/api/recognition`
vision + face gallery · `/api/zones` zones/places + nav target · `/api/temp`
motor map + snapshot · `/api/controller` locomotion (move/step/postures/modes/
E-STOP).
**WebSockets**: `/ws/logs` (live log stream + 500-line replay) ·
`/ws/motor-temps` (3D heatmap data, ~8 fps) · `/ws/terminal` (PTY shell).
## Architecture notes
- **Subprocess isolation**: `voice/sanad_voice.py` runs as a child of `main.py`
via the supervisor. If the voice loop crashes, the dashboard + arm + legs stay
up.
- **Single DDS init**: `motion/arm_controller.py` owns the one
`ChannelFactoryInitialize`; `LocoController` and the audio routes reuse that
participant rather than re-initializing.
- **Brain contract**: see `voice/model_script.py` — any new model implements
`__init__(audio_io, recorder, voice, system_prompt)`, `async run()`, `stop()`.
- **Supervisor contract**: each brain ships a sibling supervisor (e.g.
`gemini/subprocess.py`) that spawns `sanad_voice.py` with its
`SANAD_VOICE_BRAIN` and parses the brain's log markers. Template:
`voice/model_subprocess.py`.
- **Locomotion safety**: `LocoController` is disarmed every boot, has velocity
caps + a `StopMove` watchdog, and is mutually exclusive with the arm.
Voice-driven movement is **off by default** and gated by the Controller
toggle. Distances/degrees in `data/motions/instruction.json` are
**approximate and must be calibrated on the real robot** — there is no
obstacle/abort stack.
- **Audio routing**: the G1's platform-sound PulseAudio sink is NOT wired to a
physical speaker. All dashboard-triggered playback (`play_wav`, typed-replay
audio, record playback) routes through DDS `AudioClient.PlayStream` via
`audio_manager._play_pcm_via_g1`. The PyAudio path is a desktop/dev fallback.
- **Arm replay**: `motion/arm_controller.py:_replay_file_inner()` is a port of
`G1_Lootah/Manual_Recorder/g1_replay_v4_stable.py:Run()` — ramp-in → settle
hold → playback → smooth return → disable SDK. Body motors (014) lock to a
live snapshot while arm motors (1528) follow the file at 60 Hz. `_return_home()`
runs unconditionally after a cancel for a jerk-free return.
- **Camera frame transport (stdin push)**: the `CameraDaemon` lives in the
parent and caches frames in memory. `GeminiSubprocess` base64-encodes the
latest frame to the child's stdin (~2 fps); the child's `_stdin_watcher`
relays it to Gemini Live with a staleness guard. Chosen over a file drop so
the parent owns the camera once and the dashboard preview reads the same cache.
- **Motion-state channel**: `arm_controller._execute()` emits
`motion.action_started` / `_done` / `_error` on the event bus. `main.py`
forwards each to the child as `state:<json>\n`, injected to Gemini Live as
silent `[STATE-START] wave_hand` / `[STATE-DONE] wave_hand (2.3s)` text so it
can honestly answer "what are you doing?".
- **Recognition is Gemini-side**: no dlib/insightface/onnxruntime. Galleries are
pure file IO; `gemini/script.py:_send_gallery_primer()` builds one multimodal
`send_client_content` turn — every enrolled face/place's photos + a greeting
instruction — and Gemini matches incoming frames against it in-context.
## Camera vision on Jetson
The Recognition tab needs `pyrealsense2` to talk to the Intel RealSense.
**Do not `pip install pyrealsense2` on JetPack 5** — the PyPI wheel is built
against glibc 2.32+ (Ubuntu 22.04) and fails to load on JetPack 5's glibc
2.31 with `ImportError: ... version 'GLIBC_2.32' not found`.
The native runtime is already there (`apt`-installed `librealsense2`). Build
just the Python binding from source against it, into the `gemini_sdk` env:
```bash
rs-enumerate-devices # confirm the D435I shows up at OS level first
source ~/miniconda3/etc/profile.d/conda.sh && conda activate gemini_sdk
pip uninstall -y pyrealsense2 # remove the broken wheel if present
sudo apt install -y cmake build-essential git python3-dev libusb-1.0-0-dev pkg-config libssl-dev
cd /tmp && rm -rf librealsense
git clone --depth=1 --branch v2.56.5 https://github.com/IntelRealSense/librealsense.git
cd librealsense && mkdir -p build && cd build
cmake .. -DBUILD_PYTHON_BINDINGS=ON -DPYTHON_EXECUTABLE=$(which python3) \
-DBUILD_EXAMPLES=OFF -DBUILD_GRAPHICAL_EXAMPLES=OFF \
-DBUILD_UNIT_TESTS=OFF -DCHECK_FOR_UPDATES=OFF -DCMAKE_BUILD_TYPE=Release
make -j$(nproc) pyrealsense2
SITE=$(python3 -c "import sysconfig; print(sysconfig.get_paths()['purelib'])")
mkdir -p "$SITE/pyrealsense2"
cp wrappers/python/pyrealsense2*.so "$SITE/pyrealsense2/"
cp ../wrappers/python/pyrealsense2/__init__.py "$SITE/pyrealsense2/" 2>/dev/null || true
python3 -c 'import pyrealsense2 as rs; print([d.get_info(rs.camera_info.name) for d in rs.context().query_devices()])'
```
Match the `--branch` tag to the installed runtime (`dpkg -l | grep librealsense2`).
If the build isn't worth it, `CameraDaemon` falls back to `cv2.VideoCapture(0)`
automatically — fine for a plain USB webcam, but note a RealSense exposes its
*depth* stream at `/dev/video0`, not RGB, so a real USB cam is the cleaner
fallback (or pin `SANAD_CAMERA_USB_INDEX`). On x86_64 / Ubuntu 22.04+ desktops,
`pip install pyrealsense2` just works.
## Dynamic paths
Every path is derived at runtime — no hard-coded `/home/...` anywhere.
Resolution order for `BASE_DIR` in `config.py`:
1. `SANAD_PROJECT_ROOT` env var (if set).
2. `PROJECT_BASE + PROJECT_NAME` from a `.env` file in `Sanad/` or its parent.
3. `Path(__file__).resolve().parent` — auto-detected.
The project runs unchanged from either layout:
- dev: `<anywhere>/Project/Sanad/`
- deployed: `/home/unitree/Sanad/`
## Deployment (workstation → robot)
```bash
rsync -av --delete \
--exclude=__pycache__ --exclude=logs --exclude=model --exclude=.git \
/path/to/Sanad/ \
unitree@192.168.123.164:/home/unitree/Sanad/
```
Then on the robot: `Ctrl+C` the running `main.py` and re-run.
## Security
The dashboard has **no authentication**. Anyone who can reach
`http://<robot>:8000` gets full robot control — locomotion, arm, audio, file
upload/delete — and, via the **Terminal tab**, an interactive shell as the
dashboard's user. Bind it to a **trusted LAN only**; add auth before any wider
exposure.
## Troubleshooting
| Symptom | Fix |
|---|---|
| `No LowState received in 2s — refusing to replay` | `main.py` was re-executed as both `__main__` and `Project.Sanad.main`, creating two arm instances. Fix lives in the `sys.modules` alias near the top of `main.py`. Restart. |
| `G1ArmActionClient not available — skipping` for SDK actions | Same duplicate-init issue as above. |
| `No module named 'Project'` in subprocess | Bootstrap preamble in `voice/sanad_voice.py:~30` synthesises the `Project.Sanad` namespace when run as `__main__`. |
| Controller moves rejected (409) | The Controller is **disarmed by default** — hit Arm first. Reads + E-STOP are always allowed. |
| Arm action refused while "movement armed" | Arm ↔ locomotion are mutually exclusive. Disarm/stop locomotion, then trigger the arm. |
| Voice-driven walking does nothing | "Gemini Movement" toggle off, or E-STOP latched. Toggle on; clear E-STOP. Distances are uncalibrated. |
| Arm jumps at start of JSONL replay | `SETTLE_HOLD_SEC` (in `config/motion_config.json > arm_controller`) too low — try `0.7` or `1.0`. |
| Record playback silent | `audio_mgr.play_wav` only routes to G1 DDS if the Unitree SDK is importable; on desktop it falls back to the PulseAudio sink. |
| Live Voice Commands transcript stuck | Deferred trigger was queued but `trigger_enabled` toggle was off. Toggle on — or the pending-trigger poll fires it automatically once enabled. |
| Gemini "no audio" on Typed Replay | Non-deterministic; the retry chain in `voice/typed_replay.py:generate_audio` tries three prompt variants. For reliable TTS, use the offline `local_tts` SpeechT5 path. |
| Local brain exits immediately | `ollama serve` not running / model not pulled, or weights missing under `model/`. Check `logs/local_subprocess.log`. The Gemini brain is the safe default. |
| Recognition tab: "Camera could not start (no backend)" | No camera backend acquired. Check `rs-enumerate-devices` (RealSense at OS level) and `python3 -c 'import pyrealsense2'` in the `gemini_sdk` env. The glibc `ImportError` means the pip wheel is incompatible — see "Camera vision on Jetson" above. |
| Camera badge stuck on "reconnecting…" | `CameraDaemon` lost the device and is retrying with exponential backoff. Re-seat the USB 3 cable; check `logs/camera.log` for the USB-2.0 warning. |
| Gemini doesn't greet an enrolled face | Face Recognition toggle on? Vision on? (Face rec needs frames.) Check `logs/gemini_brain.log` for `face gallery primed: N person(s)`. Hit "Sync Gallery" to force a re-prime. |
| Gemini unaware of motion state | The `motion.action_*``send_state` chain only runs when Live Gemini is up. Check `logs/gemini_subprocess.log` and `logs/gemini_brain.log` for `STATE injected:` lines. |
## License / attribution
Internal project for YS Lootah Technology. Reuses/ports patterns from:
- `G1_Lootah/Manual_Recorder/g1_replay_v4_stable.py` (arm replay math)
- `SanadVoice/gemini_interact` (arm-phrase dispatch, skill registry)
- `SanadVoice/gemini_voice_v2` (local SpeechT5 TTS)
- `Project/Marcus` — camera→Gemini stdin-push transport, motion-state
injection, camera daemon resilience (auto-reconnect, USB-2.0 warning), the
`API/camera_api.py` cache shape (`get_frame_b64` / `get_fresh_frame`), and the
confirmation-phrase → locomotion pattern (`movement_dispatch`).
- Unitree `unitree_sdk2py` (G1 low-level SDK, `LocoClient`, `G1ArmActionClient`,
`AudioClient.PlayStream`).

View File

@ -271,7 +271,9 @@ def _resolve_dashboard_host() -> str:
DASHBOARD_HOST = _resolve_dashboard_host()
DASHBOARD_PORT = 8000
# Canonical SanadV3 port (matches shell_scripts/start_all.sh + docs). The
# legacy Sanad ran on :8000; SanadV3 is :8001 to never collide with it.
DASHBOARD_PORT = 8001
# -- Local TTS --
LOCAL_TTS_MODEL = "MBZUAI/speecht5_tts_clartts_ar"
@ -341,12 +343,35 @@ LIVE_TUNE: dict[str, str] = {
# G1 built-in mic — UDP multicast 239.168.123.161:5555.
# Requires wake-up conversation mode ON in Unitree app.
"SANAD_USE_G1_MIC": "1",
# ── Recognition (camera vision + face recognition) ──
# All of these are BOOT defaults. The runtime source of truth is the
# state file data/.recognition_state.json — toggled live from the
# Recognition tab and polled by the Gemini child at 1 Hz.
"SANAD_VISION_ENABLE": "0",
"SANAD_VISION_SEND_HZ": "2",
"SANAD_VISION_STALE_MS": "1500",
"SANAD_CAMERA_WIDTH": "424",
"SANAD_CAMERA_HEIGHT": "240",
"SANAD_CAMERA_FPS": "15",
"SANAD_CAMERA_JPEG_QUALITY": "70",
"SANAD_FACE_RECOGNITION_ENABLE": "0",
"SANAD_FACES_DIR": str(DATA_DIR / "faces"),
"SANAD_FACES_MAX_SAMPLES": "3",
"SANAD_FACES_PRIMER_RESIZE": "256",
"SANAD_RECOGNITION_STATE_PATH": str(DATA_DIR / ".recognition_state.json"),
"SANAD_RECOGNITION_POLL_S": "1.0",
}
# -- Camera --
CAMERA_SERVICE_PORT = 8091
DIRECT_CAMERA_URL = f"http://127.0.0.1:{CAMERA_SERVICE_PORT}"
# -- Navigation (web_nav3 / rosbridge) --
WEB_NAV3_URL = os.environ.get("WEB_NAV3_URL", "http://127.0.0.1:8765")
ROSBRIDGE_URL = os.environ.get("ROSBRIDGE_URL", "ws://127.0.0.1:9090")
NAV_ROBOT_NAME = os.environ.get("NAV_ROBOT_NAME", "sanad")
# -- DDS / hardware --
# Jetson G1 default is eth0 (the robot's internal network).
# Override with SANAD_DDS_INTERFACE=lo for desktop/sim development.

View File

@ -36,12 +36,12 @@
"gemini_defaults": {
"_comment": "Baseline Gemini API config — SINGLE SOURCE OF TRUTH. All voice modules read from here.",
"api_key": "AIzaSyDt9Xi83MDZuuPpfwfHyMD92X7ZKdGkqf8",
"api_key": "",
"model_live": "gemini-2.5-flash-native-audio-preview-12-2025",
"model_ws_uri": "wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1alpha.GenerativeService.BidiGenerateContent",
"voice_name": "Charon",
"ws_timeout_sec": 30,
"default_system_prompt": "You are Sanad (Bousandah), a wise and friendly Emirati assistant. Speak strictly in the UAE dialect (Khaleeji). Be helpful, concise, and use local greetings like 'Marhaba' and 'Ya Khoy'."
"default_system_prompt": "You are Bousandah, a wise and friendly Emirati assistant. Speak strictly in the UAE dialect (Khaleeji). Be helpful, concise, and use local greetings like 'Marhaba' and 'Ya Khoy'."
},
"g1_hardware": {
@ -76,5 +76,26 @@
"dds": {
"network_interface_default": "eth0"
},
"camera": {
"_comment": "Recognition tab camera daemon (parent process reads this). width/height/fps/jpeg_quality + the reconnect knobs configure CameraDaemon. Frames are cached in memory and pushed to the Gemini child over its stdin (no file drop). send_hz/stale_ms are read by the Gemini child via SANAD_VISION_SEND_HZ / SANAD_VISION_STALE_MS env vars (LIVE_TUNE).",
"width": 424,
"height": 240,
"fps": 15,
"jpeg_quality": 70,
"send_hz": 2,
"stale_ms": 1500,
"stale_threshold_s": 10.0,
"reconnect_min_s": 2.0,
"reconnect_max_s": 10.0,
"capture_timeout_ms": 5000
},
"faces": {
"_comment": "Face gallery for Gemini-side recognition. Folder layout: data/faces/face_{id}/{face_1.jpg, ...} + optional meta.json {\"name\": \"...\"}. Gemini does the matching — no local ML model.",
"dir_rel": "data/faces",
"max_samples_per_face": 3,
"primer_resize_long_side": 256
}
}

35
config/gemini_config.json Normal file
View File

@ -0,0 +1,35 @@
{
"_description": "Tunables for gemini/* modules. Loaded via core.config_loader.load('gemini'). API credentials (api_key, model, voice_name) still live in core_config.json > gemini_defaults — single source of truth shared with config.py.",
"client": {
"_comment": "gemini/client.py — short-session WebSocket client used by dashboard /generate + typed replay. default_system_prompt comes from core.gemini_defaults.",
"recv_timeout_sec": 30,
"reconnect_max_attempts": 3,
"reconnect_initial_delay_sec": 1.0,
"reconnect_max_delay_sec": 10.0
},
"subprocess": {
"_comment": "gemini/subprocess.py — GeminiSubprocess supervisor. Spawns voice/sanad_voice.py as a child, tails stdout for Gemini-specific log markers, pushes camera frames + motion state to the child over its stdin, exposes transcript + state to the dashboard.",
"log_tail_size": 2000,
"transcript_tail_size": 30,
"log_name": "gemini_subprocess",
"stop_timeout_sec": 3.0,
"terminate_timeout_sec": 2.0,
"frame_forward_interval_sec": 0.5,
"noisy_prefixes": [
"ALSA lib ",
"Expression 'alsa_",
"Cannot connect to server socket",
"jack server is not running"
],
"noisy_fragments": [
"Unknown PCM",
"Evaluate error",
"snd_pcm_open_noupdate",
"PaAlsaStream",
"snd_config_evaluate",
"snd_func_refer"
]
}
}

92
config/local_config.json Normal file
View File

@ -0,0 +1,92 @@
{
"_description": "Tunables for local/* — fully on-device voice pipeline (Silero VAD → Whisper → Qwen via llama.cpp → CosyVoice2). Loaded via core.config_loader.load('local').",
"subprocess": {
"_comment": "local/subprocess.py — LocalSubprocess supervisor. Mirrors gemini/subprocess.py. IMPORTANT: python_bin points at the `local` conda env (Python 3.8 + Jetson CUDA torch) so CosyVoice+Whisper run with GPU, while the dashboard/Gemini stack stays in gemini_sdk (Python 3.10).",
"python_bin": "/home/unitree/miniconda3/envs/local/bin/python",
"log_tail_size": 2000,
"transcript_tail_size": 30,
"log_name": "local_subprocess",
"stop_timeout_sec": 5.0,
"terminate_timeout_sec": 3.0,
"noisy_prefixes": [
"ALSA lib ",
"Expression 'alsa_",
"Cannot connect to server socket",
"jack server is not running"
],
"noisy_fragments": [
"Unknown PCM",
"Evaluate error",
"snd_pcm_open_noupdate",
"PaAlsaStream"
]
},
"vad": {
"_comment": "Silero VAD — CPU. Emits speech_start / speech_end events.",
"sample_rate": 16000,
"frame_ms": 32,
"threshold": 0.55,
"min_silence_ms": 400,
"min_speech_ms": 250,
"pad_start_ms": 200,
"pad_end_ms": 200,
"device": "cpu"
},
"stt": {
"_comment": "faster-whisper Large V3 Turbo, INT8 on GPU.",
"model_name": "large-v3-turbo",
"model_subdir": "faster-whisper-large-v3-turbo",
"device": "cuda",
"compute_type": "int8_float16",
"beam_size": 1,
"language": null,
"vad_filter": false,
"no_speech_threshold": 0.6,
"min_utterance_chars": 2,
"temperature": 0.0
},
"llm": {
"_comment": "Qwen 2.5 Instruct via Ollama (default) OR self-managed llama.cpp. Set backend to pick.",
"backend": "ollama",
"_ollama_comment": "Ollama daemon — assumes `ollama serve` is running; `ollama pull qwen2.5:1.5b` to fetch.",
"ollama_host": "127.0.0.1",
"ollama_port": 11434,
"ollama_model": "qwen2.5:1.5b",
"ollama_keep_alive": "5m",
"_llamacpp_comment": "Self-managed llama-server subprocess. Only used when backend='llama_cpp'.",
"model_subdir": "qwen2.5-1.5b-instruct-q4_k_m.gguf",
"server_binary": "llama-server",
"host": "127.0.0.1",
"port": 8080,
"n_gpu_layers": 99,
"ctx_size": 2048,
"threads": 4,
"startup_timeout_sec": 30,
"_shared_comment": "Generation params — both backends.",
"request_timeout_sec": 30,
"max_tokens": 200,
"temperature": 0.7,
"top_p": 0.9,
"stop": ["<|im_end|>", "\n\n\n"],
"chunk_delimiters": ".,?!؟،",
"chunk_min_chars": 8
},
"tts": {
"_comment": "CosyVoice2 0.5B streaming — GPU. Uses a 3s reference WAV for voice cloning.",
"model_subdir": "CosyVoice2-0.5B",
"reference_wav_subdir": "khaleeji_reference_3s.wav",
"reference_prompt": "",
"stream_chunk_sec": 0.25,
"sample_rate": 16000,
"queue_max": 3,
"device": "cuda"
}
}

27
config/mask_config.json Normal file
View File

@ -0,0 +1,27 @@
{
"_comment": "Shining LED face mask (BLE). Driven by the FaceController subsystem (face/mask_face.py) which imports the standalone Mask project. Needs an env with bleak + Pillow (g1_env). Free the mask from the phone app before connecting.",
"mask_dir": "",
"_mask_dir": "Path to the Mask project (flat shiningmask lib). Empty -> auto: <Project>/Mask. Env override: SANAD_MASK_DIR.",
"name_prefix": "MASK",
"_name_prefix": "BLE scan prefix; the mask advertises e.g. 'MASK-02A711'. Env: SANAD_MASK_NAME_PREFIX.",
"address": "",
"_address": "Specific BLE MAC to connect to. Empty -> scan by name_prefix. Env: SANAD_MASK_ADDRESS.",
"adapter": "",
"_adapter": "BlueZ adapter (e.g. 'hci0'). Empty -> default. Env: SANAD_MASK_ADAPTER.",
"brightness": 95,
"_brightness": "0-128. Keep <=100 to avoid LED flicker (battery-limited).",
"fps": 8.0,
"_fps": "FaceAnimator (fallback driver) frame rate (PLAY commands/sec).",
"lifelike": true,
"_lifelike": "Use the LifelikeFace driver (face/face_motion.py): eye saccades, varied blinks, listening/thinking/speaking states, reactions, smooth lip-sync. false -> basic FaceAnimator.",
"autostart": true,
"_autostart": "Auto-connect + Start face on boot (best-effort, background — never blocks startup). After the one-time frame upload, later boots just connect + animate. false -> connect/start manually from the dashboard.",
"connect_timeout": 15.0,
"connect_attempts": 5,
"eye_color": [0, 230, 255],
"_eye_color": "Face eye/iris RGB (baked into the uploaded frames). Default cyan. Set via the dashboard 'Apply colors' (persisted here).",
"mouth_color": [255, 50, 50],
"_mouth_color": "Face mouth RGB. Default red.",
"sclera_color": [255, 255, 255],
"_sclera_color": "White-of-the-eye RGB. Default white."
}

View File

@ -5,6 +5,7 @@
"_comment": "motion/arm_controller.py — enable_arm_sdk_index + replay_hz come from core.g1_hardware",
"ramp_in_steps": 60,
"ramp_out_steps": 180,
"settle_hold_sec": 0.5,
"watchdog_timeout_sec": 0.25,
"watchdog_disable_after_sec": 1.0,
"arm_indices_start": 15,
@ -12,6 +13,20 @@
"jsonl_id_start": 100
},
"loco_controller": {
"_comment": "G1_Controller/loco_controller.py — manual locomotion. NIC is shared from the arm's DDS init (config core.dds / SANAD_DDS_INTERFACE), not set here.",
"cap_walk": 0.6,
"cap_run": 1.2,
"lin_step": 0.05,
"ang_step": 0.2,
"watchdog_timeout_sec": 0.5,
"arm_block_window_sec": 1.5,
"step_duration_sec": 0.6,
"step_speed_frac": 0.5,
"loco_timeout_sec": 10.0,
"msc_timeout_sec": 5.0
},
"macro_player": {
"_comment": "motion/macro_player.py — JSONL playback",
"ramp_in_steps": 60,

View File

@ -50,44 +50,18 @@
"dir_relative": "data/recordings"
},
"system_prompt": {
"_comment": "Persona filename lives in core.script_files.persona; default text in core.gemini_defaults.default_system_prompt. This section is now metadata-only."
},
"typed_replay": {
"_comment": "voice/typed_replay.py — max_text_len comes from dashboard.api_input",
"monitor_chunk_size": 512,
"monitor_tail_sec": 0.2
},
"live_gemini_subprocess": {
"_comment": "voice/live_gemini_subprocess.py — LiveGeminiSubprocess",
"log_tail_size": 2000,
"transcript_tail_size": 30,
"log_name": "live_gemini_subprocess",
"stop_timeout_sec": 3.0,
"terminate_timeout_sec": 2.0,
"noisy_prefixes": [
"ALSA lib ",
"Expression 'alsa_",
"Cannot connect to server socket",
"jack server is not running"
],
"noisy_fragments": [
"Unknown PCM",
"Evaluate error",
"snd_pcm_open_noupdate",
"PaAlsaStream",
"snd_config_evaluate",
"snd_func_refer"
]
},
"live_voice_loop": {
"_comment": "voice/live_voice_loop.py — arm phrase dispatcher. arm_txt filename comes from core.script_files.arm_phrases",
"trigger_log_size": 100,
"poll_interval_sec": 0.1,
"deferred_default": false
"deferred_default": false,
"trigger_enabled_default": false
},
"local_tts": {
@ -97,27 +71,5 @@
"xvector_filename": "arabic_xvector_embedding.pt",
"sample_rate": 16000,
"channels": 1
},
"gemini_client": {
"_comment": "voice/gemini_client.py — default_system_prompt comes from core.gemini_defaults",
"recv_timeout_sec": 30,
"reconnect_max_attempts": 3,
"reconnect_initial_delay_sec": 1.0,
"reconnect_max_delay_sec": 10.0
},
"asr_buffer": {
"_comment": "text_utils.maybe_trigger_arm state machine defaults",
"window_sec": 2.0,
"short_token_bonus_sec": 1.0,
"join_no_space_maxlen": 2,
"max_chars": 120,
"stream_max_chars": 80,
"trigger_dedup_window_sec": 2.0,
"pending_arm_ttl_sec": 6.0,
"pending_arm_fallback_sec": 0.65,
"dup_call_window_sec": 0.25,
"dup_asr_repeat_window_sec": 0.9
}
}

View File

@ -46,7 +46,7 @@ class Brain:
self._lock = asyncio.Lock()
# Sub-modules are injected after construction so imports stay lazy.
self._voice = None # voice.gemini_client.GeminiVoiceClient
self._voice = None # gemini.client.GeminiVoiceClient
self._audio_mgr = None # voice.audio_manager.AudioManager
self._arm = None # motion.arm_controller.ArmController
self._macro_rec = None # motion.macro_recorder.MacroRecorder

71
core/persona.py Normal file
View File

@ -0,0 +1,71 @@
"""Active-persona selection — which script file Gemini loads as its system
prompt.
The operator can keep several persona variants in scripts/ (e.g.
``sanad_script.txt``, ``sanad_script_v1.txt``, ``sanad_script_v2.txt``) and pick
which one is live. The selection is a single basename stored in
``data/active_persona.txt``; the DEFAULT (and reset target) is always the
configured persona (``sanad_script.txt``). The Gemini child resolves this at
session start, so a new selection takes effect on the next voice (re)connect.
A missing/blank/stale pointer transparently falls back to the default, so this
can never break the voice worst case it loads ``sanad_script.txt``.
"""
from __future__ import annotations
from pathlib import Path
from Project.Sanad.config import DATA_DIR, SCRIPTS_DIR
ACTIVE_PERSONA_FILE = DATA_DIR / "active_persona.txt"
def default_persona_name() -> str:
"""The configured default persona filename (core.script_files.persona)."""
try:
from Project.Sanad.core.config_loader import section as _section
name = (_section("core", "script_files") or {}).get("persona")
return (name or "sanad_script.txt").strip() or "sanad_script.txt"
except Exception:
return "sanad_script.txt"
def active_persona_name() -> str:
"""Selected persona basename — the chosen variant if set AND still exists,
otherwise the default. Never raises."""
default = default_persona_name()
try:
sel = ACTIVE_PERSONA_FILE.read_text(encoding="utf-8").strip()
except Exception:
sel = ""
if sel:
cand = SCRIPTS_DIR / Path(sel).name # basename only — no traversal
if cand.is_file():
return cand.name
return default
def active_persona_path() -> Path:
"""Full path to the persona script Gemini should load right now."""
return SCRIPTS_DIR / active_persona_name()
def set_active_persona(name: str | None) -> str:
"""Persist the selected persona basename. Passing None/"" or the default
name clears the pointer (revert to default). Returns the effective active
name. Raises FileNotFoundError if a non-default name doesn't exist."""
nm = (Path(str(name)).name if name else "").strip()
default = default_persona_name()
if not nm or nm == default:
try:
ACTIVE_PERSONA_FILE.unlink()
except FileNotFoundError:
pass
except Exception:
pass
return default
if not (SCRIPTS_DIR / nm).is_file():
raise FileNotFoundError(nm)
DATA_DIR.mkdir(parents=True, exist_ok=True)
ACTIVE_PERSONA_FILE.write_text(nm, encoding="utf-8")
return nm

View File

@ -50,9 +50,16 @@ _REST_ROUTES: list[tuple[str, str, str]] = [
("live_voice", "/api/live-voice", "live-voice"),
("live_subprocess", "/api/live-subprocess", "live-subprocess"),
("typed_replay", "/api/typed-replay", "typed-replay"),
("recognition", "/api/recognition", "recognition"),
("zones", "/api/zones", "zones"),
("temp_monitor", "/api/temp", "temperature"),
("controller", "/api/controller", "controller"),
("mask", "/api/mask", "mask"),
("mask_social", "/api/mask", "mask-social"),
("navigation", "/api/nav", "navigation"),
]
_WS_ROUTES: list[str] = ["log_stream"]
_WS_ROUTES: list[str] = ["log_stream", "motor_temps", "terminal"]
_loaded_routes: list[str] = []
_failed_routes: dict[str, str] = {}
@ -109,7 +116,13 @@ async def root():
if index.exists():
from fastapi.responses import HTMLResponse
try:
return HTMLResponse(index.read_text(encoding="utf-8"))
# no-store so the browser always re-fetches the dashboard HTML/JS
# after a deploy — otherwise stale cached JS keeps calling old
# endpoints (e.g. /nav/* instead of /api/nav/*) and 404s.
return HTMLResponse(
index.read_text(encoding="utf-8"),
headers={"Cache-Control": "no-store, must-revalidate"},
)
except OSError as exc:
return {"error": f"Could not read index.html: {exc}"}
return {

View File

@ -0,0 +1,66 @@
"""In-process arbitration between Nav2 (web_nav3) and the manual LocoController.
Both stacks can drive the G1's legs via different command paths:
- Nav2 (web_nav3) publishes cmd_vel from a navigation goal/mission.
- LocoController issues LocoClient.Move()/step() from the Controller tab and
Gemini movement dispatch.
The documented hazard is "two stacks must never both drive the legs at once".
This module is a tiny thread-safe gate that lets ONE commander own the legs at a
time. controller.py sets loco_active for arm/move/step and refuses when nav is
active; navigation.py sets nav_active for goto/missions/run and refuses when loco
is active. The E-STOP / cancel paths clear the relevant flag.
Pure in-process state (no DDS, no HTTP) both routers share this single module
instance, so the flags are coherent across the dashboard process.
"""
from __future__ import annotations
import threading
_lock = threading.Lock()
_loco_active = False
_nav_active = False
def loco_active() -> bool:
with _lock:
return _loco_active
def nav_active() -> bool:
with _lock:
return _nav_active
def acquire_loco() -> bool:
"""Claim the legs for manual loco. Returns False if Nav2 holds them."""
global _loco_active
with _lock:
if _nav_active:
return False
_loco_active = True
return True
def release_loco() -> None:
global _loco_active
with _lock:
_loco_active = False
def acquire_nav() -> bool:
"""Claim the legs for Nav2. Returns False if manual loco holds them."""
global _nav_active
with _lock:
if _loco_active:
return False
_nav_active = True
return True
def release_nav() -> None:
global _nav_active
with _lock:
_nav_active = False

View File

@ -160,7 +160,13 @@ async def audio_status():
"g1_speaker_muted": g1_muted,
"g1_current_volume": _g1_current_volume,
"g1_user_volume": _g1_user_volume,
"g1_available": _g1_audio_client is not None or (_g1_init_error == ""),
# Only report available once an AudioClient has actually been
# built — reporting True before any init attempt made the UI
# advertise G1 speaker controls that then 503 on first use.
# `g1_init_error` surfaces *why* it's unavailable (or "" if
# init was never attempted yet).
"g1_available": _g1_audio_client is not None,
"g1_init_error": _g1_init_error,
"sink": sink,
"source": source,
"current": cur,
@ -312,7 +318,9 @@ async def get_g1_volume():
"""
def _do():
return {
"available": _g1_audio_client is not None or (_g1_init_error == ""),
# True only after an AudioClient was actually constructed —
# `init_error` (below) explains an unavailable/never-tried state.
"available": _g1_audio_client is not None,
"current_volume": _g1_current_volume,
"user_volume": _g1_user_volume,
"muted": _g1_current_volume == 0,
@ -343,35 +351,48 @@ async def set_g1_volume(payload: G1VolumePayload):
if not 0 <= level <= 100:
raise HTTPException(400, "level must be 0..100")
# 1) G1 chest speaker (DDS) — best-effort so it works even when an
# external sink (JBL) is the active output.
code = None
client = _get_g1_audio_client()
if client is None:
raise HTTPException(
503,
f"G1 AudioClient unavailable: {_g1_init_error or 'unknown'}",
)
try:
with _g1_audio_lock:
code = client.SetVolume(level)
_g1_current_volume = level
if level > 0:
# Only update the "preferred unmuted" level when the
# user is setting a non-zero volume. Setting 0 is a
# mute, which shouldn't overwrite their preference.
_g1_user_volume = level
except Exception as exc:
raise HTTPException(500, f"SetVolume failed: {exc}")
if client is not None:
try:
with _g1_audio_lock:
code = client.SetVolume(level)
_g1_current_volume = level
except Exception as exc:
log.warning("G1 SetVolume failed: %s", exc)
if level > 0:
_g1_user_volume = level
# 2) The ACTIVE profile's PulseAudio sink (JBL / Anker / …). Target the
# RESOLVED sink from the saved selection, NOT @DEFAULT_SINK@ — the PA
# default can be a different sink (e.g. the chest platform-sound) even
# when the JBL is the selected output, so @DEFAULT_SINK@ would move the
# wrong sink and the slider would appear to do nothing on the JBL.
pa_applied = False
try:
sink = (ad.load_state() or {}).get("sink") or "@DEFAULT_SINK@"
_pactl(["set-sink-volume", sink, "%d%%" % level])
if level > 0:
_pactl(["set-sink-mute", sink, "0"])
pa_applied = True
except Exception as exc:
log.warning("PA set-sink-volume failed: %s", exc)
if client is None and not pa_applied:
raise HTTPException(503, "No speaker available (G1 + PulseAudio both failed)")
# Persist the user's preferred level (not the current) so a
# subsequent mute-then-restart restores to the preferred level
_save_persisted_g1_volume(_g1_user_volume)
log.info("G1 volume → %d (user_pref=%d, rc=%s)",
level, _g1_user_volume, code)
log.info("volume → %d (g1_rc=%s, pa=%s, user_pref=%d)",
level, code, pa_applied, _g1_user_volume)
return {
"ok": True,
"current_volume": level,
"user_volume": _g1_user_volume,
"muted": level == 0,
"return_code": code,
"pa_applied": pa_applied,
"persisted": True,
}
return await asyncio.to_thread(_do)
@ -471,5 +492,474 @@ async def apply_audio():
audio_mgr.refresh_devices()
except Exception:
pass
# Hot-swap the live Gemini voice to the selected profile too, so picking
# a device (e.g. the JBL) moves BOTH record playback AND the live voice
# to it — without dropping the session. Best-effort; no-op if not running.
try:
from Project.Sanad.main import live_sub
pid = (ad.load_state() or {}).get("profile_id")
if (pid and live_sub is not None and hasattr(live_sub, "send_profile")
and hasattr(live_sub, "is_running") and live_sub.is_running()):
live_sub.send_profile(pid, reason="dashboard audio Apply")
except Exception:
pass
# Restore the user's SAVED volume to the selected sink (USB/BT speakers
# like the JBL otherwise come back at a low PulseAudio default). Use the
# saved level, NOT a forced 100%, so the slider/sink keep the user's
# choice across selects + restarts. Target the resolved sink.
try:
sink = (ad.load_state() or {}).get("sink") or "@DEFAULT_SINK@"
_pactl(["set-sink-volume", sink, "%d%%" % _g1_user_volume])
if _g1_user_volume > 0:
_pactl(["set-sink-mute", sink, "0"])
except Exception:
pass
return result
return await asyncio.to_thread(_do)
# ─────────────────────── Reset endpoints (Pulse + USB) ───────────────────────
#
# Two distinct recovery paths for the dashboard's audio panel:
#
# POST /api/audio/reset — SOFT: restart pulseaudio / pipewire-pulse.
# Fixes Pulse-side state (stuck profile, lost default sink, crashed
# module). Cannot recover a kernel-side missing USB capture descriptor
# — snd-usb-audio parses those at probe time and Pulse can't influence
# that. Use for "devices look weird" failures.
#
# POST /api/audio/usb-reset — HARD: unbind+rebind snd-usb-audio scoped
# to the Anker VID:PID. Forces snd-usb-audio to re-parse UAC1
# descriptors → input profile reappears even after the firmware/USB
# handshake dropped it. Use for "Anker mic missing from pactl" — the
# symptom soft-reset cannot fix.
#
# Both gate with module-level locks (no concurrent reset), refuse while Live
# Gemini is running or a record is mid-playback, and return structured
# before/after diagnostics so the dashboard can show meaningful toasts.
_RESET_LOCK = threading.Lock()
_USB_RESET_LOCK = threading.Lock()
# Anker PowerConf A3321 — used both for VID:PID matching in sysfs and for
# logging. Change here if you add support for a different USB conference
# device (Hollyland etc).
_USB_RESET_TARGETS = (
{"vid": "291a", "pid": "3301", "label": "Anker PowerConf"},
)
def _refuse_if_busy() -> None:
"""Raise HTTPException(409) if Live Gemini is active or a record is playing.
Used by both reset endpoints a userspace audio restart mid-stream
leaves the active session in a broken state (PortAudio handle pointing
at a dead Pulse, in-flight write() raises, etc.). Cheaper to refuse
than to recover.
"""
try:
from Project.Sanad.main import live_sub
except Exception:
live_sub = None
if live_sub is not None:
try:
st = live_sub.status() or {}
except Exception:
st = {}
state = (st.get("state") or "").lower()
if st.get("running") or state not in ("", "stopped", "error"):
raise HTTPException(
409, f"Stop Live Gemini before resetting audio (state={state or '?'}).",
)
try:
from Project.Sanad.main import audio_mgr
except Exception:
audio_mgr = None
if audio_mgr is not None and hasattr(audio_mgr, "playback_status"):
try:
ps = audio_mgr.playback_status() or {}
if ps.get("playing"):
raise HTTPException(
409, "Stop the active playback before resetting audio.",
)
except HTTPException:
raise
except Exception:
pass
def _detect_pa_flavour() -> str:
"""Return 'pipewire' if pipewire-pulse is the active daemon, else 'pulse'."""
try:
r = subprocess.run(
["pgrep", "-x", "pipewire-pulse"],
check=False, capture_output=True, text=True, timeout=1.0,
)
if r.returncode == 0 and (r.stdout or "").strip():
return "pipewire"
except (FileNotFoundError, subprocess.SubprocessError):
pass
return "pulse"
def _kill_audio_daemon(flavour: str) -> dict:
"""Issue the restart command for the detected daemon. Non-zero exit is a
soft warning (some installs return 1 when there's no daemon to kill)."""
if flavour == "pipewire":
cmd = ["systemctl", "--user", "restart", "pipewire-pulse.service"]
else:
cmd = ["pulseaudio", "-k"]
try:
r = subprocess.run(cmd, check=False, capture_output=True,
text=True, timeout=5.0)
info = {"cmd": " ".join(cmd), "returncode": r.returncode,
"stderr": (r.stderr or "").strip()[:300]}
if r.returncode != 0:
log.warning("audio reset: %s exited %d (%s)",
cmd[0], r.returncode, info["stderr"])
return info
except FileNotFoundError as exc:
return {"cmd": " ".join(cmd), "returncode": -1,
"stderr": f"binary missing: {exc}"}
except subprocess.TimeoutExpired:
return {"cmd": " ".join(cmd), "returncode": -1,
"stderr": "timeout (>5s)"}
def _wait_for_pactl(deadline_s: float = 5.0, interval_s: float = 0.2) -> bool:
"""Poll `pactl info` until it returns 0 or the deadline expires."""
import time as _time
end = _time.monotonic() + deadline_s
while _time.monotonic() < end:
if ad.pactl_available():
return True
_time.sleep(interval_s)
return False
@router.post("/reset")
async def reset_audio_subsystem():
"""SOFT reset — restart pulseaudio/pipewire-pulse and re-resolve devices.
Use when devices look stuck, pactl is unavailable, or the wrong sink
is being selected. **Does NOT recover a kernel-side missing USB capture
descriptor** for that symptom use /api/audio/usb-reset.
"""
if os.geteuid() == 0:
raise HTTPException(
403, "Refusing to reset audio as root — Sanad must run as the "
"unitree user so the per-user PulseAudio session is reachable.",
)
if not _RESET_LOCK.acquire(blocking=False):
raise HTTPException(429, "Reset already in progress.")
try:
_refuse_if_busy()
log.info(
"audio reset requested (uid=%s PULSE_RUNTIME_PATH=%s XDG_RUNTIME_DIR=%s)",
os.geteuid(),
os.environ.get("PULSE_RUNTIME_PATH") or "-",
os.environ.get("XDG_RUNTIME_DIR") or "-",
)
try:
from Project.Sanad.main import audio_mgr
except Exception:
audio_mgr = None
def _do() -> dict:
before = {"pactl_available": ad.pactl_available(),
"selection": ad.current_selection()}
# Quiesce AudioManager so the next play_wav rebinds cleanly.
pya_closed = False
if audio_mgr is not None:
play_lock = getattr(audio_mgr, "play_lock", None)
acquired = False
if play_lock is not None:
acquired = play_lock.acquire(timeout=2.0)
try:
try:
audio_mgr.close()
pya_closed = True
except Exception as exc:
log.warning("audio reset: audio_mgr.close failed: %s", exc)
finally:
if acquired and play_lock is not None:
play_lock.release()
flavour = _detect_pa_flavour()
kill_info = _kill_audio_daemon(flavour)
came_back = _wait_for_pactl(deadline_s=5.0)
if not came_back and flavour == "pulse":
# autospawn may be disabled — try an explicit start.
try:
subprocess.run(["pulseaudio", "--start"], check=False,
capture_output=True, text=True, timeout=3.0)
except (FileNotFoundError, subprocess.SubprocessError) as exc:
log.warning("audio reset: pulseaudio --start failed: %s", exc)
came_back = _wait_for_pactl(deadline_s=2.0)
if not came_back:
raise HTTPException(500, {
"error": "audio daemon did not return within ~7s",
"flavour": flavour, "kill": kill_info,
})
apply_result: dict = {}
try:
apply_result = ad.apply_current_selection() or {}
except Exception as exc:
log.warning("audio reset: apply_current_selection failed: %s", exc)
apply_result = {"error": str(exc)}
if audio_mgr is not None:
try:
import pyaudio
audio_mgr.pya = pyaudio.PyAudio()
audio_mgr.refresh_devices()
except Exception as exc:
log.error("audio reset: PyAudio re-init failed: %s", exc)
raise HTTPException(
500, f"PortAudio re-init failed after daemon restart: {exc}")
after_sel = ad.current_selection() or {}
detected = ad.detect_plugged_profiles() or []
after = {
"pactl_available": ad.pactl_available(),
"selection": after_sel,
"detected_profiles": [p.get("profile", {}).get("id") for p in detected],
}
return {
"ok": True, "best_effort": True, "flavour": flavour,
"kill": kill_info, "pya_reinitialized": pya_closed,
"apply_result": apply_result,
"input_recovered": bool(after_sel.get("source")),
"output_recovered": bool(after_sel.get("sink")),
"before": before, "after": after,
"hint": ("Soft reset only fixes Pulse-side state. If "
"input_recovered is False, try POST /api/audio/usb-reset "
"or physically replug the dongle."),
}
return await asyncio.to_thread(_do)
finally:
_RESET_LOCK.release()
def _find_usb_devices_by_vid_pid(vid: str, pid: str) -> list[str]:
"""Return sysfs bus-id strings (e.g. '1-3') for every USB device whose
idVendor/idProduct match. Empty list when nothing matches.
We read /sys/bus/usb/devices/* every USB *device* (not interface) has
idVendor/idProduct files. Interfaces (paths with a colon, e.g. '1-3:1.1')
do not, so they're naturally skipped.
"""
import glob
hits: list[str] = []
for path in glob.glob("/sys/bus/usb/devices/*"):
name = os.path.basename(path)
if ":" in name:
continue
try:
with open(os.path.join(path, "idVendor")) as f:
v = f.read().strip().lower()
with open(os.path.join(path, "idProduct")) as f:
p = f.read().strip().lower()
except OSError:
continue
if v == vid.lower() and p == pid.lower():
hits.append(name)
return hits
def _snd_usb_interfaces_for_device(bus_id: str) -> list[str]:
"""For USB device `bus_id` (e.g. '1-3'), return all interface names that
are currently bound to the snd-usb-audio driver (e.g. ['1-3:1.0']).
Used so we unbind ONLY the audio interfaces and don't touch HID / HUB
interfaces on the same composite device.
"""
import glob
bound: list[str] = []
base = f"/sys/bus/usb/devices/{bus_id}"
for iface in glob.glob(f"{base}/{bus_id}:*"):
driver_link = os.path.join(iface, "driver")
if not os.path.islink(driver_link):
continue
try:
driver = os.path.basename(os.readlink(driver_link))
except OSError:
continue
if driver == "snd-usb-audio":
bound.append(os.path.basename(iface))
return bound
def _write_sysfs(path: str, value: str) -> tuple[bool, str]:
"""Write `value` to a sysfs file. Returns (success, error_message).
Writes to /sys/bus/usb/drivers/snd-usb-audio/{bind,unbind} usually
require root. If permission denied, the caller should fall back to
invoking shell_scripts/reset_anker_usb.sh via sudo (one-time sudoers
setup documented in that script's header).
"""
try:
with open(path, "w") as f:
f.write(value)
return True, ""
except PermissionError as exc:
return False, f"permission denied: {path} ({exc})"
except OSError as exc:
return False, f"write failed: {path} ({exc})"
@router.post("/usb-reset")
async def usb_reset_anker():
"""HARD reset — unbind+rebind snd-usb-audio for the Anker (VID:PID
291a:3301). Forces the kernel to re-parse the USB Audio Class
descriptors, which is the only way to recover a missing capture profile
on this Jetson without a physical replug.
Tries two paths:
1. Direct sysfs write (no sudo) works if a udev rule has set
`audio` group ownership / world-write on the snd-usb-audio bind
files, or if Sanad runs as root (it shouldn't).
2. Fallback to `sudo shell_scripts/reset_anker_usb.sh` works after
a one-time sudoers entry; see that script's header for setup.
Refuses while Live Gemini or a record playback is in flight (same
guard as the soft reset).
"""
if not _USB_RESET_LOCK.acquire(blocking=False):
raise HTTPException(429, "USB reset already in progress.")
try:
_refuse_if_busy()
# Find candidate Anker USB devices currently enumerated.
candidates: list[dict] = []
for tgt in _USB_RESET_TARGETS:
for bus_id in _find_usb_devices_by_vid_pid(tgt["vid"], tgt["pid"]):
candidates.append({"bus_id": bus_id, **tgt})
if not candidates:
wanted = ", ".join(
"{}:{}".format(t["vid"], t["pid"]) for t in _USB_RESET_TARGETS
)
raise HTTPException(
404,
f"No matching USB device found (looked for {wanted}). "
"Plug the Anker dongle and try again.",
)
log.info("usb reset: candidates=%s", candidates)
def _do() -> dict:
before_detected = [
p.get("profile", {}).get("id")
for p in (ad.detect_plugged_profiles() or [])
]
results: list[dict] = []
for cand in candidates:
bus = cand["bus_id"]
ifaces = _snd_usb_interfaces_for_device(bus)
attempt = {"bus_id": bus, "label": cand["label"],
"snd_interfaces": ifaces, "method": None,
"ok": False, "error": ""}
if not ifaces:
attempt["error"] = ("no snd-usb-audio interfaces bound "
"to this device — already unbound or "
"kernel didn't claim it")
results.append(attempt)
continue
# ─── Path 1: direct sysfs write ───
unbind_path = "/sys/bus/usb/drivers/snd-usb-audio/unbind"
bind_path = "/sys/bus/usb/drivers/snd-usb-audio/bind"
direct_ok = True
direct_err = ""
for iface in ifaces:
ok, err = _write_sysfs(unbind_path, iface)
if not ok:
direct_ok = False
direct_err = err
break
if direct_ok:
import time as _time
_time.sleep(0.5)
for iface in ifaces:
ok, err = _write_sysfs(bind_path, iface)
if not ok:
direct_ok = False
direct_err = err
break
if direct_ok:
attempt.update({"method": "direct-sysfs", "ok": True})
results.append(attempt)
continue
# ─── Path 2: sudo helper script ───
from pathlib import Path as _Path
helper = (_Path(__file__).resolve().parent.parent.parent
/ "shell_scripts" / "reset_anker_usb.sh")
if not helper.exists():
attempt.update({"method": "direct-sysfs",
"error": f"{direct_err}; helper not present "
f"at {helper}"})
results.append(attempt)
continue
try:
r = subprocess.run(
["sudo", "-n", str(helper), bus],
check=False, capture_output=True, text=True, timeout=10.0,
)
attempt["method"] = "sudo-helper"
if r.returncode == 0:
attempt["ok"] = True
else:
attempt["error"] = (
f"sudo helper exited {r.returncode}: "
f"{(r.stderr or r.stdout or '').strip()[:300]}"
)
except subprocess.TimeoutExpired:
attempt["error"] = "sudo helper timed out (>10s)"
except FileNotFoundError as exc:
attempt["error"] = f"sudo not available: {exc}"
results.append(attempt)
# Settle, then re-detect
import time as _time
_time.sleep(1.0)
try:
ad.apply_current_selection()
except Exception:
pass
try:
from Project.Sanad.main import audio_mgr
if audio_mgr is not None and hasattr(audio_mgr, "refresh_devices"):
audio_mgr.refresh_devices()
except Exception:
pass
after_detected = [
p.get("profile", {}).get("id")
for p in (ad.detect_plugged_profiles() or [])
]
any_ok = any(r.get("ok") for r in results)
mic_now = any(
"anker" in (p.get("profile", {}).get("id") or "").lower()
for p in (ad.detect_plugged_profiles() or [])
)
return {
"ok": any_ok,
"candidates": results,
"before_detected_profiles": before_detected,
"after_detected_profiles": after_detected,
"input_recovered": mic_now,
"hint": (
"If ok is False, the unbind/rebind path needs sudo. "
"Run `bash shell_scripts/reset_anker_usb.sh --setup-sudoers` "
"once on the robot to install the sudoers entry, then retry."
) if not any_ok else None,
}
return await asyncio.to_thread(_do)
finally:
_USB_RESET_LOCK.release()

View File

@ -0,0 +1,338 @@
"""Controller tab — manual dashboard locomotion control (N2 Phase 1/2).
Routes live under /api/controller. All WRITE actions (move / step / postures /
modes / MotionSwitcher) require the in-memory "Enable movement" arm flag and
return 409 when disarmed. Reads (/status, /joints, /msc, /status/summary),
E-STOP and the arm toggle are ALWAYS available.
`/status/summary` is the aggregate the dashboard polls for the global subsystem
status strip (Camera / Face / Place / Movement). It is kept under /api/controller
(final path /api/controller/status/summary) so no second router is needed; note
/api/status (no /summary) is already used by the SPA, so the suffix matters.
"""
from __future__ import annotations
import asyncio
from fastapi import APIRouter, HTTPException, Query
from pydantic import BaseModel
from Project.Sanad.config import BASE_DIR
from Project.Sanad.core.logger import get_logger
from Project.Sanad.vision import recognition_state
from Project.Sanad.dashboard.routes import _arbiter
log = get_logger("controller_routes")
router = APIRouter()
STATE_PATH = BASE_DIR / "data" / ".recognition_state.json"
# ── lazy subsystem accessors ────────────────────────────────
def _get_loco():
try:
from Project.Sanad.main import loco_controller # type: ignore
return loco_controller
except Exception:
return None
def _get_camera():
try:
from Project.Sanad.main import camera # type: ignore
return camera
except Exception:
return None
def _get_live_sub():
try:
from Project.Sanad.main import live_sub # type: ignore
return live_sub
except Exception:
return None
def _get_dispatch():
try:
from Project.Sanad.main import movement_dispatch # type: ignore
return movement_dispatch
except Exception:
return None
def _require_loco():
lc = _get_loco()
if lc is None:
raise HTTPException(503, "Locomotion controller subsystem unavailable.")
return lc
def _require_armed(lc):
if not lc.is_armed():
raise HTTPException(409, "Movement is disarmed. Enable movement first.")
def _claim_loco():
"""Arbitration gate: refuse a leg command while a Nav2 goal owns the legs."""
if not _arbiter.acquire_loco():
raise HTTPException(
409, "Navigation (Nav2) is active. Cancel the nav goal before manual movement."
)
def _cancel_nav():
"""Cancel any in-flight Nav2 goal and clear the nav arbitration flag.
Used by E-STOP so the global stop halts the legs no matter which stack is
driving them. Calls the nav client in-process (no HTTP self-call); never
raises into the caller.
"""
try:
from Project.Sanad.dashboard.routes.navigation import _CLIENT as _nav_client
if _nav_client is not None:
_nav_client.cancel()
except Exception:
log.exception("estop nav cancel failed")
finally:
_arbiter.release_nav()
# ── reads ───────────────────────────────────────────────────
@router.get("/status")
async def get_status():
lc = _require_loco()
return await asyncio.to_thread(lc.status)
@router.get("/joints")
async def get_joints():
lc = _require_loco()
return await asyncio.to_thread(lc.joints)
@router.get("/msc")
async def get_msc():
lc = _require_loco()
return await asyncio.to_thread(lc.msc_check)
# ── arm flag / E-STOP (always available) ────────────────────
@router.post("/arm")
async def set_arm(on: bool = Query(...)):
lc = _require_loco()
if on:
# Refuse to arm manual loco while Nav2 owns the legs.
_claim_loco()
try:
res = await asyncio.to_thread(lc.arm_movement)
except Exception:
_arbiter.release_loco()
raise
return res
res = await asyncio.to_thread(lc.disarm_movement)
_arbiter.release_loco()
return res
@router.post("/gemini-movement")
async def set_gemini_movement(on: bool = Query(...)):
"""Enable / disable Gemini voice-driven locomotion (N2 Phase 3 gate).
Writes recognition_state.movement_enabled SEPARATE from the manual arm
flag. The Gemini child announces the toggle (spoken), and the parent
MovementDispatcher starts/stops acting on confirmation phrases. Default OFF.
"""
st = await asyncio.to_thread(recognition_state.mutate, STATE_PATH,
movement_enabled=bool(on))
# Enabling Gemini movement also clears any E-STOP latch on the dispatcher.
if on:
md = _get_dispatch()
if md is not None:
try:
md.clear_estop()
except Exception:
log.exception("clear_estop failed")
log.info("gemini-movement %s", "ON" if on else "OFF")
return {"ok": True, "movement_enabled": st.movement_enabled}
@router.post("/estop")
async def estop():
lc = _require_loco()
res = await asyncio.to_thread(lc.estop)
# Full stop: drop the manual arm flag AND latch the voice dispatcher off, so
# no source (teleop, step, or voice dispatch) can keep driving the robot. The
# dispatcher latch is used instead of flipping movement_enabled so the Gemini
# child does not deliver a spoken "movement disabled" line during an E-STOP.
try:
await asyncio.to_thread(lc.disarm_movement)
except Exception:
log.exception("estop disarm failed")
md = _get_dispatch()
if md is not None:
try:
md.emergency_stop()
except Exception:
log.exception("estop dispatcher latch failed")
# Cancel any in-flight Nav2 goal too: the legs have exactly one commander,
# and an E-STOP must halt the legs whether loco or Nav2 is driving them.
await asyncio.to_thread(_cancel_nav)
_arbiter.release_loco()
return {"ok": True, **res}
@router.post("/stop")
async def stop():
lc = _require_loco()
# Allowed even when disarmed — StopMove is always safe.
res = await asyncio.to_thread(lc.stop_move)
return res
# ── movement (armed) ────────────────────────────────────────
class MoveBody(BaseModel):
vx: float = 0.0
vy: float = 0.0
vyaw: float = 0.0
run: bool = False
@router.post("/move")
async def move(body: MoveBody):
lc = _require_loco()
_require_armed(lc)
_claim_loco()
return await asyncio.to_thread(lc.move, body.vx, body.vy, body.vyaw, body.run)
@router.post("/step")
async def step(dir: str = Query(...)):
lc = _require_loco()
_require_armed(lc)
_claim_loco()
res = await asyncio.to_thread(lc.step, dir)
if not res.get("ok"):
raise HTTPException(400, res.get("reason", "step failed"))
return res
# ── modes / postures (armed) ────────────────────────────────
@router.post("/mode/prep")
async def mode_prep():
lc = _require_loco()
_require_armed(lc)
return await asyncio.to_thread(lc.prep_mode)
@router.post("/mode/ready")
async def mode_ready():
lc = _require_loco()
_require_armed(lc)
return await asyncio.to_thread(lc.ready_start_mode)
@router.post("/posture/{name}")
async def posture(name: str):
lc = _require_loco()
_require_armed(lc)
res = await asyncio.to_thread(lc.posture, name)
if not res.get("ok") and res.get("reason"):
raise HTTPException(400, res["reason"])
return res
@router.post("/balance")
async def balance(mode: int = Query(...)):
lc = _require_loco()
_require_armed(lc)
return await asyncio.to_thread(lc.set_balance_mode, mode)
@router.post("/height")
async def height(h: float = Query(...)):
lc = _require_loco()
_require_armed(lc)
return await asyncio.to_thread(lc.set_stand_height, h)
# ── MotionSwitcher / reconnect (armed) ──────────────────────
@router.post("/msc/select-ai")
async def msc_select_ai():
lc = _require_loco()
_require_armed(lc)
return await asyncio.to_thread(lc.msc_select_ai)
@router.post("/msc/release")
async def msc_release():
lc = _require_loco()
_require_armed(lc)
return await asyncio.to_thread(lc.msc_release)
@router.post("/reconnect")
async def reconnect():
lc = _require_loco()
_require_armed(lc)
return await asyncio.to_thread(lc.reconnect)
# ── aggregate subsystem summary (always available) ──────────
@router.get("/status/summary")
async def status_summary():
"""Live on/off state for the header status strip. Never raises."""
try:
st = recognition_state.read(STATE_PATH)
except Exception:
st = recognition_state.RecognitionState()
cam = _get_camera()
camera_running = False
try:
camera_running = bool(cam is not None and cam.is_running())
except Exception:
camera_running = False
lc = _get_loco()
movement_armed = False
try:
movement_armed = bool(lc is not None and lc.is_armed())
except Exception:
movement_armed = False
sub = _get_live_sub()
gemini_running = False
try:
runner = getattr(sub, "is_running", None)
gemini_running = bool(callable(runner) and runner())
except Exception:
gemini_running = False
# Effective Gemini-movement = the file flag AND not latched off by an E-STOP.
md = _get_dispatch()
estopped = False
try:
estopped = bool(md is not None and md.is_estopped())
except Exception:
estopped = False
return {
"vision_enabled": st.vision_enabled,
"camera_running": camera_running,
"face_rec_enabled": st.face_rec_enabled,
"zone_rec_enabled": st.zone_rec_enabled,
"movement_armed": movement_armed,
"gemini_movement_enabled": st.movement_enabled and not estopped,
"gemini_running": gemini_running,
}

View File

@ -4,10 +4,15 @@ from __future__ import annotations
import asyncio
from fastapi import APIRouter, HTTPException
from fastapi import APIRouter, HTTPException, Query
from Project.Sanad.config import BASE_DIR
from Project.Sanad.vision import recognition_state
router = APIRouter()
_STATE_PATH = BASE_DIR / "data" / ".recognition_state.json"
def _sub_or_503():
from Project.Sanad.main import live_sub
@ -19,9 +24,21 @@ def _sub_or_503():
@router.get("/status")
async def subprocess_status():
from Project.Sanad.main import live_sub
# record_enabled is a live flag (recognition_state) the panel toggle drives;
# surface it so the UI shows the current state even before a session starts.
rec = bool(recognition_state.read(_STATE_PATH).record_enabled)
if live_sub is None:
return {"available": False, "state": "unavailable"}
return live_sub.status()
return {"available": False, "state": "unavailable", "record_enabled": rec}
return {**live_sub.status(), "record_enabled": rec}
@router.post("/record")
async def set_record(on: bool = Query(...)):
"""Toggle auto-recording of conversation turns to data/recordings/. Takes
effect live (the voice child syncs its recorder) no session restart."""
st = await asyncio.to_thread(
recognition_state.mutate, _STATE_PATH, record_enabled=bool(on))
return {"ok": True, "record_enabled": st.record_enabled}
@router.post("/start")

View File

@ -1,14 +1,15 @@
"""Live Voice Commands — voice-to-arm phrase trigger dispatcher.
Listens to LiveGeminiSubprocess user transcripts, matches against
Listens to GeminiSubprocess user transcripts, matches against
sanad_arm.txt phrases, and fires ARM.trigger_action_by_id.
Endpoints:
POST /start begin polling transcripts
POST /stop stop polling
POST /deferred-mode?enabled toggle instant vs deferred trigger
GET /status running, last heard, last action, etc.
GET /triggers arm trigger history (log)
POST /start begin polling transcripts
POST /stop stop polling
POST /deferred-mode?enabled toggle instant vs deferred trigger
POST /trigger-enabled?enabled master gate allow arm actions or not
GET /status running, last heard, last action, etc.
GET /triggers arm trigger history (log)
"""
from __future__ import annotations
@ -54,6 +55,14 @@ async def set_deferred(enabled: bool):
return {"ok": True, "deferred_mode": loop.deferred_mode}
@router.post("/trigger-enabled")
async def set_trigger_enabled(enabled: bool):
"""Master gate for voice → arm triggering. Default OFF."""
loop = _loop()
loop.set_trigger_enabled(enabled)
return {"ok": True, "trigger_enabled": loop.trigger_enabled}
@router.get("/triggers")
async def triggers():
loop = _loop()

View File

@ -2,16 +2,46 @@
from __future__ import annotations
import asyncio
from pathlib import Path
from fastapi import APIRouter, HTTPException
from pydantic import BaseModel
from Project.Sanad.config import AUDIO_RECORDINGS_DIR, MOTIONS_DIR
from Project.Sanad.core.logger import get_logger
log = get_logger("macros_route")
router = APIRouter()
def _block_if_movement_armed():
"""409 when locomotion movement is armed — arm motion is mutually exclusive
with walking. The arm controller's motion-block is the safety net."""
try:
from Project.Sanad.main import loco_controller # type: ignore
armed = loco_controller is not None and loco_controller.is_armed()
except HTTPException:
raise
except Exception:
return
if armed:
raise HTTPException(
409, "Arm actions are disabled while movement is enabled. "
"Disable movement in the Controller tab first.")
class MacroName(BaseModel):
name: str
class ComboPlayPayload(BaseModel):
audio_file: str = "" # filename under data/audio/ (or empty for none)
motion_file: str = "" # DEPRECATED — use action_id. Still accepted for bare JSONL by filename.
action_id: int | None = None # arm_controller action id (SDK built-in OR JSONL) — preferred
speed: float = 1.0
@router.get("/")
async def list_macros():
from Project.Sanad.main import macro_play
@ -49,6 +79,7 @@ async def stop_recording():
@router.post("/play")
async def play_macro(payload: MacroName):
from Project.Sanad.main import brain
_block_if_movement_armed()
return await brain.play_macro(payload.name)
@ -58,3 +89,150 @@ async def stop_macro():
if macro_play:
macro_play.stop()
return {"ok": True}
# ─── Ad-hoc audio + motion combined playback ─────────────────────────
# List the two catalogues so the dashboard can populate dropdowns, then
# play the chosen pair in parallel (asyncio.gather) — same scheme the
# Brain uses for `parallel`-mode skills, but ad-hoc instead of predefined.
@router.get("/audio-files")
async def list_audio_files():
"""Enumerate playable audio files under data/audio/."""
AUDIO_RECORDINGS_DIR.mkdir(parents=True, exist_ok=True)
files = []
for p in sorted(AUDIO_RECORDINGS_DIR.glob("*.wav")):
try:
files.append({
"name": p.name,
"size_kb": round(p.stat().st_size / 1024, 1),
})
except OSError:
continue
return {"files": files, "dir": str(AUDIO_RECORDINGS_DIR)}
@router.get("/motion-files")
async def list_motion_files():
"""Enumerate playable .jsonl motions under data/motions/ (thin wrapper
so the Macro Recorder dropdown doesn't have to call the replay route)."""
MOTIONS_DIR.mkdir(parents=True, exist_ok=True)
files = []
for p in sorted(MOTIONS_DIR.glob("*.jsonl")):
try:
files.append({
"name": p.name,
"size_kb": round(p.stat().st_size / 1024, 1),
})
except OSError:
continue
return {"files": files, "dir": str(MOTIONS_DIR)}
@router.post("/stop-combined")
async def stop_combined():
"""Immediately stop any in-flight combined playback.
- `arm.cancel()` breaks the replay loop and triggers the smooth
return-to-home ramp (see `_return_home` in arm_controller.py).
- `audio_mgr.stop_playback()` sends AUDIO_STOP_PLAY to the G1
chest speaker via DDS.
Both run unconditionally so Stop works even if only one side was
actually playing.
"""
from Project.Sanad.main import audio_mgr, arm
result = {"motion_stopped": False, "audio_stopped": False}
if arm is not None:
try:
arm.cancel()
result["motion_stopped"] = True
except Exception as exc:
log.warning("stop-combined: arm.cancel failed: %s", exc)
result["motion_error"] = str(exc)
if audio_mgr is not None:
try:
audio_mgr.stop_playback()
result["audio_stopped"] = True
except Exception as exc:
log.warning("stop-combined: audio stop failed: %s", exc)
result["audio_error"] = str(exc)
return {"ok": True, **result}
@router.post("/play-combined")
async def play_combined(payload: ComboPlayPayload):
"""Fire a user-picked audio clip and arm action in parallel.
Motion dispatch is via `arm.trigger_by_id(action_id)` which handles
BOTH SDK built-in actions (shake_hand, wave, ) and recorded JSONL
replays. Audio goes through `audio_mgr.play_wav` (routed to the G1
chest speaker via DDS). Either side may be omitted.
"""
from Project.Sanad.main import audio_mgr, arm
has_audio = bool(payload.audio_file)
has_motion = payload.action_id is not None or bool(payload.motion_file)
if not has_audio and not has_motion:
raise HTTPException(400, "pick at least one of audio_file / action_id / motion_file")
if has_motion:
_block_if_movement_armed() # audio-only combos still allowed while armed
tasks = []
result: dict = {
"audio_file": payload.audio_file,
"action_id": payload.action_id,
"motion_file": payload.motion_file,
}
if has_audio:
if audio_mgr is None:
raise HTTPException(503, "AudioManager not available")
audio_path = (AUDIO_RECORDINGS_DIR / payload.audio_file).resolve()
try:
audio_path.relative_to(AUDIO_RECORDINGS_DIR.resolve())
except ValueError:
raise HTTPException(400, "audio_file path traversal denied")
if not audio_path.exists():
raise HTTPException(404, f"audio not found: {payload.audio_file}")
async def _play_audio():
try:
await asyncio.to_thread(audio_mgr.play_wav, audio_path)
result["audio_played"] = audio_path.name
except Exception as exc:
log.exception("combined play: audio failed")
result["audio_error"] = str(exc)
tasks.append(_play_audio())
if has_motion:
if arm is None:
raise HTTPException(503, "ArmController not available")
async def _play_motion():
try:
if payload.action_id is not None:
# SDK built-in OR JSONL — arm.trigger_by_id handles both
await asyncio.to_thread(arm.trigger_by_id,
int(payload.action_id),
payload.speed)
result["motion_played"] = f"action_id={payload.action_id}"
else:
# Legacy path: bare JSONL filename
motion_path = (MOTIONS_DIR / payload.motion_file).resolve()
try:
motion_path.relative_to(MOTIONS_DIR.resolve())
except ValueError:
result["motion_error"] = "motion_file path traversal denied"
return
if not motion_path.exists():
result["motion_error"] = f"motion not found: {payload.motion_file}"
return
await asyncio.to_thread(arm.replay_file, str(motion_path), payload.speed)
result["motion_played"] = motion_path.name
except Exception as exc:
log.exception("combined play: motion failed")
result["motion_error"] = str(exc)
tasks.append(_play_motion())
await asyncio.gather(*tasks)
return {"ok": True, **result}

179
dashboard/routes/mask.py Normal file
View File

@ -0,0 +1,179 @@
"""Mask Face tab — Shining LED face mask control (BLE).
Routes live under /api/mask. Backed by the FaceController subsystem
(face/mask_face.py), which owns a dedicated asyncio loop + BLE connection to the
standalone Mask project's `shiningmask` library.
Every handler is failure-safe: if the subsystem or its library is unavailable it
returns 503 (GET /status returns a degraded body) rather than crash the
dashboard. FaceController raises RuntimeError for "not connected" / "face not
started"; those map to 409. Blocking BLE calls run in a thread pool so the event
loop stays responsive.
"""
from __future__ import annotations
import asyncio
from typing import List, Optional
from fastapi import APIRouter, HTTPException, Query
from pydantic import BaseModel
from Project.Sanad.core.logger import get_logger
log = get_logger("mask_routes")
router = APIRouter()
# ── lazy subsystem accessor ─────────────────────────────────
def _get_face():
try:
from Project.Sanad.main import mask_face # type: ignore
return mask_face
except Exception:
return None
def _require():
mf = _get_face()
if mf is None:
raise HTTPException(503, "Mask face subsystem unavailable.")
return mf
def _run(fn, *args, **kwargs):
"""Call a FaceController method, mapping its errors to HTTP status codes."""
try:
return fn(*args, **kwargs)
except HTTPException:
raise
except RuntimeError as exc:
raise HTTPException(409, str(exc))
except Exception as exc: # noqa: BLE001
log.exception("mask operation failed")
raise HTTPException(500, str(exc))
# ── status ──────────────────────────────────────────────────
@router.get("/status")
async def status():
"""Never raises — returns a degraded body if the subsystem is missing."""
mf = _get_face()
if mf is None:
return {"available": False, "connected": False, "lib_available": False,
"last_error": "mask face subsystem not constructed"}
s = await asyncio.to_thread(mf.status)
s["available"] = True
return s
# ── connection ──────────────────────────────────────────────
@router.post("/connect")
async def connect(timeout: Optional[float] = Query(None),
attempts: Optional[int] = Query(None)):
mf = _require()
return await asyncio.to_thread(_run, mf.connect, timeout, attempts)
@router.post("/disconnect")
async def disconnect():
mf = _require()
return await asyncio.to_thread(_run, mf.disconnect)
# ── simple commands ─────────────────────────────────────────
@router.post("/brightness")
async def brightness(level: int = Query(..., ge=0, le=255)):
mf = _require()
return await asyncio.to_thread(_run, mf.set_brightness, level)
class TextBody(BaseModel):
text: str = ""
color: List[int] = [255, 255, 255]
mode: Optional[int] = None
bg: Optional[List[int]] = None # background RGB (None -> black)
speed: Optional[int] = None # scroll speed 0-255 (None -> firmware default)
@router.post("/text")
async def text(body: TextBody):
mf = _require()
bg = tuple(body.bg) if body.bg else None
return await asyncio.to_thread(_run, mf.set_text, body.text, tuple(body.color),
body.mode, bg, body.speed)
@router.post("/image")
async def image(id: int = Query(...)):
mf = _require()
return await asyncio.to_thread(_run, mf.show_image, id)
@router.post("/animation")
async def animation(id: int = Query(...)):
mf = _require()
return await asyncio.to_thread(_run, mf.play_animation, id)
@router.post("/clear")
async def clear():
mf = _require()
return await asyncio.to_thread(_run, mf.clear_diy)
# ── animated face ───────────────────────────────────────────
@router.post("/face/start")
async def face_start(reload: bool = Query(False)):
mf = _require()
return await asyncio.to_thread(_run, mf.face_start, reload)
@router.post("/face/stop")
async def face_stop():
mf = _require()
return await asyncio.to_thread(_run, mf.face_stop)
@router.post("/face/return")
async def face_return():
"""Resume the live animated face after a text/image/animation override."""
mf = _require()
return await asyncio.to_thread(_run, mf.return_face)
class FaceColorBody(BaseModel):
eye: Optional[List[int]] = None # eye/iris RGB
mouth: Optional[List[int]] = None # mouth RGB
sclera: Optional[List[int]] = None # white-of-the-eye RGB
@router.post("/face/color")
async def face_color(body: FaceColorBody):
"""Recolor the animated face (re-uploads the frame set if the face is live)."""
mf = _require()
return await asyncio.to_thread(_run, mf.set_face_color, body.eye, body.mouth, body.sclera)
@router.post("/speaking")
async def speaking(on: bool = Query(...)):
mf = _require()
return await asyncio.to_thread(_run, mf.set_speaking, on)
@router.post("/mouth")
async def mouth(level: int = Query(..., ge=0, le=3)):
mf = _require()
return await asyncio.to_thread(_run, mf.set_mouth, level)
@router.post("/expression/{name}")
async def expression(name: str):
mf = _require()
return await asyncio.to_thread(_run, mf.show_expression, name)

View File

@ -0,0 +1,395 @@
"""Social-media / QR display on the LED mask.
Renders a QR code (for a preset Instagram account) or an uploaded image onto the
mask's 46x58 display and holds it via the FaceController's reserved scratch slot
until the animated face is resumed. The shared helper :func:`show_social_on_mask`
is also called from the Gemini ``[[SHOW:account]]`` relay wired in ``main.py``.
Routes (under /api/mask):
POST /social/{account} -> show a preset Instagram QR
POST /qr -> upload an image (QR or any picture) + show it
POST /face/resume -> stop showing the scratch image, return to the face
GET /social -> list the preset accounts
"""
from __future__ import annotations
import asyncio
import io
import logging
import os
import sys
from pathlib import Path
import re
from fastapi import APIRouter, File, HTTPException, Query, UploadFile
from fastapi.responses import FileResponse
log = logging.getLogger("sanad.mask_social")
router = APIRouter() # prefix "/api/mask" supplied by dashboard/app.py _REST_ROUTES
# Preset Instagram accounts the mask can show as a QR. The mask is a low-res
# 46x58 panel, so a full-URL QR is dense; the black margin acts as the quiet
# zone and we scale modules crisply (NEAREST) to give it the best chance.
SOCIAL = {
"bu_sunaidah": {"handle": "@bu.sunaidah",
"url": "https://instagram.com/bu.sunaidah",
"short": "da.gd/VMkH8J"}, # -> instagram.com/bu.sunaidah (v1 QR)
"yslootahtech": {"handle": "@yslootahtech",
"url": "https://instagram.com/yslootahtech",
"short": "da.gd/Qr8RO"}, # -> instagram.com/yslootahtech (v1 QR)
}
def _ensure_mask_path() -> None:
"""Make the flat Mask lib (colorface) importable from this route — using the
SAME dir the FaceController resolved (the Mask lib lives outside the repo)."""
d = os.environ.get("SANAD_MASK_DIR")
if not d:
try:
from Project.Sanad.main import mask_face as _mf # type: ignore
d = getattr(_mf, "mask_dir", None)
except Exception:
d = None
if not d:
d = str(Path(__file__).resolve().parents[2] / "Mask")
if d and d not in sys.path:
sys.path.insert(0, d)
def _get_face():
from Project.Sanad.main import mask_face # type: ignore
if mask_face is None:
raise HTTPException(status_code=503, detail="mask face unavailable")
return mask_face
_EYE_BAND = 16 # top rows reserved for the cyan eyes; the code sits below them
def _compose_under_eyes(inner) -> bytes:
"""Draw two cyan eyes across the top and place ``inner`` (a QR / image) in the
area BELOW them, then encode for the mask. Keeps the panel looking like a face
with a code under the eyes instead of a full-screen QR."""
_ensure_mask_path()
import colorface as cf
from PIL import Image, ImageDraw
W, H = cf.DISPLAY_W, cf.DISPLAY_H
inner = inner.convert("RGB")
iw, ih = inner.size
# keep the code a small badge under the eyes (~70% of the space below them)
target = max(20, int(min(W, H - _EYE_BAND - 1) * 0.72))
if iw <= target and ih <= target:
s = max(1, min(target // iw, target // ih)) # crisp integer up-scale (QR)
nw, nh = iw * s, ih * s
else:
s = min(target / iw, target / ih) # scale big images down
nw, nh = max(1, int(iw * s)), max(1, int(ih * s))
inner = inner.resize((nw, nh), Image.NEAREST)
canvas = Image.new("RGB", (W, H), (0, 0, 0))
g = ImageDraw.Draw(canvas)
eye = cf.DEFAULT_EYE
for cx in (W // 2 - 10, W // 2 + 10): # two eyes at the top
g.ellipse([cx - 5, 3, cx + 5, 13], fill=(255, 255, 255))
g.ellipse([cx - 3, 5, cx + 3, 11], fill=eye)
g.ellipse([cx - 1, 7, cx + 1, 10], fill=(0, 0, 0))
x = (W - nw) // 2
y = _EYE_BAND + (H - _EYE_BAND - nh) // 2
canvas.paste(inner, (max(0, x), max(_EYE_BAND, y)))
return cf.encode(canvas)
def _qr_bytes(url: str) -> bytes:
"""Render a QR for ``url`` FULL-SCREEN with the largest crisp (integer) module
size the 46-wide panel allows the only way it has any chance of scanning.
Only a ~version-1 QR (<=17 chars) reaches ~2 px/module; longer data is denser
and won't scan. Returns (bytes, qr_version)."""
_ensure_mask_path()
import qrcode
from PIL import Image
import colorface as cf
W, H = cf.DISPLAY_W, cf.DISPLAY_H
qr = qrcode.QRCode(error_correction=qrcode.constants.ERROR_CORRECT_L,
box_size=1, border=1)
qr.add_data(url)
qr.make(fit=True)
q = qr.make_image(fill_color=(255, 255, 255),
back_color=(0, 0, 0)).convert("RGB")
scale = max(1, min(W, H) // max(1, q.width)) # largest integer that fits
if scale > 1:
q = q.resize((q.width * scale, q.width * scale), Image.NEAREST)
canvas = Image.new("RGB", (W, H), (0, 0, 0))
canvas.paste(q, ((W - q.width) // 2, (H - q.height) // 2))
return cf.encode(canvas)
def _image_bytes(img) -> bytes:
"""Show an uploaded QR/image FULL-SCREEN, crisp (NEAREST) — best effort."""
_ensure_mask_path()
import colorface as cf
from PIL import Image
W, H = cf.DISPLAY_W, cf.DISPLAY_H
s = min(W, H)
img = img.convert("RGB").resize((s, s), Image.NEAREST)
canvas = Image.new("RGB", (W, H), (0, 0, 0))
canvas.paste(img, ((W - s) // 2, (H - s) // 2))
return cf.encode(canvas)
def show_social_on_mask(account: str) -> dict:
"""Show the account's **scannable** QR on the mask — a version-1 QR made from
a short (da.gd) link that redirects to the Instagram profile. Shared by the
dashboard button and the Gemini ``show_social`` tool. Raises for an unknown
account; propagates FaceController errors (e.g. not connected)."""
acc = SOCIAL.get(str(account).strip().lower())
if not acc:
raise HTTPException(status_code=404, detail="unknown account")
data = _qr_bytes(acc.get("short") or acc["url"]) # v1 short link -> scannable
mf = _get_face()
res = mf.show_scratch_image(data)
log.info("showing scannable social QR on mask: %s (%s)", acc["handle"], acc.get("short"))
return {"ok": True, "handle": acc["handle"], "scannable": True, **(res or {})}
@router.get("/social")
async def list_social():
return {"accounts": [{"id": k, "handle": v["handle"]} for k, v in SOCIAL.items()]}
def _friendly(exc: Exception) -> HTTPException:
"""Map FaceController errors to clean HTTP responses (esp. the common
'mask not connected' usually the mask is off / far / held by the phone app)."""
if isinstance(exc, HTTPException):
return exc
msg = str(exc)
if "not connected" in msg or "not started" in msg or "MASK" in msg:
return HTTPException(status_code=503, detail=(
"Mask not connected — power it on, bring it close to the robot, and "
"free it from the phone app."))
log.exception("mask scratch op failed")
return HTTPException(status_code=500, detail="%s: %s" % (type(exc).__name__, msg))
@router.post("/social/{account}")
async def show_social(account: str):
try:
return await asyncio.to_thread(show_social_on_mask, account)
except Exception as exc:
raise _friendly(exc)
@router.post("/qr")
async def upload_qr(file: UploadFile = File(...)):
"""Upload an image (a QR you generated, or any picture) and show it on the mask."""
raw = await file.read()
if not raw:
raise HTTPException(status_code=400, detail="empty upload")
from PIL import Image
try:
img = Image.open(io.BytesIO(raw))
img.load()
except Exception:
raise HTTPException(status_code=400, detail="not a valid image")
try:
data = await asyncio.to_thread(_image_bytes, img)
mf = _get_face()
return await asyncio.to_thread(mf.show_scratch_image, data)
except Exception as exc:
raise _friendly(exc)
@router.post("/face/resume")
async def resume_face():
"""Stop showing the scratch image and resume the animated face."""
mf = _get_face()
return await asyncio.to_thread(mf.set_expression, None)
@router.post("/face/mouth")
async def face_mouth(hidden: bool = Query(...)):
"""Show (hidden=false) or hide (hidden=true) the mouth on the animated face."""
mf = _get_face()
return await asyncio.to_thread(mf.set_mouth_hidden, hidden)
@router.post("/link")
async def face_link(on: bool = Query(...)):
"""Link (on=true) / unlink (on=false) Gemini <-> the mask.
ON connects the mask + lets Gemini drive its emotions/social.
OFF tears the link down (no BLE churn) and Gemini stops touching the mask.
Default state is OFF. Runs in a thread a link-on may briefly block while it
makes its first connect attempt."""
mf = _get_face()
return await asyncio.to_thread(mf.set_gemini_linked, on)
# ── saved QR library ────────────────────────────────────────────────
# Upload QR/images, save them by name, list/show/delete them. Stored as PNGs
# under data/qr_codes so they persist across restarts.
_QR_DIR = None
def _qr_dir() -> Path:
global _QR_DIR
if _QR_DIR is None:
try:
from Project.Sanad.config import BASE_DIR
base = Path(BASE_DIR)
except Exception:
base = Path(__file__).resolve().parents[2]
_QR_DIR = base / "data" / "qr_codes"
_QR_DIR.mkdir(parents=True, exist_ok=True)
return _QR_DIR
def _safe_name(name: str) -> str:
n = re.sub(r"[^A-Za-z0-9_.-]", "_", (name or "").strip())[:40].strip("._")
return n or "qr"
@router.post("/qr/save")
async def qr_save(name: str = Query(...), file: UploadFile = File(...)):
"""Save an uploaded QR/image into the library under ``name``."""
raw = await file.read()
if not raw:
raise HTTPException(status_code=400, detail="empty upload")
from PIL import Image
try:
img = Image.open(io.BytesIO(raw))
img.load()
except Exception:
raise HTTPException(status_code=400, detail="not a valid image")
sn = _safe_name(name)
await asyncio.to_thread(img.convert("RGB").save, str(_qr_dir() / (sn + ".png")))
return {"ok": True, "name": sn}
@router.post("/qr/save_link")
async def qr_save_link(name: str = Query(...), url: str = Query(...)):
"""Generate a QR from ``url`` and save it to the library. Returns the QR
version + whether it's short enough to actually scan on the mask (version 1)."""
u = (url or "").strip()
if not u:
raise HTTPException(status_code=400, detail="empty url")
_ensure_mask_path()
import qrcode
qr = qrcode.QRCode(error_correction=qrcode.constants.ERROR_CORRECT_L,
box_size=10, border=2)
qr.add_data(u)
qr.make(fit=True)
img = qr.make_image(fill_color=(255, 255, 255),
back_color=(0, 0, 0)).convert("RGB")
sn = _safe_name(name or u)
await asyncio.to_thread(img.save, str(_qr_dir() / (sn + ".png")))
return {"ok": True, "name": sn, "version": qr.version,
"scannable_on_mask": qr.version <= 1,
"note": ("scannable" if qr.version <= 1 else
"too dense to scan on the mask — use a shorter link")}
@router.get("/qr/library")
async def qr_library():
"""List the saved QR names."""
return {"qr": sorted(p.stem for p in _qr_dir().glob("*.png"))}
@router.get("/qr/thumb/{name}")
async def qr_thumb(name: str):
"""Serve a saved QR image (for the dashboard thumbnail)."""
p = _qr_dir() / (_safe_name(name) + ".png")
if not p.exists():
raise HTTPException(status_code=404, detail="not found")
return FileResponse(str(p), media_type="image/png")
@router.post("/qr/show/{name}")
async def qr_show(name: str):
"""Show a saved QR (under the eyes) on the mask."""
p = _qr_dir() / (_safe_name(name) + ".png")
if not p.exists():
raise HTTPException(status_code=404, detail="not found")
from PIL import Image
try:
img = Image.open(p)
data = await asyncio.to_thread(_image_bytes, img)
mf = _get_face()
return await asyncio.to_thread(mf.show_scratch_image, data)
except Exception as exc:
raise _friendly(exc)
@router.delete("/qr/{name}")
async def qr_delete(name: str):
"""Delete a saved QR from the library."""
p = _qr_dir() / (_safe_name(name) + ".png")
if p.exists():
p.unlink()
return {"ok": True, "deleted": _safe_name(name)}
# ── saved TEXT library ──────────────────────────────────────────────
# Save words/phrases and scroll any of them across the mask on demand.
_TEXT_DIR = None
def _text_dir() -> Path:
global _TEXT_DIR
if _TEXT_DIR is None:
try:
from Project.Sanad.config import BASE_DIR
base = Path(BASE_DIR)
except Exception:
base = Path(__file__).resolve().parents[2]
_TEXT_DIR = base / "data" / "mask_texts"
_TEXT_DIR.mkdir(parents=True, exist_ok=True)
return _TEXT_DIR
@router.post("/texts/save")
async def text_save(text: str = Query(...), name: str = Query("")):
"""Save a word/phrase to the text library (name defaults to the text)."""
t = (text or "").strip()[:200]
if not t:
raise HTTPException(status_code=400, detail="empty text")
nm = _safe_name(name or t)
await asyncio.to_thread((_text_dir() / (nm + ".txt")).write_text, t)
return {"ok": True, "name": nm, "text": t}
@router.get("/texts/library")
async def text_library():
"""List the saved texts."""
out = []
for p in sorted(_text_dir().glob("*.txt")):
try:
out.append({"name": p.stem, "text": p.read_text()[:80]})
except Exception:
pass
return {"texts": out}
@router.post("/texts/show/{name}")
async def text_show(name: str):
"""Scroll a saved text across the mask."""
p = _text_dir() / (_safe_name(name) + ".txt")
if not p.exists():
raise HTTPException(status_code=404, detail="not found")
txt = p.read_text()
mf = _get_face()
try:
return await asyncio.to_thread(mf.set_text, txt, (255, 255, 255), None, None, 38)
except Exception as exc:
raise _friendly(exc)
@router.delete("/texts/{name}")
async def text_delete(name: str):
"""Delete a saved text."""
p = _text_dir() / (_safe_name(name) + ".txt")
if p.exists():
p.unlink()
return {"ok": True, "deleted": _safe_name(name)}

View File

@ -10,6 +10,23 @@ from pydantic import BaseModel
router = APIRouter()
def _block_if_movement_armed():
"""409 if locomotion movement is armed — arm actions are mutually exclusive
with walking. The arm controller's own motion-block is the safety net; this
just gives the dashboard a clear message instead of a silent no-op."""
try:
from Project.Sanad.main import loco_controller # type: ignore
armed = loco_controller is not None and loco_controller.is_armed()
except HTTPException:
raise
except Exception:
return
if armed:
raise HTTPException(
409, "Arm actions are disabled while movement is enabled. "
"Disable movement in the Controller tab first.")
@router.get("/status")
async def motion_status():
from Project.Sanad.main import arm
@ -33,6 +50,7 @@ async def trigger_action(payload: TriggerPayload):
from Project.Sanad.main import arm
if arm is None:
raise HTTPException(503, "Arm controller not attached.")
_block_if_movement_armed()
speed = max(0.1, min(payload.speed, 5.0))

View File

@ -0,0 +1,402 @@
"""Navigation tab — proxy to the web_nav3 Nav2 stack.
Routes live under /api/nav (the prefix is applied centrally in dashboard/app.py,
NOT here). This router is a thin HTTP proxy: it forwards dashboard requests to a
single module-level WebNav3Client, which itself talks to the standalone web_nav3
FastAPI service (default http://127.0.0.1:8765 + rosbridge on :9090).
Fault isolation, two layers:
1. The `from ...navigation import WebNav3Client` import is GUARDED. If the
navigation package can't be imported (missing dep, syntax error), this
module still imports cleanly `_CLIENT` is None and every handler degrades
(GET /status returns {"available": False}; actions raise 503). This mirrors
how app.py loads each router in isolation.
2. WebNav3Client never raises into us by contract every method returns a
clean dict / NavStatus even when web_nav3 is unreachable so handlers just
forward the result. Blocking HTTP calls run off the event loop.
"""
from __future__ import annotations
import asyncio
from fastapi import APIRouter, HTTPException, Query
from pydantic import BaseModel
from Project.Sanad.core.logger import get_logger
from Project.Sanad.dashboard.routes import _arbiter
log = get_logger("navigation_route")
# Module-level router with NO prefix and NO tags — those are supplied by
# app.include_router(prefix="/api/nav", tags=["navigation"]) at registration time.
router = APIRouter()
# ── guarded optional import ─────────────────────────────────
# A broken navigation package must NOT stop this route module from importing —
# app.py would otherwise log the whole router as failed. Guard it and degrade.
try:
from Project.Sanad.navigation import WebNav3Client # type: ignore
_IMPORT_ERROR: str | None = None
except Exception as exc: # noqa: BLE001
WebNav3Client = None # type: ignore[assignment,misc]
_IMPORT_ERROR = f"{type(exc).__name__}: {exc}"
log.warning("navigation client unavailable — nav routes degraded: %s", _IMPORT_ERROR)
# ── config (env var -> dashboard config section -> default) ──
def _nav_config() -> dict:
"""Resolve nav connection config. Precedence: env var -> config -> default."""
import os
from Project.Sanad.core.config_loader import section as _cfg_section
cfg = _cfg_section("dashboard", "navigation")
web_nav3_url = (
os.environ.get("WEB_NAV3_URL")
or cfg.get("web_nav3_url")
or "http://127.0.0.1:8765"
)
rosbridge_url = (
os.environ.get("ROSBRIDGE_URL")
or cfg.get("rosbridge_url")
or "ws://127.0.0.1:9090"
)
robot = os.environ.get("SANAD_ROBOT_NAME") or cfg.get("robot") or "sanad"
return {
"web_nav3_url": str(web_nav3_url),
"rosbridge_url": str(rosbridge_url),
"robot": str(robot),
}
_CFG = _nav_config()
# ── single module-level client ──────────────────────────────
# One WebNav3Client for the whole dashboard, built from config. If the import
# was guarded out (above), or construction fails, _CLIENT stays None and every
# handler degrades gracefully.
if WebNav3Client is not None:
try:
_CLIENT = WebNav3Client(base_url=_CFG["web_nav3_url"], robot=_CFG["robot"])
log.info("WebNav3Client ready → %s (robot=%s)", _CFG["web_nav3_url"], _CFG["robot"])
except Exception as exc: # noqa: BLE001
_CLIENT = None
_IMPORT_ERROR = f"construct failed: {type(exc).__name__}: {exc}"
log.warning("WebNav3Client construction failed — nav routes degraded: %s", exc)
else:
_CLIENT = None
def _require():
"""Return the live client or raise 503 (for ACTION endpoints)."""
if _CLIENT is None:
raise HTTPException(503, f"Navigation client unavailable. {_IMPORT_ERROR or ''}".strip())
return _CLIENT
def _claim_nav():
"""Arbitration gate: refuse to start a Nav2 goal while manual loco owns legs."""
if not _arbiter.acquire_nav():
raise HTTPException(
409, "Manual movement (Controller) is armed. Disarm it before navigating."
)
# ── request bodies ──────────────────────────────────────────
class _NameBody(BaseModel):
name: str
class _IdBody(BaseModel):
id: object # mission ids may be int or str; forward as-is
class _StartBody(BaseModel):
mode: int = 2 # web_nav3 launch mode (e.g. 3 = localize against a saved map)
db_path: str | None = None # saved map to load (None = build fresh)
class _PoseBody(BaseModel):
name: str
x: float
y: float
yaw: float = 0.0
class _RenameBody(BaseModel):
old: str
new: str
# ── status (never raises — degraded body when unavailable) ──
@router.get("/status")
async def status():
if _CLIENT is None:
return {"available": False, "error": _IMPORT_ERROR}
nav = await asyncio.to_thread(_CLIENT.status)
# WebNav3Client.status() returns a NavStatus dataclass.
body = nav.as_dict() if hasattr(nav, "as_dict") else dict(nav)
body["available"] = True
return body
# ── places / navigation ─────────────────────────────────────
@router.get("/places")
async def places(map_name: str | None = Query(None, alias="map")):
"""List saved places. Per-MAP when ?map=<name> is given (each map keeps
its own places); else the legacy per-robot store."""
client = _require()
return await asyncio.to_thread(client.list_places, map_name)
@router.post("/goto")
async def goto(body: _NameBody):
client = _require()
_claim_nav()
res = await asyncio.to_thread(client.goto, body.name)
# A failed dispatch never drove the legs — release the gate so manual loco
# isn't locked out by a goto that never started.
if isinstance(res, dict) and not res.get("ok", True):
_arbiter.release_nav()
return res
@router.post("/start")
async def start(body: _StartBody):
client = _require()
return await asyncio.to_thread(client.start, body.mode, body.db_path)
class _DbBody(BaseModel):
db_path: str
@router.post("/load_map")
async def load_map(body: _DbBody):
"""View a saved map: stop any running bringup, then localize against it."""
client = _require()
return await asyncio.to_thread(client.load_map, body.db_path)
@router.post("/cancel")
async def cancel():
client = _require()
res = await asyncio.to_thread(client.cancel)
# WebNav3Client.cancel() is a no-op server-side (it only returns a note),
# so releasing the arbiter without truly stopping Nav2 would let the robot
# keep driving while manual loco re-acquires the legs (double-drive). Send a
# REAL goal-cancel over rosbridge first, and disarm the arrival monitor so a
# stale terminal can't fire, THEN release.
try:
from Project.Sanad.navigation.goal_monitor import request_cancel, disarm
disarm()
cancelled = await asyncio.to_thread(request_cancel)
if isinstance(res, dict):
res = {**res, "cancel_sent": bool(cancelled)}
except Exception as exc: # noqa: BLE001
log.debug("goal cancel skipped: %s", exc)
_arbiter.release_nav()
return res
@router.post("/save_here")
async def save_here(body: _NameBody):
client = _require()
return await asyncio.to_thread(client.save_here, body.name)
@router.post("/save_at")
async def save_at(body: _PoseBody, map_name: str | None = Query(None, alias="map")):
"""Save a named place at a map coordinate (from clicking the map). Per-MAP
when ?map=<name> given. Re-saving an existing name MOVES the place."""
client = _require()
return await asyncio.to_thread(client.save_at, body.name, body.x, body.y, body.yaw, map_name)
@router.post("/places/delete")
async def delete_place(body: _NameBody, map_name: str | None = Query(None, alias="map")):
"""Delete a saved place (per-map)."""
client = _require()
return await asyncio.to_thread(client.delete_place, body.name, map_name)
@router.post("/places/rename")
async def rename_place(body: _RenameBody, map_name: str | None = Query(None, alias="map")):
"""Rename a saved place (per-map)."""
client = _require()
return await asyncio.to_thread(client.rename_place, body.old, body.new, map_name)
class _MapEditsBody(BaseModel):
edits: list # [[world_x, world_y, value], ...] value 0=free/erase, 100=wall
@router.get("/map_edits")
async def get_map_edits(map_name: str = Query(..., alias="map")):
"""Saved edit overlay for a map (erased points + painted walls)."""
client = _require()
return await asyncio.to_thread(client.get_map_edits, map_name)
@router.post("/map_edits")
async def save_map_edits(body: _MapEditsBody, map_name: str = Query(..., alias="map")):
"""Persist a map's edit overlay (Map Editor)."""
client = _require()
return await asyncio.to_thread(client.save_map_edits, map_name, body.edits)
class _VoiceGotoBody(BaseModel):
place: str
def _resolve_place(client, spoken: str) -> dict:
"""Resolve a spoken place name against the ACTIVE map's places.
Strategy: exact (case-insensitive) single substring candidate
ambiguous / unknown. Returns a dict the caller (and ultimately Gemini)
can act on. Never raises.
"""
try:
st = client.status()
body = st.as_dict() if hasattr(st, "as_dict") else dict(st)
except Exception as exc: # noqa: BLE001
return {"ok": False, "reason": "status_error", "detail": str(exc)[:160]}
if not body.get("bringup_alive"):
return {"ok": False, "reason": "no_map",
"detail": "No navigation session is running — load a map first."}
active_map = body.get("active_map")
try:
places = client.list_places(active_map) or []
except Exception: # noqa: BLE001
places = []
names = [p.get("name") for p in places if isinstance(p, dict) and p.get("name")]
sl = (spoken or "").strip().lower()
if not sl:
return {"ok": False, "reason": "no_place", "map": active_map, "places": names}
exact = [n for n in names if n.lower() == sl]
if exact:
return {"ok": True, "resolved": exact[0], "map": active_map}
subs = []
for n in names:
nl = n.lower()
if sl in nl or nl in sl:
subs.append(n)
subs = list(dict.fromkeys(subs)) # de-dup, preserve order
if len(subs) == 1:
return {"ok": True, "resolved": subs[0], "map": active_map}
if len(subs) > 1:
return {"ok": False, "reason": "ambiguous", "candidates": subs, "map": active_map}
return {"ok": False, "reason": "unknown_place", "candidates": names, "map": active_map}
@router.get("/active")
async def active():
"""Navigation context for Gemini: the active map, its mode, and that map's
place names one call so the voice tools (list_places / where_am_i) don't
have to guess the active map."""
client = _require()
st = await asyncio.to_thread(client.status)
body = st.as_dict() if hasattr(st, "as_dict") else dict(st)
places = []
if body.get("bringup_alive"):
try:
pl = await asyncio.to_thread(client.list_places, body.get("active_map"))
places = [p.get("name") for p in (pl or [])
if isinstance(p, dict) and p.get("name")]
except Exception: # noqa: BLE001
places = []
return {
"map": body.get("active_map"),
"mode": body.get("mode"),
"mode_label": body.get("mode_label"),
"localizing": bool(body.get("localizing")),
"bringup_alive": bool(body.get("bringup_alive")),
"places": places,
}
@router.post("/voice_goto")
async def voice_goto(body: _VoiceGotoBody):
"""Resolve a spoken place name and drive there — Gemini's navigate_to_place.
Arbiter-gated (claims the legs for Nav2) and arms the arrival monitor so
Gemini later hears [NAV ARRIVED]/[NAV FAILED]. Never raises into the caller;
returns a structured result the model can speak from.
"""
client = _require()
res = await asyncio.to_thread(_resolve_place, client, body.place or "")
if not res.get("ok"):
return res
# Claim the legs for Nav2 — refuse (don't raise) if manual loco is armed.
if not _arbiter.acquire_nav():
return {"ok": False, "reason": "manual_armed",
"detail": "Manual movement (Controller) is armed — disarm it to navigate."}
drive = await asyncio.to_thread(client.goto, res["resolved"])
if isinstance(drive, dict) and not drive.get("ok", True):
_arbiter.release_nav()
return {"ok": False, "reason": "dispatch_failed",
"resolved": res["resolved"], "detail": drive}
# Arm arrival monitoring (best-effort; absence must not fail the drive).
try:
from Project.Sanad.navigation.goal_monitor import arm_goal
arm_goal(res["resolved"])
except Exception as exc: # noqa: BLE001
log.debug("goal monitor arm skipped: %s", exc)
return {"ok": True, "resolved": res["resolved"], "map": res.get("map")}
@router.post("/goto_pose")
async def goto_pose(body: _PoseBody):
"""Arbiter-gate a coordinate nav goal (click-to-drive).
The browser publishes the actual /goal_pose over rosbridge; this only
CLAIMS the legs for Nav2 (409 if manual loco is armed) so the two stacks
never both drive. The frontend sends the goal only after this returns ok.
"""
_require()
_claim_nav()
# Arm the arrival monitor so this click-to-drive goal releases the arbiter
# when it ends — without this, nav_active stays True forever after the goal
# completes (the browser publishes the goal but never arms anything).
try:
from Project.Sanad.navigation.goal_monitor import arm_goal
arm_goal(f"({body.x:.1f}, {body.y:.1f})")
except Exception as exc: # noqa: BLE001
log.debug("goal monitor arm skipped: %s", exc)
return {"ok": True, "x": body.x, "y": body.y, "yaw": body.yaw}
# ── maps / missions ─────────────────────────────────────────
@router.get("/maps")
async def maps():
client = _require()
return await asyncio.to_thread(client.list_maps)
@router.get("/missions")
async def missions():
client = _require()
return await asyncio.to_thread(client.list_missions)
@router.post("/missions/run")
async def run_mission(body: _IdBody):
client = _require()
_claim_nav()
res = await asyncio.to_thread(client.run_mission, body.id)
if isinstance(res, dict) and not res.get("ok", True):
_arbiter.release_nav()
return res
# ── config (what the SPA needs to render links / connect) ───
@router.get("/config")
async def config():
return {
"web_nav3_url": _CFG["web_nav3_url"],
"rosbridge_url": _CFG["rosbridge_url"],
"robot": _CFG["robot"],
}

View File

@ -0,0 +1,457 @@
"""Recognition tab — camera vision + face gallery + hot toggles.
Single router covering:
- Vision / Face Recognition toggles (hot no Gemini restart needed)
- Live camera preview (latest JPEG drop)
- Face gallery CRUD: enroll, upload, capture, rename, delete, ZIP
- Per-photo download + delete
Toggle changes write data/.recognition_state.json atomically. The Gemini
child polls that file at 1 Hz and applies changes mid-session.
"""
from __future__ import annotations
import io
from typing import Optional
from fastapi import APIRouter, File, HTTPException, Query, UploadFile
from fastapi.responses import FileResponse, Response, StreamingResponse
from pydantic import BaseModel
from Project.Sanad.config import BASE_DIR
from Project.Sanad.core.logger import get_logger
from Project.Sanad.dashboard.routes._safe_io import check_upload_size
from Project.Sanad.vision import recognition_state
log = get_logger("recognition_routes")
router = APIRouter()
# ── paths (resolved from BASE_DIR) ──────────────────────────
STATE_PATH = BASE_DIR / "data" / ".recognition_state.json"
FACES_DIR = BASE_DIR / "data" / "faces"
ALLOWED_IMAGE_EXTS = {".jpg", ".jpeg", ".png"}
# ── helpers ─────────────────────────────────────────────────
def _get_camera():
"""Lazy import to avoid circular import on dashboard load."""
try:
from Project.Sanad.main import camera # type: ignore
return camera
except Exception:
return None
def _get_gallery():
"""Lazy import — same reason."""
try:
from Project.Sanad.main import gallery # type: ignore
return gallery
except Exception:
return None
def _bump_and_write_state(**changes) -> recognition_state.RecognitionState:
"""Apply changes (vision_enabled, face_rec_enabled) and persist."""
return recognition_state.mutate(STATE_PATH, **changes)
def _bump_gallery_version() -> int:
cur = recognition_state.read(STATE_PATH)
new_version = cur.gallery_version + 1
recognition_state.mutate(STATE_PATH, gallery_version=new_version)
return new_version
# ── state + toggles ─────────────────────────────────────────
@router.get("/state")
async def get_state():
"""Return the current toggle/camera/gallery state."""
st = recognition_state.read(STATE_PATH)
cam = _get_camera()
gallery = _get_gallery()
faces_count = 0
photos_count = 0
if gallery is not None:
try:
entries = gallery.list()
faces_count = len(entries)
photos_count = sum(len(e.sample_paths) for e in entries)
except Exception:
pass
return {
"vision_enabled": st.vision_enabled,
"face_rec_enabled": st.face_rec_enabled,
"gallery_version": st.gallery_version,
"camera": cam.status() if cam is not None else {
"running": False, "backend": None, "error": "camera subsystem unavailable"
},
"faces_count": faces_count,
"photos_count": photos_count,
}
@router.post("/vision")
async def set_vision(on: bool = Query(...)):
"""Enable / disable camera vision (hot — no Gemini restart)."""
cam = _get_camera()
if cam is None:
log.warning("vision toggle requested but camera subsystem unavailable")
raise HTTPException(503, "Camera subsystem not available.")
if on and not cam.is_running():
ok = cam.start()
if not ok:
log.warning("vision ON requested but camera.start() failed: %s",
cam.error or "no backend")
_bump_and_write_state(vision_enabled=False)
raise HTTPException(503,
f"Camera could not start (no backend). {cam.error or ''}")
elif (not on) and cam.is_running():
cam.stop()
st = _bump_and_write_state(vision_enabled=bool(on))
log.info("vision %s (backend=%s)", "ON" if on else "OFF",
cam.backend if cam.is_running() else "none")
return {"ok": True, "vision_enabled": st.vision_enabled,
"camera": cam.status()}
@router.post("/face-rec")
async def set_face_rec(on: bool = Query(...)):
"""Enable / disable face recognition (hot — no Gemini restart).
The Gemini child picks the change up within ~1 s: ON re-sends the
gallery primer and tells Gemini it can recognise people; OFF tells
Gemini to disregard the gallery and stop identifying anyone. Both
take effect on the live session no reconnect needed.
"""
st = _bump_and_write_state(face_rec_enabled=bool(on))
log.info("face recognition %s", "ON" if on else "OFF")
return {"ok": True, "face_rec_enabled": st.face_rec_enabled}
@router.post("/sync")
async def sync_gallery():
"""Bump gallery_version so the child re-sends the primer if face-rec is ON."""
v = _bump_gallery_version()
log.info("gallery sync requested → v.%d", v)
return {"ok": True, "gallery_version": v}
# ── live preview ────────────────────────────────────────────
@router.get("/frame.jpg")
async def latest_frame():
"""Serve the most recent camera frame straight from the daemon's
in-memory cache (no file drop frames are also pushed to the Gemini
child over its stdin)."""
cam = _get_camera()
if cam is None:
raise HTTPException(503, "Camera subsystem unavailable.")
jpeg = cam.snapshot_jpeg()
if not jpeg:
raise HTTPException(404, "No frame captured yet.")
return Response(
content=jpeg,
media_type="image/jpeg",
headers={"Cache-Control": "no-store, must-revalidate"},
)
# ── camera resolution / quality ─────────────────────────────
class CameraConfigPayload(BaseModel):
width: Optional[int] = None
height: Optional[int] = None
fps: Optional[int] = None
jpeg_quality: Optional[int] = None
@router.post("/camera-config")
async def set_camera_config(payload: CameraConfigPayload):
"""Hot-swap the camera capture profile (resolution / fps / JPEG quality).
If the camera is running, CameraDaemon.reconfigure() rebuilds the
pipeline at the new profile (~0.5 s gap). If idle, the values just
take effect on the next start. Bounds are sanity-checked here so a
fat-fingered value can't wedge the daemon."""
cam = _get_camera()
if cam is None:
raise HTTPException(503, "Camera subsystem unavailable.")
if payload.width is not None and not (160 <= payload.width <= 1920):
raise HTTPException(400, "width out of range (1601920)")
if payload.height is not None and not (120 <= payload.height <= 1080):
raise HTTPException(400, "height out of range (1201080)")
if payload.fps is not None and not (1 <= payload.fps <= 60):
raise HTTPException(400, "fps out of range (160)")
if payload.jpeg_quality is not None and not (10 <= payload.jpeg_quality <= 95):
raise HTTPException(400, "jpeg_quality out of range (1095)")
profile = cam.reconfigure(
width=payload.width, height=payload.height,
fps=payload.fps, jpeg_quality=payload.jpeg_quality,
)
log.info("camera reconfigured via dashboard → %s", profile)
return {"ok": True, "profile": profile, "camera": cam.status()}
# ── face gallery routes ─────────────────────────────────────
def _validate_image(content: bytes, filename: str | None = None) -> None:
"""Reject non-JPEG/PNG content + oversize uploads."""
check_upload_size(content)
if len(content) < 16:
raise HTTPException(400, "Image too small / empty.")
is_jpeg = content[:3] == b"\xff\xd8\xff"
is_png = content[:8] == b"\x89PNG\r\n\x1a\n"
if not (is_jpeg or is_png):
raise HTTPException(
400,
f"Only JPEG/PNG accepted (got {filename or 'unknown'}).",
)
def _entry_to_dict(entry) -> dict:
photos = []
for p in entry.sample_paths:
try:
photos.append({"name": p.name, "size_bytes": p.stat().st_size})
except OSError:
continue
return {
"id": entry.id,
"name": entry.name,
"description": entry.description,
"added_at": entry.added_at,
"photos": photos,
}
@router.get("/faces")
async def list_faces():
gallery = _get_gallery()
if gallery is None:
raise HTTPException(503, "Face gallery subsystem unavailable.")
entries = gallery.list()
return {"faces": [_entry_to_dict(e) for e in entries],
"total": len(entries)}
class RenamePayload(BaseModel):
name: Optional[str] = None
class DescribePayload(BaseModel):
description: Optional[str] = None
@router.post("/faces/enroll")
async def enroll_from_camera(name: Optional[str] = Query(default=None),
description: Optional[str] = Query(default=None)):
"""Create a new face from the camera's latest snapshot."""
gallery = _get_gallery()
if gallery is None:
raise HTTPException(503, "Face gallery subsystem unavailable.")
cam = _get_camera()
if cam is None or not cam.is_running():
raise HTTPException(409, "Camera is not running. Toggle Vision ON first.")
# get_fresh_frame waits briefly for a current frame so the enrolled
# photo is the scene the user is posing for, not a stale buffer.
jpeg = cam.get_fresh_frame(max_age_s=0.5, timeout_s=1.5)
if not jpeg:
raise HTTPException(409, "Camera has no frame yet. Wait a moment and retry.")
entry = gallery.create_face(
[jpeg],
name=name.strip() if name else None,
description=description.strip() if description else None,
)
v = _bump_gallery_version()
log.info("enrolled face_%d via camera (name=%s, desc=%s, v.%d)",
entry.id, name or "(unnamed)",
"yes" if description else "no", v)
return {"ok": True, "face": _entry_to_dict(entry)}
@router.post("/faces/upload")
async def enroll_from_upload(
files: list[UploadFile] = File(...),
name: Optional[str] = Query(default=None),
description: Optional[str] = Query(default=None),
):
"""Create a new face from uploaded image file(s)."""
gallery = _get_gallery()
if gallery is None:
raise HTTPException(503, "Face gallery subsystem unavailable.")
if not files:
raise HTTPException(400, "At least one image file required.")
image_bytes: list[bytes] = []
for f in files:
content = await f.read()
_validate_image(content, f.filename)
image_bytes.append(content)
entry = gallery.create_face(
image_bytes,
name=name.strip() if name else None,
description=description.strip() if description else None,
)
v = _bump_gallery_version()
log.info("enrolled face_%d via upload (%d photos, name=%s, desc=%s, v.%d)",
entry.id, len(image_bytes), name or "(unnamed)",
"yes" if description else "no", v)
return {"ok": True, "face": _entry_to_dict(entry)}
@router.post("/faces/{face_id}/capture")
async def capture_to_face(face_id: int):
"""Add a new sample (from the camera) to an existing face."""
gallery = _get_gallery()
if gallery is None:
raise HTTPException(503, "Face gallery subsystem unavailable.")
cam = _get_camera()
if cam is None or not cam.is_running():
raise HTTPException(409, "Camera is not running. Toggle Vision ON first.")
jpeg = cam.get_fresh_frame(max_age_s=0.5, timeout_s=1.5)
if not jpeg:
raise HTTPException(409, "Camera has no frame yet.")
try:
fname = gallery.add_photo(face_id, jpeg)
except FileNotFoundError as exc:
raise HTTPException(404, str(exc))
v = _bump_gallery_version()
log.info("captured new photo for face_%d%s (v.%d)", face_id, fname, v)
return {"ok": True, "added": fname, "face": _entry_to_dict(gallery.get(face_id))}
@router.post("/faces/{face_id}/upload")
async def upload_to_face(face_id: int, files: list[UploadFile] = File(...)):
"""Add one or more uploaded samples to an existing face."""
gallery = _get_gallery()
if gallery is None:
raise HTTPException(503, "Face gallery subsystem unavailable.")
if gallery.get(face_id) is None:
raise HTTPException(404, f"face_{face_id} not found")
added: list[str] = []
for f in files:
content = await f.read()
_validate_image(content, f.filename)
try:
fname = gallery.add_photo(face_id, content)
added.append(fname)
except FileNotFoundError as exc:
raise HTTPException(404, str(exc))
v = _bump_gallery_version()
log.info("uploaded %d photo(s) to face_%d (v.%d)", len(added), face_id, v)
return {"ok": True, "added": added,
"face": _entry_to_dict(gallery.get(face_id))}
@router.post("/faces/{face_id}/rename")
async def rename_face(face_id: int, payload: RenamePayload):
gallery = _get_gallery()
if gallery is None:
raise HTTPException(503, "Face gallery subsystem unavailable.")
try:
gallery.rename(face_id, payload.name)
except FileNotFoundError as exc:
raise HTTPException(404, str(exc))
v = _bump_gallery_version()
log.info("renamed face_%d%s (v.%d)", face_id,
payload.name or "(unnamed)", v)
return {"ok": True, "face": _entry_to_dict(gallery.get(face_id))}
@router.post("/faces/{face_id}/describe")
async def describe_face(face_id: int, payload: DescribePayload):
"""Set / clear a face's free-text description. The description is
folded into the Gemini primer turn so Gemini can reference it."""
gallery = _get_gallery()
if gallery is None:
raise HTTPException(503, "Face gallery subsystem unavailable.")
try:
gallery.set_description(face_id, payload.description)
except FileNotFoundError as exc:
raise HTTPException(404, str(exc))
v = _bump_gallery_version()
log.info("described face_%d (%s, v.%d)", face_id,
"set" if payload.description else "cleared", v)
return {"ok": True, "face": _entry_to_dict(gallery.get(face_id))}
@router.delete("/faces/{face_id}")
async def delete_face(face_id: int):
gallery = _get_gallery()
if gallery is None:
raise HTTPException(503, "Face gallery subsystem unavailable.")
try:
gallery.delete_face(face_id)
except FileNotFoundError as exc:
raise HTTPException(404, str(exc))
v = _bump_gallery_version()
log.info("deleted face_%d (v.%d)", face_id, v)
return {"ok": True, "deleted": face_id}
@router.delete("/faces/{face_id}/photo/{photo_name}")
async def delete_photo(face_id: int, photo_name: str):
gallery = _get_gallery()
if gallery is None:
raise HTTPException(503, "Face gallery subsystem unavailable.")
# safe filename — only allow simple file names, no traversal
if "/" in photo_name or ".." in photo_name or "\x00" in photo_name:
raise HTTPException(400, "Invalid photo name.")
try:
gallery.delete_photo(face_id, photo_name)
except FileNotFoundError as exc:
raise HTTPException(404, str(exc))
except ValueError as exc:
raise HTTPException(400, str(exc))
v = _bump_gallery_version()
log.info("deleted photo %s from face_%d (v.%d)", photo_name, face_id, v)
return {"ok": True, "deleted": photo_name}
@router.get("/faces/{face_id}/photo/{photo_name}")
async def get_photo(face_id: int, photo_name: str,
download: int = Query(default=0)):
"""Serve a single photo. Add ?download=1 for attachment disposition."""
gallery = _get_gallery()
if gallery is None:
raise HTTPException(503, "Face gallery subsystem unavailable.")
if "/" in photo_name or ".." in photo_name or "\x00" in photo_name:
raise HTTPException(400, "Invalid photo name.")
path = gallery.get_photo(face_id, photo_name)
if path is None:
raise HTTPException(404, "Photo not found.")
media = "image/png" if path.suffix.lower() == ".png" else "image/jpeg"
headers = {}
if download:
headers["Content-Disposition"] = (
f'attachment; filename="face_{face_id}_{photo_name}"'
)
return FileResponse(path, media_type=media, headers=headers)
@router.get("/faces/{face_id}/download.zip")
async def download_face_zip(face_id: int):
gallery = _get_gallery()
if gallery is None:
raise HTTPException(503, "Face gallery subsystem unavailable.")
try:
data = gallery.zip_face(face_id)
except FileNotFoundError as exc:
raise HTTPException(404, str(exc))
return StreamingResponse(
io.BytesIO(data),
media_type="application/zip",
headers={
"Content-Disposition": f'attachment; filename="face_{face_id}.zip"',
"Content-Length": str(len(data)),
},
)

View File

@ -23,6 +23,12 @@ router = APIRouter()
RECORDS_INDEX = AUDIO_RECORDINGS_DIR / "records.json"
_INDEX_LOCK = threading.Lock()
# Strong refs to fire-and-forget playback tasks. The event loop only keeps a
# weak reference to tasks, so an unreferenced create_task() result can be
# garbage-collected (cancelling playback) before it finishes. Mirror replay.py.
import asyncio as _asyncio # noqa: E402
_BG_TASKS: set[_asyncio.Task] = set()
def _load_index() -> dict[str, Any]:
if not RECORDS_INDEX.exists():
@ -109,10 +115,65 @@ async def play_record(payload: RecordPlay):
if not raw_path.exists():
raise HTTPException(404, f"File not found: {raw_path.name}")
from Project.Sanad.main import audio_mgr
import threading
# Fire-and-forget on a DEDICATED daemon thread — NOT asyncio.to_thread.
# to_thread runs on the shared default executor, which gets starved while
# the dashboard services the live-voice child's reconnect chatter; that
# delayed record playback by several seconds (clip silent, counter parked).
# A dedicated thread starts immediately regardless of executor/event-loop
# load. play_wav blocks for the clip duration and serves pause/stop via
# _play_state; the UI stays responsive because this handler returns now.
# Python keeps running threads alive, so no ref is needed to prevent GC.
threading.Thread(
target=audio_mgr.play_wav, args=(raw_path, payload.record_name),
name="record-playback", daemon=True,
).start()
return {"ok": True, "record_name": payload.record_name,
"file_kind": payload.file_kind, "path": str(raw_path)}
@router.post("/pause")
async def pause_playback():
from Project.Sanad.main import audio_mgr
return audio_mgr.pause_playback()
@router.post("/resume")
async def resume_playback():
from Project.Sanad.main import audio_mgr
return audio_mgr.resume_playback()
@router.post("/seek")
async def seek_playback(position_sec: float):
"""Jump to a position (seconds) in the currently-playing clip — used by the
waveform scrubber. No-op (ok=False) if nothing is playing."""
from Project.Sanad.main import audio_mgr
return audio_mgr.seek_playback(position_sec)
@router.post("/stop")
async def stop_playback():
from Project.Sanad.main import audio_mgr
import asyncio
await asyncio.to_thread(audio_mgr.play_wav, raw_path)
return {"ok": True, "record_name": payload.record_name, "file_kind": payload.file_kind, "path": str(raw_path)}
await asyncio.to_thread(audio_mgr.stop_playback)
return {"ok": True, "stopped": True}
@router.get("/playback-status")
async def playback_status():
from Project.Sanad.main import audio_mgr
return audio_mgr.playback_status()
@router.post("/live-hold")
async def set_live_hold(on: bool):
"""Manual hold for the live-Gemini pause. on=True pauses the live voice and
keeps it paused (records won't resume it) until on=False is sent. Default
behaviour (on=False) is AUTO: records pause Gemini only for the clip."""
from Project.Sanad.main import audio_mgr
return {"live_hold": audio_mgr.set_live_voice_hold(on)}
class RecordRename(BaseModel):
@ -183,7 +244,12 @@ async def delete_record(payload: RecordDelete):
deleted_files = []
for fi in deleted_entry.get("files", {}).values():
try:
p = Path(fi.get("path", "")).resolve()
# _resolve_path handles new-style basenames (resolved under
# AUDIO_RECORDINGS_DIR) as well as legacy absolute paths.
# A raw Path(basename) would resolve vs CWD and fall outside
# base, so the relative_to guard would skip the unlink and the
# WAV would be orphaned on disk. Mirror play_record/rename_record.
p = _resolve_path(fi.get("path", "")).resolve()
p.relative_to(base) # never delete files outside recordings dir
except (ValueError, OSError):
continue
@ -194,3 +260,43 @@ async def delete_record(payload: RecordDelete):
index["records"] = kept
_save_index(index)
return {"ok": True, "deleted": payload.record_name, "deleted_files": deleted_files}
class RecordBulkDelete(BaseModel):
record_names: list[str] | None = None
all: bool = False
@router.post("/delete-bulk")
async def delete_bulk(payload: RecordBulkDelete):
"""Delete many records in one call. all=True wipes every record; otherwise
only those in record_names. Files are unlinked, guarded to the recordings
dir (same safety as /delete)."""
names = set(payload.record_names or [])
with _INDEX_LOCK:
index = _load_index()
base = AUDIO_RECORDINGS_DIR.resolve()
kept: list = []
removed: list = []
deleted_files = 0
for r in index.get("records", []):
if payload.all or r.get("record_name") in names:
removed.append(r.get("record_name"))
for fi in r.get("files", {}).values():
try:
p = _resolve_path(fi.get("path", "")).resolve()
p.relative_to(base) # never delete outside recordings dir
except (ValueError, OSError):
continue
if p.exists():
try:
p.unlink()
deleted_files += 1
except OSError:
pass
else:
kept.append(r)
index["records"] = kept
_save_index(index)
return {"ok": True, "deleted": removed, "deleted_count": len(removed),
"deleted_files": deleted_files}

View File

@ -21,6 +21,22 @@ log = get_logger("replay_route")
router = APIRouter()
def _block_if_movement_armed():
"""409 when locomotion movement is armed — arm motion (replay / teaching) is
mutually exclusive with walking."""
try:
from Project.Sanad.main import loco_controller # type: ignore
armed = loco_controller is not None and loco_controller.is_armed()
except HTTPException:
raise
except Exception:
return
if armed:
raise HTTPException(
409, "Arm actions are disabled while movement is enabled. "
"Disable movement in the Controller tab first.")
# -- models --
class ReplayRequest(BaseModel):
@ -94,6 +110,7 @@ _BG_TASKS: set[asyncio.Task] = set()
async def test_replay(payload: ReplayRequest):
"""Test-play a motion file at the given speed."""
from Project.Sanad.main import arm
_block_if_movement_armed()
if arm.is_busy:
raise HTTPException(409, "Arm is busy.")
path = safe_path_under(MOTIONS_DIR, payload.name)
@ -114,9 +131,16 @@ async def test_replay(payload: ReplayRequest):
@router.post("/cancel")
async def cancel_replay():
"""Stop the current replay — the smooth return-to-home runs as the
final phase of the replay itself.
Matches g1_replay_v4_stable.py's behaviour: the play loop breaks on
the cancel flag, then the same Run() function executes its
return-home ramp + DisableSDK. No separate scheduling needed.
"""
from Project.Sanad.main import arm
arm.cancel()
return {"ok": True, "message": "Cancel signal sent."}
return {"ok": True, "message": "Cancelled — returning to home pose smoothly."}
@router.get("/status")
@ -135,6 +159,7 @@ async def start_teaching(payload: TeachRequest):
from Project.Sanad.main import teacher
if teacher is None:
raise HTTPException(503, "Teaching module not available.")
_block_if_movement_armed()
if teacher.is_recording:
raise HTTPException(409, "Teaching session already active.")
existing = MOTIONS_DIR / f"{payload.name}.jsonl"

View File

@ -2,6 +2,7 @@
from __future__ import annotations
import asyncio
from datetime import datetime
from pathlib import Path
@ -9,6 +10,7 @@ from fastapi import APIRouter, HTTPException
from pydantic import BaseModel
from Project.Sanad.config import SCRIPTS_DIR
from Project.Sanad.core import persona as _persona
from Project.Sanad.dashboard.routes._safe_io import (
atomic_write_text, MAX_UPLOAD_BYTES,
)
@ -31,6 +33,8 @@ def _safe_path(name: str) -> Path:
@router.get("/")
async def list_scripts():
SCRIPTS_DIR.mkdir(parents=True, exist_ok=True)
active = _persona.active_persona_name()
default = _persona.default_persona_name()
items = []
for p in sorted(SCRIPTS_DIR.iterdir(), key=lambda x: x.name.lower()):
if not p.is_file():
@ -40,8 +44,48 @@ async def list_scripts():
"name": p.name,
"size_bytes": st.st_size,
"modified_at": datetime.fromtimestamp(st.st_mtime).isoformat(timespec="seconds"),
"active": p.name == active, # the persona Gemini loads now
"is_default": p.name == default, # the fallback (sanad_script.txt)
})
return {"path": str(SCRIPTS_DIR), "files": items}
return {"path": str(SCRIPTS_DIR), "files": items,
"active": active, "default": default}
class ScriptActive(BaseModel):
name: str | None = None # None / "" / the default name → revert to default
restart: bool = False # also restart the live voice so it takes effect now
@router.get("/active")
async def get_active():
"""Which persona Gemini will load, and the default it falls back to."""
return {"active": _persona.active_persona_name(),
"default": _persona.default_persona_name()}
@router.post("/active")
async def set_active(payload: ScriptActive):
"""Select the persona script Gemini uses. With restart=true, the live voice
session is bounced so the new persona takes effect immediately; otherwise it
applies on the next voice (re)connect."""
try:
active = _persona.set_active_persona(payload.name)
except FileNotFoundError:
raise HTTPException(404, f"Script not found: {payload.name}")
restarted = False
if payload.restart:
try:
from Project.Sanad.main import live_sub
if live_sub is not None and hasattr(live_sub, "start"):
if hasattr(live_sub, "is_running") and live_sub.is_running():
await asyncio.to_thread(live_sub.stop)
await asyncio.sleep(1.5)
await asyncio.to_thread(live_sub.start)
restarted = True
except Exception:
pass # selection is saved regardless of restart success
return {"ok": True, "active": active,
"default": _persona.default_persona_name(), "restarted": restarted}
class ScriptLoad(BaseModel):
@ -116,5 +160,9 @@ async def delete_script(payload: ScriptDelete):
path = _safe_path(payload.name)
if not path.exists():
raise HTTPException(404, f"Not found: {payload.name}")
if path.name == _persona.default_persona_name():
raise HTTPException(409, f"Cannot delete the default persona ({path.name}).")
path.unlink()
# If the active selection was the deleted file, resolution auto-falls-back
# to the default — no extra cleanup needed.
return {"ok": True, "deleted": payload.name}

View File

@ -5,18 +5,24 @@ from __future__ import annotations
import asyncio
import os
import platform
import shutil
import socket
import sys
from pathlib import Path
from typing import Any
from fastapi import APIRouter
from fastapi import APIRouter, HTTPException
from pydantic import BaseModel
from Project.Sanad.config import (
AUDIO_RECORDINGS_DIR,
BASE_DIR,
DASHBOARD_HOST,
DASHBOARD_INTERFACE,
DASHBOARD_PORT,
DATA_DIR,
DDS_NETWORK_INTERFACE,
LOGS_DIR,
list_network_interfaces,
)
from Project.Sanad.core.logger import get_logger
@ -26,6 +32,36 @@ log = get_logger("system_route")
router = APIRouter()
def _runtime_bind() -> tuple[str, int]:
"""The host/port the server is ACTUALLY bound to.
main.py launches `uvicorn.run(_app, host=args.host, port=args.port)` with
the CLI --host/--port (start_sanad.sh passes `--port $PORT`, default 8001),
which can differ from the import-time DASHBOARD_HOST/DASHBOARD_PORT config
defaults (port 8000). Reading the live argv reports the real URL instead of
a stale config value. Falls back to the config constants when an arg is
absent (e.g. argparse default in effect)."""
host = DASHBOARD_HOST
port = DASHBOARD_PORT
argv = sys.argv
for i, tok in enumerate(argv):
if tok == "--host" and i + 1 < len(argv):
host = argv[i + 1]
elif tok.startswith("--host="):
host = tok.split("=", 1)[1]
elif tok == "--port" and i + 1 < len(argv):
try:
port = int(argv[i + 1])
except (TypeError, ValueError):
pass
elif tok.startswith("--port="):
try:
port = int(tok.split("=", 1)[1])
except (TypeError, ValueError):
pass
return host, port
def _safe_status(component, name: str) -> dict[str, Any]:
if component is None:
return {"available": False}
@ -90,8 +126,9 @@ async def system_info():
except Exception:
interfaces = []
# Determine the URL the dashboard is reachable at
bound_host = DASHBOARD_HOST
# Determine the URL the dashboard is reachable at — use the ACTUAL
# runtime bind args (argv), not the import-time config defaults.
bound_host, bound_port = _runtime_bind()
if bound_host == "0.0.0.0":
# Try to find the wlan0 IP for display purposes
up_ifaces = [i for i in interfaces if i["is_up"] and i["ip"] and not i["ip"].startswith("127.")]
@ -112,8 +149,8 @@ async def system_info():
"interface": DASHBOARD_INTERFACE,
"bound_host": bound_host,
"display_host": display_host,
"port": DASHBOARD_PORT,
"url": f"http://{display_host}:{DASHBOARD_PORT}",
"port": bound_port,
"url": f"http://{display_host}:{bound_port}",
},
"dds": {
"interface": DDS_NETWORK_INTERFACE,
@ -131,3 +168,148 @@ async def system_info():
}
return await asyncio.to_thread(_do)
# ───────────────────── storage tracking + cleanup ─────────────────────
# Categories surfaced in the Settings → Storage panel. `cleanable` ones get a
# Clean button + are included in "Clean all"; the rest (faces/motions/zones)
# are shown for tracking only — they're operational assets (enrollments,
# motion configs) managed in their own tabs, not disposable clutter.
_STORAGE_CATS = [
("recordings", "Conversation recordings", DATA_DIR / "recordings", True),
("records", "Named records (Typed Replay)", AUDIO_RECORDINGS_DIR, True),
("logs", "Logs", LOGS_DIR, True),
("faces", "Enrolled faces", DATA_DIR / "faces", False),
("motions", "Motion replays + config", DATA_DIR / "motions", False),
("photos", "Photos", DATA_DIR / "photos", False),
("zones", "Vision zones", DATA_DIR / "zones", False),
]
_CLEANABLE = {k for k, _l, _p, c in _STORAGE_CATS if c}
def _dir_stats(path: Path) -> tuple[int, int]:
"""(total_bytes, file_count) of a dir tree. Missing dir → (0, 0)."""
total, n = 0, 0
try:
for root, _dirs, files in os.walk(path):
for f in files:
try:
total += os.path.getsize(os.path.join(root, f))
n += 1
except OSError:
pass
except Exception:
pass
return total, n
def _human(b: float) -> str:
f = float(b)
for u in ("B", "KB", "MB", "GB", "TB"):
if f < 1024 or u == "TB":
return f"{f:.0f} {u}" if u == "B" else f"{f:.1f} {u}"
f /= 1024
return f"{f:.1f} TB"
@router.get("/storage")
async def storage_usage():
"""Per-category data/log sizes + disk free, for the Storage panel."""
def _do():
cats = []
for key, label, path, cleanable in _STORAGE_CATS:
size, files = _dir_stats(Path(path))
cats.append({
"key": key, "label": label, "path": str(path),
"size_bytes": size, "size_human": _human(size),
"files": files, "cleanable": cleanable,
})
data_b, _ = _dir_stats(DATA_DIR)
logs_b, _ = _dir_stats(LOGS_DIR)
try:
du = shutil.disk_usage(str(BASE_DIR))
disk = {
"free_human": _human(du.free), "total_human": _human(du.total),
"used_pct": round(100.0 * (du.total - du.free) / du.total, 1),
}
except Exception:
disk = {}
return {
"categories": cats,
"data_bytes": data_b, "data_human": _human(data_b),
"logs_human": _human(logs_b),
"total_human": _human(data_b + logs_b),
"disk": disk,
}
return await asyncio.to_thread(_do)
class _CleanReq(BaseModel):
target: str # recordings | records | logs | all
def _clean_recordings() -> tuple[int, int]:
d = DATA_DIR / "recordings"
freed, n = 0, 0
for f in list(d.glob("*.wav")) + [d / "index.json"]:
if f.is_file():
try:
freed += f.stat().st_size
f.unlink()
n += 1
except OSError:
pass
return n, freed
def _clean_records() -> tuple[int, int]:
d = AUDIO_RECORDINGS_DIR
freed, n = 0, 0
for f in list(d.glob("*.wav")) + [d / "records.json"]:
if f.is_file():
try:
freed += f.stat().st_size
f.unlink()
n += 1
except OSError:
pass
return n, freed
def _clean_logs() -> tuple[int, int]:
# Truncate (not delete) — active loggers hold append-mode handles, so
# truncating to 0 clears content cleanly without losing the fd.
freed, n = 0, 0
for f in Path(LOGS_DIR).glob("*.log"):
try:
freed += f.stat().st_size
open(f, "w").close()
n += 1
except OSError:
pass
return n, freed
@router.post("/storage/clean")
async def storage_clean(req: _CleanReq):
"""Clean a disposable category (recordings | records | logs) or 'all'.
Recordings/records are deleted; logs are truncated. Assets (faces, motions,
zones) are never touched here."""
t = (req.target or "").strip().lower()
if t != "all" and t not in _CLEANABLE:
raise HTTPException(400, f"target must be 'all' or one of {sorted(_CLEANABLE)}")
def _do():
targets = ["recordings", "records", "logs"] if t == "all" else [t]
fns = {"recordings": _clean_recordings, "records": _clean_records,
"logs": _clean_logs}
result, total = {}, 0
for tg in targets:
n, freed = fns[tg]()
result[tg] = {"items": n, "freed_bytes": freed, "freed_human": _human(freed)}
total += freed
log.info("storage clean %s → freed %s", targets, _human(total))
return {"ok": True, "cleaned": targets,
"total_freed_bytes": total, "total_freed_human": _human(total),
"result": result}
return await asyncio.to_thread(_do)

View File

@ -0,0 +1,81 @@
"""REST endpoints backing the 3D motor-temperature dashboard (N1).
Serves the motor name/mesh mapping + thresholds, and a one-shot temperature
snapshot (the front-end's initial fetch fallback). The live stream is over
/ws/motor-temps (dashboard/websockets/motor_temps.py). The 3D view itself is
the static page at /static/temp3d/index.html.
"""
from __future__ import annotations
import time
from fastapi import APIRouter
from Project.Sanad.dashboard.temp_motor_map import (
MOTOR_NAMES,
MOTOR_TO_MESH,
TEMP_HOT_THRESHOLD,
TEMP_MAX,
TEMP_MIN,
TEMP_WARM_THRESHOLD,
build_payload,
)
router = APIRouter()
def _get_arm():
"""Lazy import — avoids a circular import on dashboard load."""
try:
from Project.Sanad.main import arm # type: ignore
return arm
except Exception:
return None
@router.get("/mapping")
async def motor_mapping():
"""Motor id → name / mesh map + the temperature gradient thresholds."""
return {
"motor_names": MOTOR_NAMES,
"motor_to_mesh": MOTOR_TO_MESH,
"thresholds": {
"min": TEMP_MIN,
"max": TEMP_MAX,
"warm": TEMP_WARM_THRESHOLD,
"hot": TEMP_HOT_THRESHOLD,
},
}
@router.get("/motors")
async def motors_snapshot():
"""One-shot motor temperature + position snapshot (Marcus payload shape)."""
arm = _get_arm()
temps: list = []
positions: list = []
if arm is not None:
try:
temps = arm.get_motor_temps()
except Exception:
temps = []
try:
positions = arm.get_current_q()
except Exception:
positions = []
return build_payload(temps, positions, time.time())
@router.get("/battery")
async def battery_status():
"""Live G1 battery (BMS) snapshot: state-of-charge %, voltage, current,
charge/discharge status, pack temperature, cycles. `available=False` until
the BMS topic (rt/lf/bmsstate) delivers its first message."""
arm = _get_arm()
if arm is None or not hasattr(arm, "get_battery"):
return {"available": False}
try:
return arm.get_battery()
except Exception:
return {"available": False}

View File

@ -193,7 +193,7 @@ async def update_api_key(payload: ApiKeyPayload):
raise HTTPException(500, f"Could not save config: {exc}")
# Hot-swap the in-memory module globals.
# Both Project.Sanad.config AND Project.Sanad.voice.gemini_client
# Both Project.Sanad.config AND Project.Sanad.gemini.client
# have their OWN reference to GEMINI_API_KEY (the latter was created
# at `from Project.Sanad.config import GEMINI_API_KEY` at import time).
# Python's `from X import Y` binds a local name — updating config.Y
@ -205,10 +205,10 @@ async def update_api_key(payload: ApiKeyPayload):
log.exception("could not patch config.GEMINI_API_KEY")
try:
import Project.Sanad.voice.gemini_client as _gc
import Project.Sanad.gemini.client as _gc
_gc.GEMINI_API_KEY = key
except Exception:
log.exception("could not patch gemini_client.GEMINI_API_KEY")
log.exception("could not patch gemini.client.GEMINI_API_KEY")
# Disconnect any live session so reconnect uses the new key.
from Project.Sanad.main import voice_client

597
dashboard/routes/zones.py Normal file
View File

@ -0,0 +1,597 @@
"""Zones tab — zone → place → linked-faces management + "go here" destination.
Hierarchy (replaces the old flat places):
Zone (name + description)
Place (name + description + optional reference photos + linked face ids)
Routes live under /api/zones. Toggle + CRUD changes write
data/.recognition_state.json (the SAME file faces use); the Gemini child polls
it at 1 Hz and re-primes / announces mid-session. The "go here" endpoints set a
navigation target the robot will head to once N2 locomotion is wired for now
they just record the target and feed Gemini the place's reference.
"""
from __future__ import annotations
import asyncio
import io
from typing import Optional
from fastapi import APIRouter, File, HTTPException, Query, UploadFile
from fastapi.responses import FileResponse, StreamingResponse
from pydantic import BaseModel
from Project.Sanad.config import BASE_DIR
from Project.Sanad.core.logger import get_logger
from Project.Sanad.dashboard.routes._safe_io import check_upload_size
from Project.Sanad.vision import recognition_state
log = get_logger("zones_routes")
router = APIRouter()
STATE_PATH = BASE_DIR / "data" / ".recognition_state.json"
# ── lazy subsystem accessors ────────────────────────────────
def _get_camera():
try:
from Project.Sanad.main import camera # type: ignore
return camera
except Exception:
return None
def _get_zone_gallery():
try:
from Project.Sanad.main import zone_gallery # type: ignore
return zone_gallery
except Exception:
return None
def _get_face_gallery():
try:
from Project.Sanad.main import gallery # type: ignore
return gallery
except Exception:
return None
def _require_zones():
g = _get_zone_gallery()
if g is None:
raise HTTPException(503, "Zone gallery subsystem unavailable.")
return g
def _bump_zones_version() -> int:
cur = recognition_state.read(STATE_PATH)
v = cur.zones_version + 1
recognition_state.mutate(STATE_PATH, zones_version=v)
return v
def _validate_image(content: bytes, filename: str | None = None) -> None:
check_upload_size(content)
if len(content) < 16:
raise HTTPException(400, "Image too small / empty.")
if not (content[:3] == b"\xff\xd8\xff" or content[:8] == b"\x89PNG\r\n\x1a\n"):
raise HTTPException(400, f"Only JPEG/PNG accepted (got {filename or 'unknown'}).")
def _safe_photo_name(name: str) -> None:
if "/" in name or ".." in name or "\x00" in name:
raise HTTPException(400, "Invalid photo name.")
def _resolve_faces(face_ids: list[int]) -> list[dict]:
"""Turn linked face ids into [{id, name}] using the face gallery."""
g = _get_face_gallery()
out = []
for fid in face_ids:
name = None
if g is not None:
try:
e = g.get(fid)
name = e.name if e else None
except Exception:
name = None
out.append({"id": fid, "name": name})
return out
def _place_to_dict(p) -> dict:
d = p.to_dict()
d["faces"] = _resolve_faces(p.face_ids)
return d
def _zone_to_dict(z) -> dict:
return {
"id": z.id, "name": z.name, "description": z.description,
"linked_map": getattr(z, "linked_map", None),
"added_at": z.added_at,
"places": [_place_to_dict(p) for p in z.places],
}
async def _maybe_drive_to_place(zone, place) -> Optional[dict]:
"""If the place links a nav2 place AND its zone's map is the one currently
localized, actually DRIVE there (arbiter-gated + arm arrival monitor).
Returns the drive outcome, or None when the place isn't drivable (no link).
Best-effort: never raises into the caller."""
nav_place = getattr(place, "nav_place", None)
linked_map = getattr(zone, "linked_map", None)
if not nav_place or not linked_map:
return None
try:
from Project.Sanad.dashboard.routes import navigation as navmod
from Project.Sanad.dashboard.routes import _arbiter
except Exception:
return {"ok": False, "reason": "nav_unavailable"}
client = getattr(navmod, "_CLIENT", None)
if client is None:
return {"ok": False, "reason": "nav_unavailable"}
try:
st = await asyncio.to_thread(client.status)
body = st.as_dict() if hasattr(st, "as_dict") else dict(st)
except Exception as exc: # noqa: BLE001
return {"ok": False, "reason": "status_error", "detail": str(exc)[:120]}
if not body.get("bringup_alive"):
return {"ok": False, "reason": "no_map"}
# The robot can only drive in the currently-localized map. Require the
# zone's linked map to match (compare on the sanitized .db stem).
active = (body.get("active_map") or "").strip().lower()
want = (linked_map or "").strip().lower()
if want.endswith(".db"):
want = want[:-3]
if active and want and active != want:
return {"ok": False, "reason": "wrong_map",
"active": body.get("active_map"), "want": linked_map}
if not _arbiter.acquire_nav():
return {"ok": False, "reason": "manual_armed"}
drive = await asyncio.to_thread(client.goto, nav_place)
if isinstance(drive, dict) and not drive.get("ok", True):
_arbiter.release_nav()
return {"ok": False, "reason": "dispatch_failed", "detail": drive}
try:
from Project.Sanad.navigation.goal_monitor import arm_goal
arm_goal(nav_place)
except Exception:
pass
return {"ok": True, "resolved": nav_place}
def _nav_target_dict(st, gallery) -> Optional[dict]:
zid, pid = st.nav_target_zone_id, st.nav_target_place_id
if not zid or not pid:
return None
zone_name = place_name = None
if gallery is not None:
try:
z = gallery.get_zone(zid)
zone_name = z.name if z else None
p = gallery.get_place(zid, pid)
place_name = p.name if p else None
except Exception:
pass
return {"zone_id": zid, "place_id": pid,
"zone_name": zone_name, "place_name": place_name}
# ── state + toggle ──────────────────────────────────────────
@router.get("/state")
async def get_state():
st = recognition_state.read(STATE_PATH)
g = _get_zone_gallery()
zones_count = places_count = 0
if g is not None:
try:
zones = g.list_zones()
zones_count = len(zones)
places_count = sum(len(z.places) for z in zones)
except Exception:
pass
return {
"zone_rec_enabled": st.zone_rec_enabled,
"zones_version": st.zones_version,
"zones_count": zones_count,
"places_count": places_count,
"nav_target": _nav_target_dict(st, g),
}
@router.post("/zone-rec")
async def set_zone_rec(on: bool = Query(...)):
"""Enable / disable the robot's knowledge of zones & places (hot)."""
st = recognition_state.mutate(STATE_PATH, zone_rec_enabled=bool(on))
log.info("zone recognition %s", "ON" if on else "OFF")
return {"ok": True, "zone_rec_enabled": st.zone_rec_enabled}
@router.post("/sync")
async def sync_zones():
v = _bump_zones_version()
log.info("zones sync requested → v.%d", v)
return {"ok": True, "zones_version": v}
# ── zones CRUD ──────────────────────────────────────────────
class NamePayload(BaseModel):
name: Optional[str] = None
class DescribePayload(BaseModel):
description: Optional[str] = None
class FacesPayload(BaseModel):
face_ids: list[int] = []
class LinkMapPayload(BaseModel):
# nav2 map .db basename (e.g. "office.db"); None/"" unlinks.
map: Optional[str] = None
class NavPlacePayload(BaseModel):
# nav2 place name in the zone's linked map; None/"" unlinks.
nav_place: Optional[str] = None
@router.get("")
async def list_zones():
g = _require_zones()
zones = g.list_zones()
return {"zones": [_zone_to_dict(z) for z in zones], "total": len(zones)}
@router.post("/create")
async def create_zone(name: Optional[str] = Query(default=None),
description: Optional[str] = Query(default=None)):
g = _require_zones()
if not (name or "").strip() and not (description or "").strip():
raise HTTPException(400, "A zone needs at least a name or a description.")
z = g.create_zone(name=name, description=description)
_bump_zones_version()
return {"ok": True, "zone": _zone_to_dict(z)}
@router.post("/{zone_id}/rename")
async def rename_zone(zone_id: int, payload: NamePayload):
g = _require_zones()
try:
g.rename_zone(zone_id, payload.name)
except FileNotFoundError as exc:
raise HTTPException(404, str(exc))
_bump_zones_version()
return {"ok": True, "zone": _zone_to_dict(g.get_zone(zone_id))}
@router.post("/{zone_id}/describe")
async def describe_zone(zone_id: int, payload: DescribePayload):
g = _require_zones()
try:
g.describe_zone(zone_id, payload.description)
except FileNotFoundError as exc:
raise HTTPException(404, str(exc))
_bump_zones_version()
return {"ok": True, "zone": _zone_to_dict(g.get_zone(zone_id))}
@router.delete("/{zone_id}")
async def delete_zone(zone_id: int):
g = _require_zones()
try:
g.delete_zone(zone_id)
except FileNotFoundError as exc:
raise HTTPException(404, str(exc))
# If the active destination was inside this zone, clear it.
st = recognition_state.read(STATE_PATH)
if st.nav_target_zone_id == zone_id:
recognition_state.mutate(STATE_PATH, nav_target_zone_id=0, nav_target_place_id=0)
_bump_zones_version()
return {"ok": True, "deleted": zone_id}
# ── places CRUD (within a zone) ─────────────────────────────
@router.post("/{zone_id}/places/create")
async def create_place(
zone_id: int,
name: Optional[str] = Query(default=None),
description: Optional[str] = Query(default=None),
face_ids: list[int] = Query(default=[]),
nav_place: Optional[str] = Query(default=None),
files: Optional[list[UploadFile]] = File(default=None),
):
g = _require_zones()
if g.get_zone(zone_id) is None:
raise HTTPException(404, f"zone_{zone_id} not found")
if not (name or "").strip() and not (description or "").strip():
raise HTTPException(400, "A place needs at least a name or a description.")
image_bytes: list[bytes] = []
for f in (files or []):
content = await f.read()
if not content:
continue
_validate_image(content, f.filename)
image_bytes.append(content)
p = g.create_place(zone_id, name=name, description=description,
face_ids=face_ids, image_bytes_list=image_bytes or None,
nav_place=nav_place)
_bump_zones_version()
return {"ok": True, "place": _place_to_dict(p)}
@router.post("/{zone_id}/link_map")
async def link_zone_map(zone_id: int, payload: LinkMapPayload):
"""Bind (or unbind) the zone to a nav2 map .db. Required before its places
can link to that map's nav places / before Gemini Nav can drive in it."""
g = _require_zones()
try:
g.set_zone_map(zone_id, payload.map)
except FileNotFoundError as exc:
raise HTTPException(404, str(exc))
_bump_zones_version()
return {"ok": True, "zone": _zone_to_dict(g.get_zone(zone_id))}
@router.post("/{zone_id}/places/{place_id}/nav_link")
async def link_place_nav(zone_id: int, place_id: int, payload: NavPlacePayload):
"""Link (or unlink) a place to a nav2 place name in the zone's map — this is
what makes the place drivable from voice / 'Go here'."""
g = _require_zones()
try:
g.set_place_nav(zone_id, place_id, payload.nav_place)
except FileNotFoundError as exc:
raise HTTPException(404, str(exc))
_bump_zones_version()
return {"ok": True, "place": _place_to_dict(g.get_place(zone_id, place_id))}
@router.post("/{zone_id}/places/{place_id}/rename")
async def rename_place(zone_id: int, place_id: int, payload: NamePayload):
g = _require_zones()
try:
g.rename_place(zone_id, place_id, payload.name)
except FileNotFoundError as exc:
raise HTTPException(404, str(exc))
_bump_zones_version()
return {"ok": True, "place": _place_to_dict(g.get_place(zone_id, place_id))}
@router.post("/{zone_id}/places/{place_id}/describe")
async def describe_place(zone_id: int, place_id: int, payload: DescribePayload):
g = _require_zones()
try:
g.describe_place(zone_id, place_id, payload.description)
except FileNotFoundError as exc:
raise HTTPException(404, str(exc))
_bump_zones_version()
return {"ok": True, "place": _place_to_dict(g.get_place(zone_id, place_id))}
@router.post("/{zone_id}/places/{place_id}/faces")
async def set_place_faces(zone_id: int, place_id: int, payload: FacesPayload):
"""Replace the set of saved faces linked to this place."""
g = _require_zones()
try:
g.set_place_faces(zone_id, place_id, payload.face_ids)
except FileNotFoundError as exc:
raise HTTPException(404, str(exc))
_bump_zones_version()
return {"ok": True, "place": _place_to_dict(g.get_place(zone_id, place_id))}
@router.post("/{zone_id}/places/{place_id}/capture")
async def capture_to_place(zone_id: int, place_id: int):
g = _require_zones()
cam = _get_camera()
if cam is None or not cam.is_running():
raise HTTPException(409, "Camera is not running. Toggle Vision ON first.")
jpeg = cam.get_fresh_frame(max_age_s=0.5, timeout_s=1.5)
if not jpeg:
raise HTTPException(409, "Camera has no frame yet.")
try:
fname = g.add_photo(zone_id, place_id, jpeg)
except FileNotFoundError as exc:
raise HTTPException(404, str(exc))
_bump_zones_version()
return {"ok": True, "added": fname, "place": _place_to_dict(g.get_place(zone_id, place_id))}
@router.post("/{zone_id}/places/{place_id}/upload")
async def upload_to_place(zone_id: int, place_id: int,
files: list[UploadFile] = File(...)):
g = _require_zones()
if g.get_place(zone_id, place_id) is None:
raise HTTPException(404, f"zone_{zone_id}/place_{place_id} not found")
added: list[str] = []
for f in files:
content = await f.read()
_validate_image(content, f.filename)
try:
added.append(g.add_photo(zone_id, place_id, content))
except FileNotFoundError as exc:
raise HTTPException(404, str(exc))
_bump_zones_version()
return {"ok": True, "added": added, "place": _place_to_dict(g.get_place(zone_id, place_id))}
@router.delete("/{zone_id}/places/{place_id}")
async def delete_place(zone_id: int, place_id: int):
g = _require_zones()
try:
g.delete_place(zone_id, place_id)
except FileNotFoundError as exc:
raise HTTPException(404, str(exc))
st = recognition_state.read(STATE_PATH)
if st.nav_target_zone_id == zone_id and st.nav_target_place_id == place_id:
recognition_state.mutate(STATE_PATH, nav_target_zone_id=0, nav_target_place_id=0)
_bump_zones_version()
return {"ok": True, "deleted": place_id}
@router.delete("/{zone_id}/places/{place_id}/photo/{photo_name}")
async def delete_place_photo(zone_id: int, place_id: int, photo_name: str):
g = _require_zones()
_safe_photo_name(photo_name)
try:
g.delete_photo(zone_id, place_id, photo_name)
except FileNotFoundError as exc:
raise HTTPException(404, str(exc))
_bump_zones_version()
return {"ok": True, "deleted": photo_name}
@router.get("/{zone_id}/places/{place_id}/photo/{photo_name}")
async def get_place_photo(zone_id: int, place_id: int, photo_name: str,
download: int = Query(default=0)):
g = _require_zones()
_safe_photo_name(photo_name)
path = g.get_photo(zone_id, place_id, photo_name)
if path is None:
raise HTTPException(404, "Photo not found.")
media = "image/png" if path.suffix.lower() == ".png" else "image/jpeg"
headers = {}
if download:
headers["Content-Disposition"] = (
f'attachment; filename="zone_{zone_id}_place_{place_id}_{photo_name}"')
return FileResponse(path, media_type=media, headers=headers)
@router.get("/{zone_id}/places/{place_id}/download.zip")
async def download_place_zip(zone_id: int, place_id: int):
g = _require_zones()
try:
data = g.zip_place(zone_id, place_id)
except FileNotFoundError as exc:
raise HTTPException(404, str(exc))
return StreamingResponse(
io.BytesIO(data), media_type="application/zip",
headers={
"Content-Disposition": f'attachment; filename="zone_{zone_id}_place_{place_id}.zip"',
"Content-Length": str(len(data)),
},
)
# ── "go here" navigation target ─────────────────────────────
@router.post("/{zone_id}/places/{place_id}/go")
async def go_to_place(zone_id: int, place_id: int):
"""Set this place as the active destination AND, if the place links a nav2
place in this zone's (currently-localized) map, actually drive there.
Two effects: (1) records nav_target so the Gemini child primes on the
reference photo + announces the destination; (2) if drivable, dispatches a
Nav2 goal (arbiter-gated, with arrival monitoring). A place with no nav link
is announce-only, as before."""
g = _require_zones()
p = g.get_place(zone_id, place_id)
if p is None:
raise HTTPException(404, f"zone_{zone_id}/place_{place_id} not found")
recognition_state.mutate(STATE_PATH,
nav_target_zone_id=zone_id,
nav_target_place_id=place_id)
log.info("nav target set → zone_%d/place_%d (%s)", zone_id, place_id,
p.name or "(unnamed)")
zone = g.get_zone(zone_id)
drive = await _maybe_drive_to_place(zone, p)
return {"ok": True,
"nav_target": {"zone_id": zone_id, "place_id": place_id,
"place_name": p.name},
"drive": drive}
@router.post("/nav/clear")
async def clear_nav_target():
recognition_state.mutate(STATE_PATH, nav_target_zone_id=0, nav_target_place_id=0)
log.info("nav target cleared")
return {"ok": True, "nav_target": None}
def _resolve_map_path(client, linked_map: str) -> Optional[str]:
"""Find the .db path for a zone's linked map name via the nav client."""
want = (linked_map or "").strip().lower()
want_stem = want[:-3] if want.endswith(".db") else want
try:
maps = client.list_maps() or []
except Exception:
return None
for m in maps:
nm = (m.get("name") or "").strip().lower()
if nm == want or (nm[:-3] if nm.endswith(".db") else nm) == want_stem:
return m.get("path")
return None
@router.post("/{zone_id}/gemini_nav/start")
async def gemini_nav_start(zone_id: int):
"""Enter 'Gemini Nav' for a zone: localize the zone's map, turn on camera +
face + zone recognition + movement, ensure the Gemini session is live, and
greet the user so they can converse to navigate.
The robot only ever runs ONE map; this loads the zone's map in localize-only
mode (so it cannot fresh-map while driving), exactly as the user requires.
"""
g = _require_zones()
zone = g.get_zone(zone_id)
if zone is None:
raise HTTPException(404, f"zone_{zone_id} not found")
linked_map = getattr(zone, "linked_map", None)
if not linked_map:
raise HTTPException(400, "This zone has no linked nav2 map — link one first.")
# 1) Localize the zone's map (single bringup, mode 3 — no fresh mapping).
loaded: dict = {"ok": False, "reason": "nav_unavailable"}
try:
from Project.Sanad.dashboard.routes import navigation as navmod
client = getattr(navmod, "_CLIENT", None)
if client is not None:
db_path = await asyncio.to_thread(_resolve_map_path, client, linked_map)
if db_path:
loaded = await asyncio.to_thread(client.load_map, db_path)
else:
loaded = {"ok": False, "reason": "map_not_found", "map": linked_map}
except Exception as exc: # noqa: BLE001
loaded = {"ok": False, "reason": "load_error", "detail": str(exc)[:160]}
# 2) Camera + face + zone recognition + movement ON for the session.
recognition_state.mutate(STATE_PATH,
vision_enabled=True, face_rec_enabled=True,
zone_rec_enabled=True, movement_enabled=True)
_bump_zones_version()
# 3) Ensure the Gemini session is live, then greet (zone + drivable places).
session_started = False
try:
from Project.Sanad.main import live_sub
if live_sub is not None:
if hasattr(live_sub, "is_running") and not live_sub.is_running():
await asyncio.to_thread(live_sub.start)
session_started = True
drivable = [p.name or p.nav_place for p in zone.places
if getattr(p, "nav_place", None)]
zname = zone.name or f"zone {zone_id}"
if drivable:
placelist = ", ".join(str(x) for x in drivable)
greet = (f"You are now in the '{zname}' zone. You can drive the "
f"user to: {placelist}. Greet the user warmly in your "
f"normal Khaleeji style and ask where they would like to go.")
else:
greet = (f"You are now in the '{zname}' zone, but no drivable "
f"places are linked to its map yet. Greet the user and "
f"say places still need to be linked before you can drive.")
if hasattr(live_sub, "send_state"):
live_sub.send_state("nav_zone", greet)
except Exception as exc: # noqa: BLE001
log.warning("gemini_nav greet failed: %s", exc)
return {"ok": True, "zone_id": zone_id, "zone": _zone_to_dict(zone),
"loaded": loaded, "session_started": session_started}

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

File diff suppressed because it is too large Load Diff

Some files were not shown because too many files have changed in this diff Show More