Compare commits
10 Commits
71c45027f5
...
699e89f336
| Author | SHA1 | Date | |
|---|---|---|---|
| 699e89f336 | |||
| 4210c4cc61 | |||
| ca0de44401 | |||
| 811a391932 | |||
| edddb7e0c3 | |||
| 54b1e745ca | |||
| f7da15da1b | |||
| 1693776f3f | |||
| cf5e916120 | |||
| 94e4a9c4cb |
7
.claude/settings.json
Normal file
7
.claude/settings.json
Normal file
@ -0,0 +1,7 @@
|
||||
{
|
||||
"permissions": {
|
||||
"allow": [
|
||||
"Bash(node -e ' *)"
|
||||
]
|
||||
}
|
||||
}
|
||||
136
FEATURES.md
Normal file
136
FEATURES.md
Normal file
@ -0,0 +1,136 @@
|
||||
# SanadV3 — Feature Catalog
|
||||
|
||||
Sanad is a bilingual (Arabic/English) humanoid receptionist/assistant on a
|
||||
Unitree G1 (Jetson Orin NX, ROS 2 Foxy, Livox MID-360). This catalogs
|
||||
**what's built today** (Part A) and **what's on the roadmap** (Part B).
|
||||
|
||||
---
|
||||
|
||||
# Part A — Current features (built & running)
|
||||
|
||||
Verified from the live subsystem registry (19 subsystems), dashboard tabs
|
||||
(12), and API routers (22).
|
||||
|
||||
## 1. Voice & Conversation
|
||||
- **Gemini live voice** — real-time bilingual AR/EN spoken conversation (native-audio model)
|
||||
- **Offline brain** — local pipeline via `ollama` (`SANAD_VOICE_BRAIN=local`), no cloud
|
||||
- **Wake phrases** — configurable wake-word manager
|
||||
- **Typed replay** — type text, robot speaks it (with speaker-monitor capture)
|
||||
- **Local TTS** — on-device text-to-speech engine
|
||||
- **Prompt management** — edit the system prompt from the dashboard
|
||||
- **Lip-sync** — mask mouth driven by TTS `MOUTH` markers
|
||||
- **Barge-in** — interrupt speech (volume-scaled threshold)
|
||||
|
||||
## 2. Motion & Arm
|
||||
- **35 arm actions** — 16 SDK built-ins + 19 custom JSONL motions
|
||||
- **Macro record / playback** — capture and replay motion sequences
|
||||
- **Teaching mode** — kinesthetic teach-and-repeat
|
||||
- **Skills** — composed higher-level behaviors (skills.json)
|
||||
- **Movement dispatch** — voice → motion (53 fixed + 10 parametric phrases, cooldown-gated)
|
||||
- **Arm motion-block** — auto-inhibits arm moves while locomotion is active (safety interlock)
|
||||
|
||||
## 3. Locomotion
|
||||
- **LocoClient + MotionSwitcher** — walk / pose control via Unitree SDK (eth0)
|
||||
- **E-STOP** — dashboard kill button
|
||||
- **Single Ctrl+C teardown** — one signal cleanly stops every subsystem (~2s)
|
||||
|
||||
## 4. LED Face Mask
|
||||
- **Animated expressions** — neutral, smile, blink, look L/R, talk1–3, surprised, sad
|
||||
- **Gestural-speaking events** — face reacts while speaking
|
||||
- **Lip-sync** — mouth animates to speech
|
||||
|
||||
## 5. Vision & Recognition
|
||||
- **Face recognition** — identify people via camera
|
||||
- **Face gallery** — enroll/manage known faces
|
||||
- **Zone gallery / zones** — visual zone recognition
|
||||
- **Camera feed** — attached to the live voice subprocess (vision-in-the-loop)
|
||||
|
||||
## 6. Navigation (web_nav3 integration)
|
||||
- **Live Map tab** — full embedded web_nav3 dashboard (set-pose, goals, bringup)
|
||||
- **Navigation tab** — native canvas viewer (saved/live map, places, missions)
|
||||
- **map_relay** — re-publishes the latched `/map` @1Hz so the map renders even when stationary
|
||||
- **Saved maps** — load & view a pre-built `.db` (localize mode)
|
||||
- **Places** — save named poses, one-click "Go"
|
||||
- **Missions** — multi-waypoint routes (defined in web_nav3)
|
||||
- **Cancel goal** — stop an active goal without tearing down bringup
|
||||
- **SLAM** — RTABMap LiDAR-ICP, drift-corrected mapping/localization
|
||||
|
||||
## 7. Audio
|
||||
- **Device manager** — sink/source selection, live refresh
|
||||
- **Audio profiles** — builtin / anker / hollyland_builtin (auto-switch on plug/unplug)
|
||||
|
||||
## 8. Operations, System & Diagnostics
|
||||
- **System control** — start/stop subsystems, status
|
||||
- **Temperature monitor** — motor temps (live websocket stream)
|
||||
- **Controller** — gamepad/teleop input
|
||||
- **Web terminal** — shell in the browser (websocket)
|
||||
- **Logs** — live log stream
|
||||
- **Recordings & replay** — record/playback sessions
|
||||
- **Scripts** — run saved scripts
|
||||
|
||||
## Dashboard infrastructure
|
||||
- 12 tabs, **fault-isolated routers** (one broken module never breaks the dashboard)
|
||||
- WebSocket streams: log_stream, motor_temps, terminal
|
||||
- No-store HTML (no stale-cache 404s after deploy)
|
||||
- Lazy subsystem imports (missing dep → that subsystem unavailable, rest runs)
|
||||
|
||||
---
|
||||
|
||||
# Part B — Roadmap (to add)
|
||||
|
||||
Tiers = priority. 🏗️ = load-bearing · ⚠️ = Foxy constraint.
|
||||
|
||||
## Tier 1 — Autonomous behaviors (the product)
|
||||
1. **Voice-driven navigation** — "Sanad, go to the lobby" → nav goal
|
||||
2. **Greeter mission** — recognized face → navigate → greet → express
|
||||
3. **Named-person greeting** — identity → personalized line
|
||||
4. **Patrol / guided tours** — ordered places, speech at each stop
|
||||
5. **Return-to-base / dock-on-idle** — auto-home on idle/low battery
|
||||
|
||||
## Tier 2 — Navigation & map (harden + edit)
|
||||
6. 🏗️ **Map republish relay** — ✅ DONE (map_relay)
|
||||
7. **Click-to-goal on Nav tab canvas**
|
||||
8. **Live nav telemetry** — distance/ETA/waypoint, "arrived" toast
|
||||
9. **Battery + nav-state status bar**
|
||||
10. **Geofence zones on the map**
|
||||
11. **Cancel-goal button** — ✅ DONE
|
||||
### Map editing & annotation (all build on #6)
|
||||
12. **Erase tool** — paint cells free; wipe ghost obstacles + the SLAM "spokes"
|
||||
13. **Obstacle paint ("black points" / virtual walls)** — ⚠️ Foxy-safe KeepoutFilter substitute
|
||||
14. **Shape tools + brush size** — line/rectangle/polygon
|
||||
15. **Non-destructive overlay + undo/redo**
|
||||
16. **Persist & auto-reload edits per map**
|
||||
17. **Crop / trim map bounds**
|
||||
|
||||
## Tier 3 — Voice & interaction
|
||||
18. **Barge-in from dashboard**
|
||||
19. **Quick-phrase soundboard**
|
||||
20. **Conversation memory / visitor log**
|
||||
21. **Per-speaker AR/EN auto-detect**
|
||||
22. **Scheduled announcements**
|
||||
23. **Bake edited map → PGM/YAML** (static map_server deploy)
|
||||
|
||||
## Tier 4 — Face & presence
|
||||
24. **Gaze / head-track recognized face**
|
||||
25. **Emotion-from-context** (sentiment → expression)
|
||||
26. **Idle breathing / look-around**
|
||||
27. **Lip-sync to TTS amplitude** (enhance existing markers)
|
||||
|
||||
## Tier 5 — Operator, fleet & reliability
|
||||
28. 🏗️ **Global E-STOP button** — ✅ exists; surface consistently
|
||||
29. **Health watchdog** — auto-restart dead subsystem + alert
|
||||
30. **Per-subsystem enable/disable toggles**
|
||||
31. **Behavior recorder → replay** (nav+voice timelines)
|
||||
32. **Mission editor UI** (visual sequence builder)
|
||||
33. **Remote access / tunnel**
|
||||
34. **Reverse-proxy web_nav3 through :8001** — one origin, no iframe cross-port issues
|
||||
|
||||
## Tier 6 — Future / blocked
|
||||
35. **Speed / caution zones** — needs Galactic SpeedFilter or custom layer
|
||||
36. **Multi-robot fleet** (SanadV3 ↔ BotBrain) — needs LocoClient arbitration + coordinator
|
||||
|
||||
---
|
||||
|
||||
### Recommended next build order
|
||||
**#1 voice→nav** → **#2 greeter mission** (the product), then **#12/#13 map editing**
|
||||
(clean the spokes + virtual walls). #6 republish relay and #11 cancel are already done.
|
||||
12
G1_Controller/__init__.py
Normal file
12
G1_Controller/__init__.py
Normal file
@ -0,0 +1,12 @@
|
||||
"""G1_Controller — manual dashboard locomotion control (N2 Phase 1).
|
||||
|
||||
`LocoController` wraps the Unitree `LocoClient` + `MotionSwitcherClient` for
|
||||
operator-driven walking, postures and a discrete step pad. It reuses the arm
|
||||
controller's single process-wide DDS init (one `ChannelFactoryInitialize`) and
|
||||
is gated behind an in-memory "Enable movement" arm flag that defaults OFF every
|
||||
boot. See dashboard/routes/controller.py for the REST surface.
|
||||
"""
|
||||
|
||||
from Project.Sanad.G1_Controller.loco_controller import LocoController
|
||||
|
||||
__all__ = ["LocoController"]
|
||||
593
G1_Controller/loco_controller.py
Normal file
593
G1_Controller/loco_controller.py
Normal file
@ -0,0 +1,593 @@
|
||||
"""LocoController — manual G1 locomotion via the Unitree LocoClient (N2 Phase 1).
|
||||
|
||||
Ported from the proven scripts in G1_Lootah/Controller (g1_mode_controller.py,
|
||||
keyboard_controller.py, hanger_boot_sequence.py). Design notes:
|
||||
|
||||
* **One DDS init per process.** The arm controller owns the single
|
||||
`ChannelFactoryInitialize(0, nic)` (motion/arm_controller.py). This class
|
||||
NEVER initialises DDS — it lazily builds its `LocoClient` /
|
||||
`MotionSwitcherClient` only after `arm._initialized` is True.
|
||||
* **Default DISARMED.** `_armed` starts False every boot and gates every WRITE
|
||||
method. Reads (status / fsm / joints), E-STOP and disarm are ALWAYS allowed.
|
||||
* **StopMove watchdog.** Continuous `Move(..., True)` never self-terminates, so a
|
||||
daemon thread StopMoves if no `move()` refresh arrives within
|
||||
`watchdog_timeout_sec`. The frontend re-sends setpoints at ~10 Hz, so a tab
|
||||
close / network drop trips the watchdog within the timeout.
|
||||
* **Velocity caps.** Symmetric clamp on vx/vy/vyaw — Walk 0.6, Run 1.2.
|
||||
* **Allow-anytime-warn.** move/step never hard-block on FSM; if not walk-ready
|
||||
they still execute but return a `warning`.
|
||||
* **Sim fallback.** When `unitree_sdk2py` is absent (workstation), every write
|
||||
returns `{"simulated": True}` (never raises) so the whole UI is testable.
|
||||
|
||||
SDK facts confirmed from source — do not "fix" them:
|
||||
* `LocoClient.Move(vx, vy, vyaw, True)` — the continuous-mode kwarg is misspelled
|
||||
`continous_move` (one n); we pass it POSITIONALLY to avoid a TypeError.
|
||||
* `LocoClient` has NO StandUp()/Squat() — use SetFsmId(4)/SetFsmId(2).
|
||||
* FSM id / mode are read via the private RPC `bot._Call(7001/7002, "{}")`.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import threading
|
||||
import time
|
||||
from typing import Any, Optional
|
||||
|
||||
from Project.Sanad.core.config_loader import section as _cfg_section
|
||||
from Project.Sanad.core.logger import get_logger
|
||||
|
||||
log = get_logger("loco_controller")
|
||||
|
||||
# -- SDK import (optional) -----------------------------------------------------
|
||||
try:
|
||||
from unitree_sdk2py.g1.loco.g1_loco_client import LocoClient
|
||||
from unitree_sdk2py.comm.motion_switcher.motion_switcher_client import (
|
||||
MotionSwitcherClient,
|
||||
)
|
||||
_HAS_SDK = True
|
||||
except ImportError:
|
||||
LocoClient = None
|
||||
MotionSwitcherClient = None
|
||||
_HAS_SDK = False
|
||||
log.warning("Unitree SDK not available — LocoController in simulation mode")
|
||||
|
||||
# LocoClient general RPC api-ids for FSM read-back (stable across SDK builds).
|
||||
ROBOT_API_ID_LOCO_GET_FSM_ID = 7001
|
||||
ROBOT_API_ID_LOCO_GET_FSM_MODE = 7002
|
||||
|
||||
# G1 29-DoF joint names for indices 12-28 (0-11 legs, 12-14 waist, 15-21 left
|
||||
# arm, 22-28 right arm). Used by the Diagnostics joint read-out.
|
||||
JOINT_NAMES = {
|
||||
12: "WAIST_YAW", 13: "WAIST_ROLL", 14: "WAIST_PITCH",
|
||||
15: "L_SHOULDER_PITCH", 16: "L_SHOULDER_ROLL", 17: "L_SHOULDER_YAW",
|
||||
18: "L_ELBOW", 19: "L_WRIST_ROLL", 20: "L_WRIST_PITCH", 21: "L_WRIST_YAW",
|
||||
22: "R_SHOULDER_PITCH", 23: "R_SHOULDER_ROLL", 24: "R_SHOULDER_YAW",
|
||||
25: "R_ELBOW", 26: "R_WRIST_ROLL", 27: "R_WRIST_PITCH", 28: "R_WRIST_YAW",
|
||||
}
|
||||
|
||||
# Discrete step pad — (vx, vy, vyaw) sign per direction; magnitude is
|
||||
# step_speed_frac * cap_walk (a gentle single step).
|
||||
_STEP_DIRS = {
|
||||
"forward": (1.0, 0.0, 0.0),
|
||||
"backward": (-1.0, 0.0, 0.0),
|
||||
"slide_left": (0.0, 1.0, 0.0),
|
||||
"slide_right": (0.0, -1.0, 0.0),
|
||||
"rotate_left": (0.0, 0.0, 1.0),
|
||||
"rotate_right": (0.0, 0.0, -1.0),
|
||||
}
|
||||
|
||||
_POSTURES = (
|
||||
"zero_torque", "damp", "stand_up", "squat", "sit",
|
||||
"low_stand", "high_stand", "lie_to_stand",
|
||||
)
|
||||
|
||||
|
||||
class LocoController:
|
||||
"""Thread-safe manual locomotion control with a simulation fallback."""
|
||||
|
||||
def __init__(self, arm=None):
|
||||
self._arm = arm # shared ArmController (owns the ONE DDS init)
|
||||
self._bot = None # LocoClient (lazy)
|
||||
self._msc = None # MotionSwitcherClient (lazy)
|
||||
self._lc_ready = False
|
||||
self._lock = threading.RLock() # serialise all loco client WRITE calls
|
||||
self._armed = False # in-memory MANUAL gate — OFF every boot
|
||||
|
||||
self._cur_v = (0.0, 0.0, 0.0) # last commanded (vx, vy, vyaw)
|
||||
self._teleop_active = False
|
||||
self._last_msc_mode: Optional[str] = None
|
||||
|
||||
# watchdog
|
||||
self._last_move_ts = 0.0
|
||||
self._wd_thread: Optional[threading.Thread] = None
|
||||
self._wd_stop = threading.Event()
|
||||
self._wd_stop.set() # not running until armed
|
||||
# Monotonic stop-generation counter, bumped under _lock by
|
||||
# estop/stop/disarm. move()/step()/prep_mode() capture it at start and
|
||||
# bail the instant it changes — so E-STOP preempts an in-flight motion
|
||||
# immediately AND can never be silently "un-cancelled" by a concurrent
|
||||
# command (a lock-free Event clear() could; an int compare under the
|
||||
# lock cannot).
|
||||
self._stop_gen = 0
|
||||
# Serializes the discrete blocking operations (step/prep_mode) so two
|
||||
# can't overlap and interleave Move commands. Continuous teleop move()
|
||||
# is intentionally NOT guarded by this.
|
||||
self._discrete_busy = False
|
||||
|
||||
cfg = _cfg_section("motion", "loco_controller")
|
||||
self._cap_walk = float(cfg.get("cap_walk", 0.6))
|
||||
self._cap_run = float(cfg.get("cap_run", 1.2))
|
||||
self._lin_step = float(cfg.get("lin_step", 0.05))
|
||||
self._ang_step = float(cfg.get("ang_step", 0.2))
|
||||
self._wd_timeout = float(cfg.get("watchdog_timeout_sec", 0.5))
|
||||
self._block_window = float(cfg.get("arm_block_window_sec", 1.5))
|
||||
self._step_dur = float(cfg.get("step_duration_sec", 0.6))
|
||||
self._step_frac = float(cfg.get("step_speed_frac", 0.5))
|
||||
self._loco_timeout = float(cfg.get("loco_timeout_sec", 10.0))
|
||||
self._msc_timeout = float(cfg.get("msc_timeout_sec", 5.0))
|
||||
|
||||
# ── client lifecycle ─────────────────────────────────────────────────────
|
||||
|
||||
def _ensure_client(self) -> bool:
|
||||
"""Lazily build LocoClient + MotionSwitcherClient. Returns readiness.
|
||||
|
||||
Never initialises DDS — requires the shared arm to have already run the
|
||||
single ChannelFactoryInitialize.
|
||||
"""
|
||||
if not _HAS_SDK:
|
||||
return False
|
||||
if self._lc_ready:
|
||||
return True
|
||||
if self._arm is None or not getattr(self._arm, "_initialized", False):
|
||||
return False
|
||||
with self._lock:
|
||||
if self._lc_ready:
|
||||
return True
|
||||
try:
|
||||
bot = LocoClient()
|
||||
bot.SetTimeout(self._loco_timeout)
|
||||
bot.Init()
|
||||
msc = MotionSwitcherClient()
|
||||
msc.SetTimeout(self._msc_timeout)
|
||||
msc.Init()
|
||||
self._bot = bot
|
||||
self._msc = msc
|
||||
self._lc_ready = True
|
||||
log.info("LocoClient + MotionSwitcherClient ready")
|
||||
except Exception as exc:
|
||||
log.error("LocoClient init failed: %s", exc)
|
||||
self._lc_ready = False
|
||||
return self._lc_ready
|
||||
|
||||
def _safe_call(self, name: str, fn, *a, **kw):
|
||||
try:
|
||||
return True, fn(*a, **kw)
|
||||
except Exception as exc:
|
||||
log.error("%s failed: %s", name, exc)
|
||||
return False, None
|
||||
|
||||
def _rpc_get_int(self, api_id: int):
|
||||
bot = self._bot
|
||||
if bot is None:
|
||||
return None
|
||||
try:
|
||||
code, data = bot._Call(api_id, "{}")
|
||||
if code == 0 and data:
|
||||
return json.loads(data).get("data")
|
||||
except Exception:
|
||||
pass
|
||||
return None
|
||||
|
||||
@staticmethod
|
||||
def _clamp(v: float, cap: float) -> float:
|
||||
return max(-cap, min(cap, float(v)))
|
||||
|
||||
# ── FSM / readiness ──────────────────────────────────────────────────────
|
||||
|
||||
def fsm_id(self):
|
||||
return self._rpc_get_int(ROBOT_API_ID_LOCO_GET_FSM_ID)
|
||||
|
||||
def fsm_mode(self):
|
||||
return self._rpc_get_int(ROBOT_API_ID_LOCO_GET_FSM_MODE)
|
||||
|
||||
def _walk_ready_warning(self) -> Optional[str]:
|
||||
"""allow-anytime-warn: None when ready, else a human message."""
|
||||
if not self._lc_ready:
|
||||
return None
|
||||
fid = self.fsm_id()
|
||||
fmode = self.fsm_mode()
|
||||
if fid == 200 and fmode not in (None, 2):
|
||||
return None
|
||||
return (f"Robot not in walk-ready FSM (id={fid}, mode={fmode}). "
|
||||
f"Command sent anyway.")
|
||||
|
||||
# ── arm flag + watchdog ──────────────────────────────────────────────────
|
||||
|
||||
def is_armed(self) -> bool:
|
||||
return self._armed
|
||||
|
||||
def movement_active(self) -> bool:
|
||||
"""True when the robot may be walking: manual armed, teleop active, OR a
|
||||
move/step issued within the block window. Used as the arm's motion-block
|
||||
predicate so the arm never replays while the robot is (or just was)
|
||||
moving — regardless of whether the MANUAL gate or the GEMINI gate
|
||||
(Phase 3 voice dispatch, which calls move/step directly) triggered it."""
|
||||
if self._armed or self._teleop_active:
|
||||
return True
|
||||
return (time.monotonic() - self._last_move_ts) < self._block_window
|
||||
|
||||
def arm_movement(self) -> dict:
|
||||
"""Unlock manual control. Cancels any in-flight arm motion first so the
|
||||
arm and locomotion are never active simultaneously (movement wins)."""
|
||||
try:
|
||||
if self._arm is not None and getattr(self._arm, "is_busy", False):
|
||||
log.info("arming movement — cancelling in-flight arm motion")
|
||||
self._arm.cancel()
|
||||
except Exception:
|
||||
log.exception("arm.cancel() on arm_movement failed")
|
||||
with self._lock:
|
||||
self._armed = True
|
||||
self._start_watchdog()
|
||||
log.info("movement ARMED")
|
||||
return {"ok": True, "armed": True}
|
||||
|
||||
def disarm_movement(self) -> dict:
|
||||
with self._lock:
|
||||
self._stop_gen += 1 # break any in-flight step/prep/move
|
||||
self._armed = False
|
||||
self._teleop_active = False
|
||||
self._wd_stop.set()
|
||||
try:
|
||||
self._raw_stop()
|
||||
except Exception:
|
||||
log.exception("StopMove on disarm failed")
|
||||
log.info("movement DISARMED")
|
||||
return {"ok": True, "armed": False}
|
||||
|
||||
def _start_watchdog(self):
|
||||
self._wd_stop.clear()
|
||||
if self._wd_thread is None or not self._wd_thread.is_alive():
|
||||
self._wd_thread = threading.Thread(
|
||||
target=self._watchdog_loop, daemon=True, name="loco-watchdog")
|
||||
self._wd_thread.start()
|
||||
|
||||
def _watchdog_loop(self):
|
||||
period = max(0.02, min(0.1, self._wd_timeout / 2.0))
|
||||
while not self._wd_stop.is_set():
|
||||
fire = False
|
||||
park = False
|
||||
# Read-and-decide under the lock (atomic check-then-act); the actual
|
||||
# StopMove runs after release so the critical section stays tiny.
|
||||
with self._lock:
|
||||
if self._teleop_active and (time.monotonic() - self._last_move_ts) > self._wd_timeout:
|
||||
self._teleop_active = False
|
||||
fire = True
|
||||
# Self-park once there's nothing left to guard. The Gemini
|
||||
# dispatch path uses step() directly and never calls
|
||||
# disarm_movement(), so without this the watchdog would spin for
|
||||
# the rest of the process lifetime after the first voice step.
|
||||
if not self._armed and not self._teleop_active and not self._discrete_busy:
|
||||
park = True
|
||||
if fire:
|
||||
log.warning("watchdog: teleop setpoint stale (>%.2fs) — StopMove",
|
||||
self._wd_timeout)
|
||||
try:
|
||||
self._raw_stop()
|
||||
except Exception:
|
||||
log.exception("watchdog StopMove failed")
|
||||
if park:
|
||||
# Nothing left to guard — stop the thread (a later move/step
|
||||
# re-arms it via _start_watchdog()). Done AFTER any stale-stop
|
||||
# above so we never skip a pending StopMove.
|
||||
self._wd_stop.set()
|
||||
break
|
||||
self._wd_stop.wait(period)
|
||||
|
||||
def _raw_stop(self) -> bool:
|
||||
"""Issue StopMove if the client is up; no-op in sim. Lock-light."""
|
||||
if not self._lc_ready or self._bot is None:
|
||||
return False
|
||||
with self._lock:
|
||||
ok, _ = self._safe_call("StopMove", self._bot.StopMove)
|
||||
return ok
|
||||
|
||||
# ── movement ─────────────────────────────────────────────────────────────
|
||||
|
||||
def move(self, vx: float, vy: float, vyaw: float, run: bool = False) -> dict:
|
||||
cap = self._cap_run if run else self._cap_walk
|
||||
cvx, cvy, cvyaw = self._clamp(vx, cap), self._clamp(vy, cap), self._clamp(vyaw, cap)
|
||||
capped = (cvx, cvy, cvyaw) != (float(vx), float(vy), float(vyaw))
|
||||
warning = self._walk_ready_warning()
|
||||
sent = {"vx": cvx, "vy": cvy, "vyaw": cvyaw}
|
||||
|
||||
with self._lock:
|
||||
my_gen = self._stop_gen # capture under lock
|
||||
|
||||
if not self._ensure_client():
|
||||
with self._lock: # sim: record intent for UI/watchdog
|
||||
self._cur_v = (cvx, cvy, cvyaw)
|
||||
self._last_move_ts = time.monotonic()
|
||||
self._teleop_active = True
|
||||
self._start_watchdog()
|
||||
return {"ok": True, "sent": sent, "capped": capped,
|
||||
"warning": warning, "simulated": True}
|
||||
with self._lock:
|
||||
# If an E-STOP / stop / disarm landed since we captured my_gen, do NOT
|
||||
# (re)command velocity — and do NOT stamp the motion flags (so a
|
||||
# cancelled tick doesn't extend the arm-block window).
|
||||
if self._stop_gen != my_gen:
|
||||
return {"ok": False, "cancelled": True, "sent": sent,
|
||||
"capped": capped, "warning": warning, "simulated": False}
|
||||
self._cur_v = (cvx, cvy, cvyaw)
|
||||
self._last_move_ts = time.monotonic()
|
||||
self._teleop_active = True
|
||||
self._safe_call("SetBalanceMode", self._bot.SetBalanceMode, 1)
|
||||
ok, _ = self._safe_call("Move", self._bot.Move, cvx, cvy, cvyaw, True)
|
||||
self._start_watchdog()
|
||||
return {"ok": bool(ok), "sent": sent, "capped": capped,
|
||||
"warning": warning, "simulated": False}
|
||||
|
||||
def stop_move(self) -> dict:
|
||||
"""Halt translation/rotation. Allowed even when disarmed."""
|
||||
with self._lock:
|
||||
self._stop_gen += 1
|
||||
self._teleop_active = False
|
||||
if not self._ensure_client():
|
||||
return {"ok": True, "simulated": True}
|
||||
ok = self._raw_stop()
|
||||
return {"ok": bool(ok), "simulated": False}
|
||||
|
||||
def estop(self) -> dict:
|
||||
"""Emergency stop = StopMove only (no Damp / FSM change → keeps posture).
|
||||
ALWAYS allowed, even disarmed and in sim. Bumps the stop generation so any
|
||||
in-flight move()/step()/prep_mode() bails immediately (no lock wait)."""
|
||||
with self._lock:
|
||||
self._stop_gen += 1
|
||||
self._teleop_active = False
|
||||
self._cur_v = (0.0, 0.0, 0.0)
|
||||
if not self._ensure_client():
|
||||
log.warning("E-STOP (sim)")
|
||||
return {"ok": True, "simulated": True}
|
||||
ok = self._raw_stop()
|
||||
log.warning("E-STOP — StopMove issued")
|
||||
return {"ok": bool(ok), "simulated": False}
|
||||
|
||||
def step(self, direction: str) -> dict:
|
||||
"""Discrete one-step pad: Move for step_duration then StopMove.
|
||||
Blocking (~step_duration); call via asyncio.to_thread from the route.
|
||||
|
||||
The sleep loop does NOT hold self._lock, so E-STOP / StopMove (which take
|
||||
the lock briefly) preempt it immediately; the loop also bails the moment
|
||||
the stop generation changes."""
|
||||
if direction not in _STEP_DIRS:
|
||||
return {"ok": False, "reason": f"unknown direction: {direction}"}
|
||||
sx, sy, syaw = _STEP_DIRS[direction]
|
||||
k = self._cap_walk * self._step_frac
|
||||
vx, vy, vyaw = sx * k, sy * k, syaw * k
|
||||
warning = self._walk_ready_warning()
|
||||
with self._lock:
|
||||
if self._discrete_busy:
|
||||
return {"ok": False, "dir": direction, "reason": "busy",
|
||||
"warning": warning, "simulated": not self._lc_ready}
|
||||
self._discrete_busy = True
|
||||
my_gen = self._stop_gen
|
||||
self._last_move_ts = time.monotonic()
|
||||
self._teleop_active = True
|
||||
self._start_watchdog()
|
||||
if not self._ensure_client():
|
||||
with self._lock:
|
||||
self._teleop_active = False
|
||||
self._discrete_busy = False
|
||||
return {"ok": True, "dir": direction, "warning": warning, "simulated": True}
|
||||
try:
|
||||
with self._lock:
|
||||
if self._stop_gen != my_gen: # stopped before we began
|
||||
return {"ok": False, "dir": direction, "cancelled": True,
|
||||
"warning": warning, "simulated": False}
|
||||
self._safe_call("SetBalanceMode", self._bot.SetBalanceMode, 1)
|
||||
self._safe_call("Move", self._bot.Move, vx, vy, vyaw, True)
|
||||
t_end = time.monotonic() + self._step_dur
|
||||
while time.monotonic() < t_end:
|
||||
if self._stop_gen != my_gen:
|
||||
break
|
||||
with self._lock:
|
||||
self._last_move_ts = time.monotonic() # keep watchdog fed
|
||||
time.sleep(0.05)
|
||||
finally:
|
||||
with self._lock:
|
||||
self._safe_call("StopMove", self._bot.StopMove)
|
||||
self._teleop_active = False
|
||||
self._discrete_busy = False
|
||||
return {"ok": True, "dir": direction, "warning": warning, "simulated": False}
|
||||
|
||||
# ── postures / modes ─────────────────────────────────────────────────────
|
||||
|
||||
def prep_mode(self) -> dict:
|
||||
"""PREP — StopMove → Damp → StandUp(FSM4) → height ramp → BalanceStand(0).
|
||||
Exact order from g1_mode_controller.prep_mode, minus the blocking input().
|
||||
Blocking (~1s); call via asyncio.to_thread."""
|
||||
if not self._ensure_client():
|
||||
return {"ok": True, "mode": "prep", "simulated": True}
|
||||
with self._lock:
|
||||
if self._discrete_busy:
|
||||
return {"ok": False, "mode": "prep", "reason": "busy", "simulated": False}
|
||||
self._discrete_busy = True
|
||||
my_gen = self._stop_gen
|
||||
self._safe_call("StopMove", self._bot.StopMove)
|
||||
self._safe_call("Damp", self._bot.Damp)
|
||||
self._safe_call("SetFsmId(4)", self._bot.SetFsmId, 4)
|
||||
try:
|
||||
# Height ramp OUTSIDE the lock so E-STOP can preempt at any time.
|
||||
h = 0.02
|
||||
while h <= 0.5 + 1e-9:
|
||||
if self._stop_gen != my_gen:
|
||||
log.warning("PREP cancelled (E-STOP)")
|
||||
return {"ok": False, "mode": "prep", "cancelled": True, "simulated": False}
|
||||
with self._lock:
|
||||
self._safe_call("SetStandHeight", self._bot.SetStandHeight, round(h, 3))
|
||||
time.sleep(0.03)
|
||||
h += 0.02
|
||||
with self._lock:
|
||||
self._safe_call("BalanceStand", self._bot.BalanceStand, 0)
|
||||
self._safe_call("SetStandHeight", self._bot.SetStandHeight, 0.22)
|
||||
finally:
|
||||
with self._lock:
|
||||
self._discrete_busy = False
|
||||
log.info("PREP complete")
|
||||
return {"ok": True, "mode": "prep", "simulated": False}
|
||||
|
||||
def ready_start_mode(self) -> dict:
|
||||
"""READY = PREP then Start (FSM 200 / balance engaged)."""
|
||||
self.prep_mode()
|
||||
if not self._ensure_client():
|
||||
return {"ok": True, "mode": "ready", "simulated": True}
|
||||
with self._lock:
|
||||
if hasattr(self._bot, "Start"):
|
||||
ok, _ = self._safe_call("Start", self._bot.Start)
|
||||
else:
|
||||
ok, _ = self._safe_call("SetFsmId(200)", self._bot.SetFsmId, 200)
|
||||
log.info("READY/START complete")
|
||||
return {"ok": bool(ok), "mode": "ready", "simulated": False}
|
||||
|
||||
def posture(self, name: str) -> dict:
|
||||
if name not in _POSTURES:
|
||||
return {"ok": False, "reason": f"unknown posture: {name}"}
|
||||
if not self._ensure_client():
|
||||
return {"ok": True, "posture": name, "simulated": True}
|
||||
bot = self._bot
|
||||
with self._lock:
|
||||
if name == "zero_torque":
|
||||
ok, _ = self._safe_call("ZeroTorque", bot.ZeroTorque)
|
||||
elif name == "damp":
|
||||
ok, _ = self._safe_call("Damp", bot.Damp)
|
||||
elif name == "stand_up":
|
||||
ok, _ = self._safe_call("SetFsmId(4)", bot.SetFsmId, 4)
|
||||
elif name == "squat":
|
||||
ok, _ = self._safe_call("SetFsmId(2)", bot.SetFsmId, 2)
|
||||
elif name == "sit":
|
||||
ok, _ = self._safe_call("Sit", bot.Sit)
|
||||
elif name == "low_stand":
|
||||
ok, _ = self._safe_call("LowStand", bot.LowStand)
|
||||
elif name == "high_stand":
|
||||
ok, _ = self._safe_call("HighStand", bot.HighStand)
|
||||
elif name == "lie_to_stand":
|
||||
if hasattr(bot, "Lie2StandUp"):
|
||||
ok, _ = self._safe_call("Lie2StandUp", bot.Lie2StandUp)
|
||||
else:
|
||||
ok, _ = self._safe_call("SetFsmId(702)", bot.SetFsmId, 702)
|
||||
else: # unreachable (guarded above)
|
||||
ok = False
|
||||
return {"ok": bool(ok), "posture": name, "simulated": False}
|
||||
|
||||
def set_balance_mode(self, mode: int) -> dict:
|
||||
if not self._ensure_client():
|
||||
return {"ok": True, "balance_mode": int(mode), "simulated": True}
|
||||
with self._lock:
|
||||
ok, _ = self._safe_call("SetBalanceMode", self._bot.SetBalanceMode, int(mode))
|
||||
return {"ok": bool(ok), "balance_mode": int(mode), "simulated": False}
|
||||
|
||||
def set_stand_height(self, h: float) -> dict:
|
||||
if not self._ensure_client():
|
||||
return {"ok": True, "height": float(h), "simulated": True}
|
||||
with self._lock:
|
||||
ok, _ = self._safe_call("SetStandHeight", self._bot.SetStandHeight, float(h))
|
||||
return {"ok": bool(ok), "height": float(h), "simulated": False}
|
||||
|
||||
# ── MotionSwitcher ───────────────────────────────────────────────────────
|
||||
|
||||
def msc_check(self) -> dict:
|
||||
if not self._ensure_client() or self._msc is None:
|
||||
return {"mode_name": None, "simulated": not self._lc_ready}
|
||||
try:
|
||||
ret = self._msc.CheckMode()
|
||||
name = None
|
||||
if isinstance(ret, tuple) and len(ret) >= 2 and isinstance(ret[1], dict):
|
||||
name = ret[1].get("name")
|
||||
elif isinstance(ret, dict):
|
||||
name = ret.get("name")
|
||||
self._last_msc_mode = name
|
||||
return {"mode_name": name}
|
||||
except Exception as exc:
|
||||
log.error("msc_check failed: %s", exc)
|
||||
return {"mode_name": None}
|
||||
|
||||
def msc_select_ai(self) -> dict:
|
||||
if not self._ensure_client() or self._msc is None:
|
||||
return {"ok": True, "simulated": True}
|
||||
with self._lock:
|
||||
ok, _ = self._safe_call("SelectMode(ai)", self._msc.SelectMode, "ai")
|
||||
return {"ok": bool(ok), "simulated": False}
|
||||
|
||||
def msc_release(self) -> dict:
|
||||
if not self._ensure_client() or self._msc is None:
|
||||
return {"ok": True, "simulated": True}
|
||||
with self._lock:
|
||||
ok, _ = self._safe_call("ReleaseMode", self._msc.ReleaseMode)
|
||||
return {"ok": bool(ok), "simulated": False}
|
||||
|
||||
def reconnect(self) -> dict:
|
||||
"""Drop and rebuild Loco + MSC clients (does NOT re-init the DDS factory)."""
|
||||
with self._lock:
|
||||
self._bot = None
|
||||
self._msc = None
|
||||
self._lc_ready = False
|
||||
ok = self._ensure_client()
|
||||
return {"ok": bool(ok), "lc_ready": self._lc_ready}
|
||||
|
||||
# ── reads ────────────────────────────────────────────────────────────────
|
||||
|
||||
def joints(self) -> dict:
|
||||
q: list = []
|
||||
try:
|
||||
if self._arm is not None:
|
||||
q = self._arm.get_current_q()
|
||||
except Exception:
|
||||
q = []
|
||||
out = []
|
||||
for idx in range(12, 29):
|
||||
val = q[idx] if idx < len(q) else 0.0
|
||||
out.append({"idx": idx, "name": JOINT_NAMES.get(idx, f"motor_{idx}"),
|
||||
"q": float(val)})
|
||||
return {"joints": out}
|
||||
|
||||
def status(self) -> dict:
|
||||
# Polling /status lazily brings up the client once arm DDS is ready.
|
||||
self._ensure_client()
|
||||
fid = self.fsm_id() if self._lc_ready else None
|
||||
fmode = self.fsm_mode() if self._lc_ready else None
|
||||
walk_ready = bool(self._lc_ready and fid == 200 and fmode not in (None, 2))
|
||||
return {
|
||||
"sdk_available": _HAS_SDK,
|
||||
"lc_ready": self._lc_ready,
|
||||
"armed": self._armed,
|
||||
"fsm_id": fid,
|
||||
"fsm_mode": fmode,
|
||||
"walk_ready": walk_ready,
|
||||
"msc_mode": self._last_msc_mode,
|
||||
"teleop_active": self._teleop_active,
|
||||
"last_velocity": {"vx": self._cur_v[0], "vy": self._cur_v[1], "vyaw": self._cur_v[2]},
|
||||
"caps": {"walk": self._cap_walk, "run": self._cap_run},
|
||||
"arm_initialized": bool(self._arm is not None and getattr(self._arm, "_initialized", False)),
|
||||
}
|
||||
|
||||
# ── shutdown helper ──────────────────────────────────────────────────────
|
||||
|
||||
def shutdown(self):
|
||||
"""Best-effort StopMove + disarm for process shutdown.
|
||||
|
||||
Uses _raw_stop() (NOT estop()) so teardown never builds a brand-new
|
||||
LocoClient: estop() → _ensure_client() would lazily construct a client
|
||||
and run bot.Init() (a DDS RPC) during interpreter teardown when we were
|
||||
armed-but-never-built (Enable movement clicked, never moved, then
|
||||
Ctrl+C). _raw_stop() no-ops when no client was ever created. Bump the
|
||||
stop generation so any in-flight motion bails immediately."""
|
||||
with self._lock:
|
||||
self._stop_gen += 1
|
||||
self._teleop_active = False
|
||||
self._cur_v = (0.0, 0.0, 0.0)
|
||||
try:
|
||||
self._raw_stop() # no-op when _bot is None — never re-inits
|
||||
except Exception:
|
||||
log.exception("StopMove on shutdown failed")
|
||||
finally:
|
||||
self.disarm_movement()
|
||||
412
README.md
Normal file
412
README.md
Normal file
@ -0,0 +1,412 @@
|
||||
# Sanad
|
||||
|
||||
Voice + motion assistant for the Unitree G1 humanoid. **Gemini Live** (or a
|
||||
fully-offline pipeline) handles bilingual Arabic/English conversation; an arm
|
||||
controller plays built-in SDK poses and recorded JSONL macros; a locomotion
|
||||
controller walks/turns the robot; an optional camera feeds **Gemini-side face &
|
||||
place recognition**; everything is orchestrated through a fault-isolated
|
||||
**FastAPI dashboard** on `http://<robot>:8000`.
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────────────────────────┐
|
||||
│ Dashboard (FastAPI) ── http://<robot>:8000 │
|
||||
│ ├─ Operations Quick-fire arm actions + gestural-speaking │
|
||||
│ ├─ Voice & Audio Live Gemini, Typed Replay, Wake Phrases, Audio │
|
||||
│ ├─ Motion & Replay SDK actions, JSONL replays, macros, teaching │
|
||||
│ ├─ Controller Locomotion teleop, postures, FSM modes, E-STOP │
|
||||
│ ├─ Recognition Camera vision + face gallery + zones/places │
|
||||
│ ├─ Recordings Skill registry, saved Gemini turns │
|
||||
│ ├─ Temperature Live 3D motor-temperature heatmap (three.js) │
|
||||
│ ├─ Terminal In-browser shell (PTY) to the robot │
|
||||
│ └─ Settings & Logs System info, tail/stream live logs │
|
||||
└──────────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
├─ voice/sanad_voice.py (subprocess — model-agnostic voice loop)
|
||||
│ ├─ gemini/script.py (Gemini Live brain — audio+video+state)
|
||||
│ └─ local/script.py (offline brain — VAD→STT→LLM→TTS)
|
||||
├─ gemini/client.py (short-session client for Typed Replay)
|
||||
├─ gemini/subprocess.py (spawns+supervises sanad_voice.py;
|
||||
│ pushes camera frames + motion state
|
||||
│ to the child over its stdin)
|
||||
├─ voice/movement_dispatch.py(Gemini spoken phrase → locomotion)
|
||||
├─ vision/camera.py (RealSense/USB capture daemon)
|
||||
├─ vision/face_gallery.py (data/faces/ CRUD for the primer turn)
|
||||
├─ vision/zone_gallery.py (data/zones/ places + "go here" targets)
|
||||
├─ motion/arm_controller.py (G1 arm DDS publisher — owns DDS init)
|
||||
├─ G1_Controller/loco_controller.py (G1 locomotion via LocoClient)
|
||||
├─ voice/audio_io.py (mic + speaker abstraction — 3 profiles)
|
||||
└─ core/brain.py (skill dispatcher, event bus)
|
||||
```
|
||||
|
||||
### Camera + face/place recognition data flow
|
||||
|
||||
```
|
||||
CameraDaemon (parent, in-memory JPEG+b64 cache)
|
||||
├─→ dashboard /api/recognition/frame.jpg ── snapshot_jpeg()
|
||||
└─→ GeminiSubprocess._frame_forwarder ── get_frame_b64()
|
||||
│ "frame:<b64>\n" over stdin
|
||||
ArmController ─emit→ event bus ─→ main.py ─→ live_sub.send_state()
|
||||
│ "state:<json>\n" over stdin
|
||||
▼
|
||||
gemini/script.py _stdin_watcher thread
|
||||
├─ frame: → _LATEST_FRAME → _send_frame_loop →
|
||||
│ session.send_realtime_input(video=Blob)
|
||||
└─ state: → _STATE_PENDING → _send_state_loop →
|
||||
session.send_realtime_input(text=…)
|
||||
|
||||
Recognition toggles (vision / face-rec / zone-rec / movement) are written by the
|
||||
dashboard to data/.recognition_state.json and POLLED by the Gemini child at 1 Hz
|
||||
— so flipping a toggle takes effect mid-session with NO restart.
|
||||
```
|
||||
|
||||
|
||||
## Quick start (on the robot)
|
||||
|
||||
```bash
|
||||
conda activate gemini_sdk
|
||||
cd ~/Sanad
|
||||
python3 main.py
|
||||
```
|
||||
|
||||
Then open `http://<robot-ip>:8000` in a browser. (The dashboard binds to the
|
||||
`wlan0` IP by default — see *Runtime selection* to override.)
|
||||
|
||||
Fully-offline brain (no cloud): `SANAD_VOICE_BRAIN=local python3 main.py`
|
||||
(requires `ollama serve` + the local model env — see *Voice brains*).
|
||||
|
||||
> **Gemini API key — required, none ships with the repo.** The `api_key`
|
||||
> fields in `config/core_config.json` (`gemini_defaults`) and
|
||||
> `data/motions/config.json` (`gemini`) are intentionally empty (`""`).
|
||||
> The voice loop cannot connect until you supply one, by any of:
|
||||
> - **Dashboard** → *Voice & Audio → Gemini API Key* — paste + save, hot-swaps live (no restart). Persists to `data/motions/config.json`.
|
||||
> - **Env var** — `export SANAD_GEMINI_API_KEY=AIza...` before `python3 main.py`.
|
||||
> - **Config file** — set `gemini_defaults.api_key` in `config/core_config.json`.
|
||||
>
|
||||
> Precedence (highest first): `data/motions/config.json` → `SANAD_GEMINI_API_KEY` → `config/core_config.json`. Get a key at <https://aistudio.google.com/apikey>.
|
||||
|
||||
|
||||
## Dashboard features
|
||||
|
||||
### Operations
|
||||
Quick-fire SDK + JSONL arm actions (chip buttons), gestural-speaking toggle.
|
||||
|
||||
### Voice & Audio
|
||||
- **Live Voice Commands** — fire arm gestures from the *user's* transcript
|
||||
(wake-phrase → arm action). Master gate + Deferred-trigger toggle.
|
||||
- **Live Gemini Process** — start/stop the voice conversation subprocess, tail
|
||||
its log. Choose the Gemini cloud brain or the offline brain via
|
||||
`SANAD_VOICE_BRAIN`.
|
||||
- **Typed Replay** — Gemini reads typed text aloud (wrapped with a
|
||||
"repeat verbatim" prompt); optionally records the clip.
|
||||
- **Gemini API Key** — hot-swap the key without restart.
|
||||
- **Wake Phrase Manager** — add/remove phrase → action bindings.
|
||||
- **Audio Controls** — mic/speaker mute, G1 chest-speaker volume (DDS), device
|
||||
profile selection, PulseAudio soft-reset and Anker USB hard-reset.
|
||||
|
||||
### Motion & Replay
|
||||
- **Motion Control** — list SDK (built-in) + JSONL (recorded) actions, select +
|
||||
play. Cancel smoothly returns to `arm_home.jsonl`.
|
||||
- **Replay Manager** — upload `.jsonl` files, test-play with speed, Teaching
|
||||
Mode (kinesthetic record — limp the arm and hand-guide it).
|
||||
- **Macro Recorder** — record a new audio+motion pair, OR pick any WAV + any
|
||||
motion (SDK or JSONL) and play them in parallel.
|
||||
|
||||
### Controller *(locomotion)*
|
||||
Manual teleoperation of the G1's **legs** via the Unitree `LocoClient`.
|
||||
**Disarmed every boot**; all motion writes require Arm first.
|
||||
- **Move / Step** — continuous teleop (vx/vy/vyaw) or discrete one-shot steps.
|
||||
- **Postures & FSM modes** — zero-torque, damp, squat, sit, stand, balance,
|
||||
stand-height; prep/ready sequences; MotionSwitcher select-AI/release.
|
||||
- **Gemini Movement** — toggle voice-driven walking: the `MovementDispatcher`
|
||||
parses Gemini's *own spoken confirmation phrases* ("Turning right." /
|
||||
"أستدير يميناً.") and drives the legs (gated on this toggle + an E-STOP latch).
|
||||
- **E-STOP** — always available; `StopMove` + disarm + latch the dispatcher.
|
||||
|
||||
> **Safety:** the arm and locomotion are **mutually exclusive** —
|
||||
> `arm.set_motion_block(loco.movement_active)` makes every arm
|
||||
> replay/gesture refuse while the robot is (or just was, within ~1.5 s) walking.
|
||||
|
||||
### Recognition
|
||||
Camera vision + Gemini-side **face** and **zone/place** recognition. All are
|
||||
**off by default**; each is a **hot toggle** (≈1 s to take effect, no restart).
|
||||
- **Camera Vision** — `CameraDaemon` captures from a RealSense (preferred) or
|
||||
USB camera; the supervisor streams JPEG frames to Gemini Live so it can answer
|
||||
"what do you see?". Live preview panel. Auto-reconnects on USB unplug/stall
|
||||
and warns if a RealSense negotiated USB 2.0 (Marcus-ported resilience).
|
||||
- **Face Recognition** — manage `data/faces/face_{id}/` galleries: enroll from
|
||||
the live camera or upload photos, rename, describe, download (per-photo or
|
||||
ZIP), delete. On session start (and on any gallery change) the child sends a
|
||||
**primer turn** carrying every enrolled face + a Khaleeji greeting
|
||||
instruction — **Gemini matches in-context, so there is no local
|
||||
face-recognition model**. Recognition needs vision on.
|
||||
- **Zones & Places** — `data/zones/zone_{zid}/place_{pid}/` two-level gallery:
|
||||
reference photos per place, optional linked face_ids, and a **"go here"** nav
|
||||
target (`nav_target_zone/place_id` in the recognition-state file) for
|
||||
place-aware navigation.
|
||||
- **Sync Gallery** — force-resend the face/zone primer to the live session.
|
||||
|
||||
### Recordings
|
||||
Skill Registry (predefined audio+motion+callback skills from `skills.json`) +
|
||||
Saved Records (captured Gemini turn recordings; play/pause/stop/rename/delete).
|
||||
|
||||
### Temperature
|
||||
Live **3D motor-temperature heatmap** — a standalone three.js viewer
|
||||
(`dashboard/static/temp3d/`) loads the G1 29-DoF URDF + STL meshes and colors
|
||||
each joint blue→red from the arm controller's throttled `rt/lowstate` snapshot,
|
||||
streamed over `/ws/motor-temps` at ~8 fps. No second DDS subscriber.
|
||||
|
||||
### Terminal
|
||||
In-browser **PTY shell** to the robot (`/ws/terminal`, xterm.js) — a `bash -i`
|
||||
as the dashboard's user, with resize + backpressure, bounded to 4 sessions.
|
||||
(See *Security* — this is full shell access to whoever reaches the URL.)
|
||||
|
||||
### Settings & Logs
|
||||
System info (host, network interfaces, DDS interface, bound dashboard host/port,
|
||||
per-subsystem status, audio devices), live log stream (`/ws/logs`), per-file
|
||||
tail, snapshot, and a one-blob "Copy All Logs" bundle.
|
||||
|
||||
|
||||
## Directory layout
|
||||
|
||||
| Path | Contents |
|
||||
|---|---|
|
||||
| `main.py` | Entry point — fault-isolated boot of all subsystems + the dashboard. Doubles as the service container (route handlers `import` its module globals). |
|
||||
| `config.py` | Runtime constants + layout-agnostic path resolution; layers `data/motions/config.json` over the JSON config at import. |
|
||||
| `config/` | Per-subsystem JSON: `core`, `voice`, `gemini`, `local`, `motion`, `dashboard`. |
|
||||
| `core/` | `brain.py` (skill dispatcher), `event_bus.py`, `skill_registry.py`, `config_loader.py`, `logger.py` (rotating + WS push), `asyncio_compat.py` (3.8 `to_thread` shim). |
|
||||
| `gemini/` | Gemini Live — `client.py` (one-shot), `script.py` (live brain: audio + video + motion-state), `subprocess.py` (supervisor + stdin frame/state push). |
|
||||
| `local/` | Fully-offline brain — `vad.py` (Silero), `stt.py` (faster-whisper), `llm.py` (Qwen via Ollama/llama.cpp), `tts.py` (CosyVoice2), `script.py` (the brain), `subprocess.py` (supervisor). Opt-in via `SANAD_VOICE_BRAIN=local`. |
|
||||
| `voice/` | `sanad_voice.py` (subprocess entry, model-agnostic), `audio_io.py` / `audio_manager.py` / `audio_devices.py` (mic/speaker), `local_tts.py` (SpeechT5 Arabic TTS), `live_voice_loop.py` (user-transcript → arm gesture), `movement_dispatch.py` (Gemini-phrase → locomotion), `typed_replay.py`, `wake_phrase_manager.py`, `text_utils.py` (Arabic normalization + phrase matching), `model_script.py` / `model_subprocess.py` (brain templates). |
|
||||
| `motion/` | `arm_controller.py` (production 5-phase JSONL replay engine, owns the single DDS init), `macro_player.py`, `macro_recorder.py`, `teaching.py`. (`sanad_arm_controller.py` is a legacy alternate — not wired by `main.py`.) |
|
||||
| `G1_Controller/` | `loco_controller.py` — locomotion via Unitree `LocoClient` (move/step/postures/FSM/E-STOP); reuses the arm's DDS participant. |
|
||||
| `vision/` | `camera.py` (RealSense/USB daemon, auto-reconnect), `face_gallery.py`, `zone_gallery.py`, `recognition_state.py` (atomic-JSON toggle IPC). |
|
||||
| `dashboard/` | `app.py` (FastAPI factory + fault-isolated router registration), `routes/*.py` (20 REST routers), `websockets/*.py` (logs, motor-temps, terminal), `static/index.html` (single-page UI), `static/temp3d/` (3D viewer). |
|
||||
| `scripts/` | Persona files — `sanad_script.txt` (voice persona "Bousandah"), `sanad_rule.txt`, `sanad_arm.txt` (voice→arm phrases). |
|
||||
| `data/` | Runtime state — `motions/*.jsonl` (arm trajectories) + `instruction.json` (locomotion phrase map) + `skills.json` + `config.json` (dashboard-editable), `recordings/` (captured turns + macros), `faces/face_{id}/` + `zones/zone_{zid}/place_{pid}/` (galleries), `audio/` (typed-replay WAVs + records index), `.recognition_state.json` (toggle IPC). |
|
||||
| `model/` | Local SpeechT5 / Whisper / CosyVoice2 weights when using the offline pipeline. |
|
||||
| `logs/` | Per-module rotating logs. |
|
||||
|
||||
|
||||
## Voice brains
|
||||
|
||||
The child `voice/sanad_voice.py` is model-agnostic and selects a brain via
|
||||
`SANAD_VOICE_BRAIN`. Every brain implements the same contract
|
||||
(`__init__(audio_io, recorder, voice, system_prompt)`, `async run()`, `stop()`)
|
||||
and ships a sibling supervisor that spawns the child and parses its
|
||||
`USER:` / `BOT:` / state log markers.
|
||||
|
||||
| Value | Brain | Pipeline |
|
||||
|---|---|---|
|
||||
| `gemini` *(default)* | `gemini/script.py` | Gemini Live native-audio (full-duplex speech-to-speech, server-side VAD, vision frames, face/zone primers, voice→movement). Cloud. |
|
||||
| `local` | `local/script.py` | Silero VAD → faster-whisper (large-v3-turbo, CUDA int8) → Qwen2.5 (Ollama/llama.cpp) → CosyVoice2 streaming TTS. Fully on-device. |
|
||||
| `model` | `voice/model_script.py` | Template/stub for adding a new provider (OpenAI Realtime, Claude Voice, …). |
|
||||
|
||||
To add a brain: drop a file in `voice/` or a new `<brand>/` folder and add a
|
||||
branch to `voice/sanad_voice.py:_build_brain()`; ship a supervisor modeled on
|
||||
`voice/model_subprocess.py`.
|
||||
|
||||
|
||||
## Runtime selection (env vars)
|
||||
|
||||
| Var | Values | Default | Effect |
|
||||
|---|---|---|---|
|
||||
| `SANAD_VOICE_BRAIN` | `gemini`, `local`, `model` | `gemini` | Which brain the subprocess loads (see `voice/sanad_voice.py:_build_brain`). |
|
||||
| `SANAD_AUDIO_PROFILE` | `builtin`, `anker`, `hollyland_builtin` | `builtin` | Mic + speaker pair. `builtin` = G1 UDP mic + G1 chest speaker via DDS. |
|
||||
| `SANAD_DDS_INTERFACE` | network iface | `eth0` | DDS network for G1 low-level comms (arm + locomotion + speaker). |
|
||||
| `SANAD_DASHBOARD_HOST` / `_INTERFACE` | IP / iface | `wlan0` IP | Dashboard bind address. |
|
||||
| `SANAD_GEMINI_API_KEY` | string | `""` (empty) | Gemini API key. No key ships in the repo — set this, paste one in the dashboard (**Voice & Audio → Gemini API Key**), or fill `gemini_defaults.api_key` in `config/core_config.json`. See [Quick start](#quick-start-on-the-robot). |
|
||||
| `SANAD_GEMINI_MODEL` / `_VOICE` | string | reads config | Override the Gemini model id / prebuilt voice. |
|
||||
| `SANAD_G1_VOLUME` | `0`–`100` | `100` | G1 chest-speaker volume; also scales the barge-in threshold. |
|
||||
| `SANAD_LIVE_SCRIPT` | path | auto | Override the subprocess entry script path. |
|
||||
| `SANAD_RECORD` | `0` or `1` | `1` | Record every Gemini turn to `data/recordings/`. |
|
||||
| `SANAD_AEC_ENABLE` | `0` or `1` | `1` | Enable WebRTC AEC3 (if the Python binding is installed). |
|
||||
| `SANAD_VISION_ENABLE` | `0` or `1` | `0` | Boot default for camera vision. **Runtime truth is the Recognition-tab toggle** → `data/.recognition_state.json`, hot-applied without a restart. |
|
||||
| `SANAD_FACE_RECOGNITION_ENABLE` | `0` or `1` | `0` | Boot default for Gemini-side face recognition. Also a hot toggle. |
|
||||
| `SANAD_VISION_SEND_HZ` | float | `2` | Frames/sec the Gemini child relays to Live. |
|
||||
| `SANAD_CAMERA_WIDTH` / `_HEIGHT` / `_FPS` | int | `424` / `240` / `15` | Capture profile. Also settable per-deploy in `config/core_config.json > camera`. |
|
||||
| `SANAD_CAMERA_USB_INDEX` | int | auto | Pin a `/dev/videoN` node (avoids picking a RealSense IR stream). |
|
||||
| `SANAD_FACES_MAX_SAMPLES` | int | `3` | Max photos per person fed into the gallery primer turn (token budget). |
|
||||
| `SANAD_PROJECT_ROOT` | path | auto | Override the project root (see *Dynamic paths*). |
|
||||
|
||||
> All `SANAD_VISION_*` / `SANAD_CAMERA_*` / `SANAD_FACE_*` vars are **boot
|
||||
> defaults** forwarded to the Gemini child via `LIVE_TUNE`. Once running, the
|
||||
> Recognition tab's toggles (vision / face-rec / zone-rec / movement) are the
|
||||
> live source of truth in `data/.recognition_state.json`, polled at 1 Hz.
|
||||
|
||||
CLI flags: `python3 main.py --host <ip> --port 8000 --network <dds_iface>`;
|
||||
`--check-env` prints a subsystem/environment diagnostic and exits.
|
||||
|
||||
|
||||
## API surface
|
||||
|
||||
All routes are registered defensively — a router whose import fails is recorded
|
||||
(`GET /api/_dashboard_status`) and the server still boots without it.
|
||||
|
||||
**REST** (prefix → controls): `/api` health · `/api/system` info ·
|
||||
`/api/voice` Gemini/local generate+connect+key · `/api/motion` arm actions ·
|
||||
`/api/skills` skill registry · `/api/macros` record/play · `/api/replay` JSONL
|
||||
CRUD + teaching · `/api/audio` mute/volume/devices/reset · `/api/scripts`
|
||||
persona files · `/api/records` saved WAVs · `/api/prompt` system prompt ·
|
||||
`/api/wake-phrases` bindings · `/api/live-voice` arm-phrase dispatcher ·
|
||||
`/api/live-subprocess` Gemini child · `/api/typed-replay` TTS · `/api/recognition`
|
||||
vision + face gallery · `/api/zones` zones/places + nav target · `/api/temp`
|
||||
motor map + snapshot · `/api/controller` locomotion (move/step/postures/modes/
|
||||
E-STOP).
|
||||
|
||||
**WebSockets**: `/ws/logs` (live log stream + 500-line replay) ·
|
||||
`/ws/motor-temps` (3D heatmap data, ~8 fps) · `/ws/terminal` (PTY shell).
|
||||
|
||||
|
||||
## Architecture notes
|
||||
|
||||
- **Subprocess isolation**: `voice/sanad_voice.py` runs as a child of `main.py`
|
||||
via the supervisor. If the voice loop crashes, the dashboard + arm + legs stay
|
||||
up.
|
||||
- **Single DDS init**: `motion/arm_controller.py` owns the one
|
||||
`ChannelFactoryInitialize`; `LocoController` and the audio routes reuse that
|
||||
participant rather than re-initializing.
|
||||
- **Brain contract**: see `voice/model_script.py` — any new model implements
|
||||
`__init__(audio_io, recorder, voice, system_prompt)`, `async run()`, `stop()`.
|
||||
- **Supervisor contract**: each brain ships a sibling supervisor (e.g.
|
||||
`gemini/subprocess.py`) that spawns `sanad_voice.py` with its
|
||||
`SANAD_VOICE_BRAIN` and parses the brain's log markers. Template:
|
||||
`voice/model_subprocess.py`.
|
||||
- **Locomotion safety**: `LocoController` is disarmed every boot, has velocity
|
||||
caps + a `StopMove` watchdog, and is mutually exclusive with the arm.
|
||||
Voice-driven movement is **off by default** and gated by the Controller
|
||||
toggle. Distances/degrees in `data/motions/instruction.json` are
|
||||
**approximate and must be calibrated on the real robot** — there is no
|
||||
obstacle/abort stack.
|
||||
- **Audio routing**: the G1's platform-sound PulseAudio sink is NOT wired to a
|
||||
physical speaker. All dashboard-triggered playback (`play_wav`, typed-replay
|
||||
audio, record playback) routes through DDS `AudioClient.PlayStream` via
|
||||
`audio_manager._play_pcm_via_g1`. The PyAudio path is a desktop/dev fallback.
|
||||
- **Arm replay**: `motion/arm_controller.py:_replay_file_inner()` is a port of
|
||||
`G1_Lootah/Manual_Recorder/g1_replay_v4_stable.py:Run()` — ramp-in → settle
|
||||
hold → playback → smooth return → disable SDK. Body motors (0–14) lock to a
|
||||
live snapshot while arm motors (15–28) follow the file at 60 Hz. `_return_home()`
|
||||
runs unconditionally after a cancel for a jerk-free return.
|
||||
- **Camera frame transport (stdin push)**: the `CameraDaemon` lives in the
|
||||
parent and caches frames in memory. `GeminiSubprocess` base64-encodes the
|
||||
latest frame to the child's stdin (~2 fps); the child's `_stdin_watcher`
|
||||
relays it to Gemini Live with a staleness guard. Chosen over a file drop so
|
||||
the parent owns the camera once and the dashboard preview reads the same cache.
|
||||
- **Motion-state channel**: `arm_controller._execute()` emits
|
||||
`motion.action_started` / `_done` / `_error` on the event bus. `main.py`
|
||||
forwards each to the child as `state:<json>\n`, injected to Gemini Live as
|
||||
silent `[STATE-START] wave_hand` / `[STATE-DONE] wave_hand (2.3s)` text so it
|
||||
can honestly answer "what are you doing?".
|
||||
- **Recognition is Gemini-side**: no dlib/insightface/onnxruntime. Galleries are
|
||||
pure file IO; `gemini/script.py:_send_gallery_primer()` builds one multimodal
|
||||
`send_client_content` turn — every enrolled face/place's photos + a greeting
|
||||
instruction — and Gemini matches incoming frames against it in-context.
|
||||
|
||||
|
||||
## Camera vision on Jetson
|
||||
|
||||
The Recognition tab needs `pyrealsense2` to talk to the Intel RealSense.
|
||||
**Do not `pip install pyrealsense2` on JetPack 5** — the PyPI wheel is built
|
||||
against glibc 2.32+ (Ubuntu 22.04) and fails to load on JetPack 5's glibc
|
||||
2.31 with `ImportError: ... version 'GLIBC_2.32' not found`.
|
||||
|
||||
The native runtime is already there (`apt`-installed `librealsense2`). Build
|
||||
just the Python binding from source against it, into the `gemini_sdk` env:
|
||||
|
||||
```bash
|
||||
rs-enumerate-devices # confirm the D435I shows up at OS level first
|
||||
|
||||
source ~/miniconda3/etc/profile.d/conda.sh && conda activate gemini_sdk
|
||||
pip uninstall -y pyrealsense2 # remove the broken wheel if present
|
||||
sudo apt install -y cmake build-essential git python3-dev libusb-1.0-0-dev pkg-config libssl-dev
|
||||
|
||||
cd /tmp && rm -rf librealsense
|
||||
git clone --depth=1 --branch v2.56.5 https://github.com/IntelRealSense/librealsense.git
|
||||
cd librealsense && mkdir -p build && cd build
|
||||
cmake .. -DBUILD_PYTHON_BINDINGS=ON -DPYTHON_EXECUTABLE=$(which python3) \
|
||||
-DBUILD_EXAMPLES=OFF -DBUILD_GRAPHICAL_EXAMPLES=OFF \
|
||||
-DBUILD_UNIT_TESTS=OFF -DCHECK_FOR_UPDATES=OFF -DCMAKE_BUILD_TYPE=Release
|
||||
make -j$(nproc) pyrealsense2
|
||||
SITE=$(python3 -c "import sysconfig; print(sysconfig.get_paths()['purelib'])")
|
||||
mkdir -p "$SITE/pyrealsense2"
|
||||
cp wrappers/python/pyrealsense2*.so "$SITE/pyrealsense2/"
|
||||
cp ../wrappers/python/pyrealsense2/__init__.py "$SITE/pyrealsense2/" 2>/dev/null || true
|
||||
|
||||
python3 -c 'import pyrealsense2 as rs; print([d.get_info(rs.camera_info.name) for d in rs.context().query_devices()])'
|
||||
```
|
||||
|
||||
Match the `--branch` tag to the installed runtime (`dpkg -l | grep librealsense2`).
|
||||
If the build isn't worth it, `CameraDaemon` falls back to `cv2.VideoCapture(0)`
|
||||
automatically — fine for a plain USB webcam, but note a RealSense exposes its
|
||||
*depth* stream at `/dev/video0`, not RGB, so a real USB cam is the cleaner
|
||||
fallback (or pin `SANAD_CAMERA_USB_INDEX`). On x86_64 / Ubuntu 22.04+ desktops,
|
||||
`pip install pyrealsense2` just works.
|
||||
|
||||
|
||||
## Dynamic paths
|
||||
|
||||
Every path is derived at runtime — no hard-coded `/home/...` anywhere.
|
||||
Resolution order for `BASE_DIR` in `config.py`:
|
||||
|
||||
1. `SANAD_PROJECT_ROOT` env var (if set).
|
||||
2. `PROJECT_BASE + PROJECT_NAME` from a `.env` file in `Sanad/` or its parent.
|
||||
3. `Path(__file__).resolve().parent` — auto-detected.
|
||||
|
||||
The project runs unchanged from either layout:
|
||||
- dev: `<anywhere>/Project/Sanad/`
|
||||
- deployed: `/home/unitree/Sanad/`
|
||||
|
||||
|
||||
## Deployment (workstation → robot)
|
||||
|
||||
```bash
|
||||
rsync -av --delete \
|
||||
--exclude=__pycache__ --exclude=logs --exclude=model --exclude=.git \
|
||||
/path/to/Sanad/ \
|
||||
unitree@192.168.123.164:/home/unitree/Sanad/
|
||||
```
|
||||
|
||||
Then on the robot: `Ctrl+C` the running `main.py` and re-run.
|
||||
|
||||
|
||||
## Security
|
||||
|
||||
The dashboard has **no authentication**. Anyone who can reach
|
||||
`http://<robot>:8000` gets full robot control — locomotion, arm, audio, file
|
||||
upload/delete — and, via the **Terminal tab**, an interactive shell as the
|
||||
dashboard's user. Bind it to a **trusted LAN only**; add auth before any wider
|
||||
exposure.
|
||||
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | Fix |
|
||||
|---|---|
|
||||
| `No LowState received in 2s — refusing to replay` | `main.py` was re-executed as both `__main__` and `Project.Sanad.main`, creating two arm instances. Fix lives in the `sys.modules` alias near the top of `main.py`. Restart. |
|
||||
| `G1ArmActionClient not available — skipping` for SDK actions | Same duplicate-init issue as above. |
|
||||
| `No module named 'Project'` in subprocess | Bootstrap preamble in `voice/sanad_voice.py:~30` synthesises the `Project.Sanad` namespace when run as `__main__`. |
|
||||
| Controller moves rejected (409) | The Controller is **disarmed by default** — hit Arm first. Reads + E-STOP are always allowed. |
|
||||
| Arm action refused while "movement armed" | Arm ↔ locomotion are mutually exclusive. Disarm/stop locomotion, then trigger the arm. |
|
||||
| Voice-driven walking does nothing | "Gemini Movement" toggle off, or E-STOP latched. Toggle on; clear E-STOP. Distances are uncalibrated. |
|
||||
| Arm jumps at start of JSONL replay | `SETTLE_HOLD_SEC` (in `config/motion_config.json > arm_controller`) too low — try `0.7` or `1.0`. |
|
||||
| Record playback silent | `audio_mgr.play_wav` only routes to G1 DDS if the Unitree SDK is importable; on desktop it falls back to the PulseAudio sink. |
|
||||
| Live Voice Commands transcript stuck | Deferred trigger was queued but `trigger_enabled` toggle was off. Toggle on — or the pending-trigger poll fires it automatically once enabled. |
|
||||
| Gemini "no audio" on Typed Replay | Non-deterministic; the retry chain in `voice/typed_replay.py:generate_audio` tries three prompt variants. For reliable TTS, use the offline `local_tts` SpeechT5 path. |
|
||||
| Local brain exits immediately | `ollama serve` not running / model not pulled, or weights missing under `model/`. Check `logs/local_subprocess.log`. The Gemini brain is the safe default. |
|
||||
| Recognition tab: "Camera could not start (no backend)" | No camera backend acquired. Check `rs-enumerate-devices` (RealSense at OS level) and `python3 -c 'import pyrealsense2'` in the `gemini_sdk` env. The glibc `ImportError` means the pip wheel is incompatible — see "Camera vision on Jetson" above. |
|
||||
| Camera badge stuck on "reconnecting…" | `CameraDaemon` lost the device and is retrying with exponential backoff. Re-seat the USB 3 cable; check `logs/camera.log` for the USB-2.0 warning. |
|
||||
| Gemini doesn't greet an enrolled face | Face Recognition toggle on? Vision on? (Face rec needs frames.) Check `logs/gemini_brain.log` for `face gallery primed: N person(s)`. Hit "Sync Gallery" to force a re-prime. |
|
||||
| Gemini unaware of motion state | The `motion.action_*` → `send_state` chain only runs when Live Gemini is up. Check `logs/gemini_subprocess.log` and `logs/gemini_brain.log` for `STATE injected:` lines. |
|
||||
|
||||
|
||||
## License / attribution
|
||||
|
||||
Internal project for YS Lootah Technology. Reuses/ports patterns from:
|
||||
- `G1_Lootah/Manual_Recorder/g1_replay_v4_stable.py` (arm replay math)
|
||||
- `SanadVoice/gemini_interact` (arm-phrase dispatch, skill registry)
|
||||
- `SanadVoice/gemini_voice_v2` (local SpeechT5 TTS)
|
||||
- `Project/Marcus` — camera→Gemini stdin-push transport, motion-state
|
||||
injection, camera daemon resilience (auto-reconnect, USB-2.0 warning), the
|
||||
`API/camera_api.py` cache shape (`get_frame_b64` / `get_fresh_frame`), and the
|
||||
confirmation-phrase → locomotion pattern (`movement_dispatch`).
|
||||
- Unitree `unitree_sdk2py` (G1 low-level SDK, `LocoClient`, `G1ArmActionClient`,
|
||||
`AudioClient.PlayStream`).
|
||||
27
config.py
27
config.py
@ -271,7 +271,9 @@ def _resolve_dashboard_host() -> str:
|
||||
|
||||
|
||||
DASHBOARD_HOST = _resolve_dashboard_host()
|
||||
DASHBOARD_PORT = 8000
|
||||
# Canonical SanadV3 port (matches shell_scripts/start_all.sh + docs). The
|
||||
# legacy Sanad ran on :8000; SanadV3 is :8001 to never collide with it.
|
||||
DASHBOARD_PORT = 8001
|
||||
|
||||
# -- Local TTS --
|
||||
LOCAL_TTS_MODEL = "MBZUAI/speecht5_tts_clartts_ar"
|
||||
@ -341,12 +343,35 @@ LIVE_TUNE: dict[str, str] = {
|
||||
# G1 built-in mic — UDP multicast 239.168.123.161:5555.
|
||||
# Requires wake-up conversation mode ON in Unitree app.
|
||||
"SANAD_USE_G1_MIC": "1",
|
||||
|
||||
# ── Recognition (camera vision + face recognition) ──
|
||||
# All of these are BOOT defaults. The runtime source of truth is the
|
||||
# state file data/.recognition_state.json — toggled live from the
|
||||
# Recognition tab and polled by the Gemini child at 1 Hz.
|
||||
"SANAD_VISION_ENABLE": "0",
|
||||
"SANAD_VISION_SEND_HZ": "2",
|
||||
"SANAD_VISION_STALE_MS": "1500",
|
||||
"SANAD_CAMERA_WIDTH": "424",
|
||||
"SANAD_CAMERA_HEIGHT": "240",
|
||||
"SANAD_CAMERA_FPS": "15",
|
||||
"SANAD_CAMERA_JPEG_QUALITY": "70",
|
||||
"SANAD_FACE_RECOGNITION_ENABLE": "0",
|
||||
"SANAD_FACES_DIR": str(DATA_DIR / "faces"),
|
||||
"SANAD_FACES_MAX_SAMPLES": "3",
|
||||
"SANAD_FACES_PRIMER_RESIZE": "256",
|
||||
"SANAD_RECOGNITION_STATE_PATH": str(DATA_DIR / ".recognition_state.json"),
|
||||
"SANAD_RECOGNITION_POLL_S": "1.0",
|
||||
}
|
||||
|
||||
# -- Camera --
|
||||
CAMERA_SERVICE_PORT = 8091
|
||||
DIRECT_CAMERA_URL = f"http://127.0.0.1:{CAMERA_SERVICE_PORT}"
|
||||
|
||||
# -- Navigation (web_nav3 / rosbridge) --
|
||||
WEB_NAV3_URL = os.environ.get("WEB_NAV3_URL", "http://127.0.0.1:8765")
|
||||
ROSBRIDGE_URL = os.environ.get("ROSBRIDGE_URL", "ws://127.0.0.1:9090")
|
||||
NAV_ROBOT_NAME = os.environ.get("NAV_ROBOT_NAME", "sanad")
|
||||
|
||||
# -- DDS / hardware --
|
||||
# Jetson G1 default is eth0 (the robot's internal network).
|
||||
# Override with SANAD_DDS_INTERFACE=lo for desktop/sim development.
|
||||
|
||||
@ -36,12 +36,12 @@
|
||||
|
||||
"gemini_defaults": {
|
||||
"_comment": "Baseline Gemini API config — SINGLE SOURCE OF TRUTH. All voice modules read from here.",
|
||||
"api_key": "AIzaSyDt9Xi83MDZuuPpfwfHyMD92X7ZKdGkqf8",
|
||||
"api_key": "",
|
||||
"model_live": "gemini-2.5-flash-native-audio-preview-12-2025",
|
||||
"model_ws_uri": "wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1alpha.GenerativeService.BidiGenerateContent",
|
||||
"voice_name": "Charon",
|
||||
"ws_timeout_sec": 30,
|
||||
"default_system_prompt": "You are Sanad (Bousandah), a wise and friendly Emirati assistant. Speak strictly in the UAE dialect (Khaleeji). Be helpful, concise, and use local greetings like 'Marhaba' and 'Ya Khoy'."
|
||||
"default_system_prompt": "You are Bousandah, a wise and friendly Emirati assistant. Speak strictly in the UAE dialect (Khaleeji). Be helpful, concise, and use local greetings like 'Marhaba' and 'Ya Khoy'."
|
||||
},
|
||||
|
||||
"g1_hardware": {
|
||||
@ -76,5 +76,26 @@
|
||||
|
||||
"dds": {
|
||||
"network_interface_default": "eth0"
|
||||
},
|
||||
|
||||
"camera": {
|
||||
"_comment": "Recognition tab camera daemon (parent process reads this). width/height/fps/jpeg_quality + the reconnect knobs configure CameraDaemon. Frames are cached in memory and pushed to the Gemini child over its stdin (no file drop). send_hz/stale_ms are read by the Gemini child via SANAD_VISION_SEND_HZ / SANAD_VISION_STALE_MS env vars (LIVE_TUNE).",
|
||||
"width": 424,
|
||||
"height": 240,
|
||||
"fps": 15,
|
||||
"jpeg_quality": 70,
|
||||
"send_hz": 2,
|
||||
"stale_ms": 1500,
|
||||
"stale_threshold_s": 10.0,
|
||||
"reconnect_min_s": 2.0,
|
||||
"reconnect_max_s": 10.0,
|
||||
"capture_timeout_ms": 5000
|
||||
},
|
||||
|
||||
"faces": {
|
||||
"_comment": "Face gallery for Gemini-side recognition. Folder layout: data/faces/face_{id}/{face_1.jpg, ...} + optional meta.json {\"name\": \"...\"}. Gemini does the matching — no local ML model.",
|
||||
"dir_rel": "data/faces",
|
||||
"max_samples_per_face": 3,
|
||||
"primer_resize_long_side": 256
|
||||
}
|
||||
}
|
||||
|
||||
35
config/gemini_config.json
Normal file
35
config/gemini_config.json
Normal file
@ -0,0 +1,35 @@
|
||||
{
|
||||
"_description": "Tunables for gemini/* modules. Loaded via core.config_loader.load('gemini'). API credentials (api_key, model, voice_name) still live in core_config.json > gemini_defaults — single source of truth shared with config.py.",
|
||||
|
||||
"client": {
|
||||
"_comment": "gemini/client.py — short-session WebSocket client used by dashboard /generate + typed replay. default_system_prompt comes from core.gemini_defaults.",
|
||||
"recv_timeout_sec": 30,
|
||||
"reconnect_max_attempts": 3,
|
||||
"reconnect_initial_delay_sec": 1.0,
|
||||
"reconnect_max_delay_sec": 10.0
|
||||
},
|
||||
|
||||
"subprocess": {
|
||||
"_comment": "gemini/subprocess.py — GeminiSubprocess supervisor. Spawns voice/sanad_voice.py as a child, tails stdout for Gemini-specific log markers, pushes camera frames + motion state to the child over its stdin, exposes transcript + state to the dashboard.",
|
||||
"log_tail_size": 2000,
|
||||
"transcript_tail_size": 30,
|
||||
"log_name": "gemini_subprocess",
|
||||
"stop_timeout_sec": 3.0,
|
||||
"terminate_timeout_sec": 2.0,
|
||||
"frame_forward_interval_sec": 0.5,
|
||||
"noisy_prefixes": [
|
||||
"ALSA lib ",
|
||||
"Expression 'alsa_",
|
||||
"Cannot connect to server socket",
|
||||
"jack server is not running"
|
||||
],
|
||||
"noisy_fragments": [
|
||||
"Unknown PCM",
|
||||
"Evaluate error",
|
||||
"snd_pcm_open_noupdate",
|
||||
"PaAlsaStream",
|
||||
"snd_config_evaluate",
|
||||
"snd_func_refer"
|
||||
]
|
||||
}
|
||||
}
|
||||
92
config/local_config.json
Normal file
92
config/local_config.json
Normal file
@ -0,0 +1,92 @@
|
||||
{
|
||||
"_description": "Tunables for local/* — fully on-device voice pipeline (Silero VAD → Whisper → Qwen via llama.cpp → CosyVoice2). Loaded via core.config_loader.load('local').",
|
||||
|
||||
"subprocess": {
|
||||
"_comment": "local/subprocess.py — LocalSubprocess supervisor. Mirrors gemini/subprocess.py. IMPORTANT: python_bin points at the `local` conda env (Python 3.8 + Jetson CUDA torch) so CosyVoice+Whisper run with GPU, while the dashboard/Gemini stack stays in gemini_sdk (Python 3.10).",
|
||||
"python_bin": "/home/unitree/miniconda3/envs/local/bin/python",
|
||||
"log_tail_size": 2000,
|
||||
"transcript_tail_size": 30,
|
||||
"log_name": "local_subprocess",
|
||||
"stop_timeout_sec": 5.0,
|
||||
"terminate_timeout_sec": 3.0,
|
||||
"noisy_prefixes": [
|
||||
"ALSA lib ",
|
||||
"Expression 'alsa_",
|
||||
"Cannot connect to server socket",
|
||||
"jack server is not running"
|
||||
],
|
||||
"noisy_fragments": [
|
||||
"Unknown PCM",
|
||||
"Evaluate error",
|
||||
"snd_pcm_open_noupdate",
|
||||
"PaAlsaStream"
|
||||
]
|
||||
},
|
||||
|
||||
"vad": {
|
||||
"_comment": "Silero VAD — CPU. Emits speech_start / speech_end events.",
|
||||
"sample_rate": 16000,
|
||||
"frame_ms": 32,
|
||||
"threshold": 0.55,
|
||||
"min_silence_ms": 400,
|
||||
"min_speech_ms": 250,
|
||||
"pad_start_ms": 200,
|
||||
"pad_end_ms": 200,
|
||||
"device": "cpu"
|
||||
},
|
||||
|
||||
"stt": {
|
||||
"_comment": "faster-whisper Large V3 Turbo, INT8 on GPU.",
|
||||
"model_name": "large-v3-turbo",
|
||||
"model_subdir": "faster-whisper-large-v3-turbo",
|
||||
"device": "cuda",
|
||||
"compute_type": "int8_float16",
|
||||
"beam_size": 1,
|
||||
"language": null,
|
||||
"vad_filter": false,
|
||||
"no_speech_threshold": 0.6,
|
||||
"min_utterance_chars": 2,
|
||||
"temperature": 0.0
|
||||
},
|
||||
|
||||
"llm": {
|
||||
"_comment": "Qwen 2.5 Instruct via Ollama (default) OR self-managed llama.cpp. Set backend to pick.",
|
||||
"backend": "ollama",
|
||||
|
||||
"_ollama_comment": "Ollama daemon — assumes `ollama serve` is running; `ollama pull qwen2.5:1.5b` to fetch.",
|
||||
"ollama_host": "127.0.0.1",
|
||||
"ollama_port": 11434,
|
||||
"ollama_model": "qwen2.5:1.5b",
|
||||
"ollama_keep_alive": "5m",
|
||||
|
||||
"_llamacpp_comment": "Self-managed llama-server subprocess. Only used when backend='llama_cpp'.",
|
||||
"model_subdir": "qwen2.5-1.5b-instruct-q4_k_m.gguf",
|
||||
"server_binary": "llama-server",
|
||||
"host": "127.0.0.1",
|
||||
"port": 8080,
|
||||
"n_gpu_layers": 99,
|
||||
"ctx_size": 2048,
|
||||
"threads": 4,
|
||||
"startup_timeout_sec": 30,
|
||||
|
||||
"_shared_comment": "Generation params — both backends.",
|
||||
"request_timeout_sec": 30,
|
||||
"max_tokens": 200,
|
||||
"temperature": 0.7,
|
||||
"top_p": 0.9,
|
||||
"stop": ["<|im_end|>", "\n\n\n"],
|
||||
"chunk_delimiters": ".,?!؟،",
|
||||
"chunk_min_chars": 8
|
||||
},
|
||||
|
||||
"tts": {
|
||||
"_comment": "CosyVoice2 0.5B streaming — GPU. Uses a 3s reference WAV for voice cloning.",
|
||||
"model_subdir": "CosyVoice2-0.5B",
|
||||
"reference_wav_subdir": "khaleeji_reference_3s.wav",
|
||||
"reference_prompt": "",
|
||||
"stream_chunk_sec": 0.25,
|
||||
"sample_rate": 16000,
|
||||
"queue_max": 3,
|
||||
"device": "cuda"
|
||||
}
|
||||
}
|
||||
27
config/mask_config.json
Normal file
27
config/mask_config.json
Normal file
@ -0,0 +1,27 @@
|
||||
{
|
||||
"_comment": "Shining LED face mask (BLE). Driven by the FaceController subsystem (face/mask_face.py) which imports the standalone Mask project. Needs an env with bleak + Pillow (g1_env). Free the mask from the phone app before connecting.",
|
||||
"mask_dir": "",
|
||||
"_mask_dir": "Path to the Mask project (flat shiningmask lib). Empty -> auto: <Project>/Mask. Env override: SANAD_MASK_DIR.",
|
||||
"name_prefix": "MASK",
|
||||
"_name_prefix": "BLE scan prefix; the mask advertises e.g. 'MASK-02A711'. Env: SANAD_MASK_NAME_PREFIX.",
|
||||
"address": "",
|
||||
"_address": "Specific BLE MAC to connect to. Empty -> scan by name_prefix. Env: SANAD_MASK_ADDRESS.",
|
||||
"adapter": "",
|
||||
"_adapter": "BlueZ adapter (e.g. 'hci0'). Empty -> default. Env: SANAD_MASK_ADAPTER.",
|
||||
"brightness": 95,
|
||||
"_brightness": "0-128. Keep <=100 to avoid LED flicker (battery-limited).",
|
||||
"fps": 8.0,
|
||||
"_fps": "FaceAnimator (fallback driver) frame rate (PLAY commands/sec).",
|
||||
"lifelike": true,
|
||||
"_lifelike": "Use the LifelikeFace driver (face/face_motion.py): eye saccades, varied blinks, listening/thinking/speaking states, reactions, smooth lip-sync. false -> basic FaceAnimator.",
|
||||
"autostart": true,
|
||||
"_autostart": "Auto-connect + Start face on boot (best-effort, background — never blocks startup). After the one-time frame upload, later boots just connect + animate. false -> connect/start manually from the dashboard.",
|
||||
"connect_timeout": 15.0,
|
||||
"connect_attempts": 5,
|
||||
"eye_color": [0, 230, 255],
|
||||
"_eye_color": "Face eye/iris RGB (baked into the uploaded frames). Default cyan. Set via the dashboard 'Apply colors' (persisted here).",
|
||||
"mouth_color": [255, 50, 50],
|
||||
"_mouth_color": "Face mouth RGB. Default red.",
|
||||
"sclera_color": [255, 255, 255],
|
||||
"_sclera_color": "White-of-the-eye RGB. Default white."
|
||||
}
|
||||
@ -5,6 +5,7 @@
|
||||
"_comment": "motion/arm_controller.py — enable_arm_sdk_index + replay_hz come from core.g1_hardware",
|
||||
"ramp_in_steps": 60,
|
||||
"ramp_out_steps": 180,
|
||||
"settle_hold_sec": 0.5,
|
||||
"watchdog_timeout_sec": 0.25,
|
||||
"watchdog_disable_after_sec": 1.0,
|
||||
"arm_indices_start": 15,
|
||||
@ -12,6 +13,20 @@
|
||||
"jsonl_id_start": 100
|
||||
},
|
||||
|
||||
"loco_controller": {
|
||||
"_comment": "G1_Controller/loco_controller.py — manual locomotion. NIC is shared from the arm's DDS init (config core.dds / SANAD_DDS_INTERFACE), not set here.",
|
||||
"cap_walk": 0.6,
|
||||
"cap_run": 1.2,
|
||||
"lin_step": 0.05,
|
||||
"ang_step": 0.2,
|
||||
"watchdog_timeout_sec": 0.5,
|
||||
"arm_block_window_sec": 1.5,
|
||||
"step_duration_sec": 0.6,
|
||||
"step_speed_frac": 0.5,
|
||||
"loco_timeout_sec": 10.0,
|
||||
"msc_timeout_sec": 5.0
|
||||
},
|
||||
|
||||
"macro_player": {
|
||||
"_comment": "motion/macro_player.py — JSONL playback",
|
||||
"ramp_in_steps": 60,
|
||||
|
||||
@ -50,44 +50,18 @@
|
||||
"dir_relative": "data/recordings"
|
||||
},
|
||||
|
||||
"system_prompt": {
|
||||
"_comment": "Persona filename lives in core.script_files.persona; default text in core.gemini_defaults.default_system_prompt. This section is now metadata-only."
|
||||
},
|
||||
|
||||
"typed_replay": {
|
||||
"_comment": "voice/typed_replay.py — max_text_len comes from dashboard.api_input",
|
||||
"monitor_chunk_size": 512,
|
||||
"monitor_tail_sec": 0.2
|
||||
},
|
||||
|
||||
"live_gemini_subprocess": {
|
||||
"_comment": "voice/live_gemini_subprocess.py — LiveGeminiSubprocess",
|
||||
"log_tail_size": 2000,
|
||||
"transcript_tail_size": 30,
|
||||
"log_name": "live_gemini_subprocess",
|
||||
"stop_timeout_sec": 3.0,
|
||||
"terminate_timeout_sec": 2.0,
|
||||
"noisy_prefixes": [
|
||||
"ALSA lib ",
|
||||
"Expression 'alsa_",
|
||||
"Cannot connect to server socket",
|
||||
"jack server is not running"
|
||||
],
|
||||
"noisy_fragments": [
|
||||
"Unknown PCM",
|
||||
"Evaluate error",
|
||||
"snd_pcm_open_noupdate",
|
||||
"PaAlsaStream",
|
||||
"snd_config_evaluate",
|
||||
"snd_func_refer"
|
||||
]
|
||||
},
|
||||
|
||||
"live_voice_loop": {
|
||||
"_comment": "voice/live_voice_loop.py — arm phrase dispatcher. arm_txt filename comes from core.script_files.arm_phrases",
|
||||
"trigger_log_size": 100,
|
||||
"poll_interval_sec": 0.1,
|
||||
"deferred_default": false
|
||||
"deferred_default": false,
|
||||
"trigger_enabled_default": false
|
||||
},
|
||||
|
||||
"local_tts": {
|
||||
@ -97,27 +71,5 @@
|
||||
"xvector_filename": "arabic_xvector_embedding.pt",
|
||||
"sample_rate": 16000,
|
||||
"channels": 1
|
||||
},
|
||||
|
||||
"gemini_client": {
|
||||
"_comment": "voice/gemini_client.py — default_system_prompt comes from core.gemini_defaults",
|
||||
"recv_timeout_sec": 30,
|
||||
"reconnect_max_attempts": 3,
|
||||
"reconnect_initial_delay_sec": 1.0,
|
||||
"reconnect_max_delay_sec": 10.0
|
||||
},
|
||||
|
||||
"asr_buffer": {
|
||||
"_comment": "text_utils.maybe_trigger_arm state machine defaults",
|
||||
"window_sec": 2.0,
|
||||
"short_token_bonus_sec": 1.0,
|
||||
"join_no_space_maxlen": 2,
|
||||
"max_chars": 120,
|
||||
"stream_max_chars": 80,
|
||||
"trigger_dedup_window_sec": 2.0,
|
||||
"pending_arm_ttl_sec": 6.0,
|
||||
"pending_arm_fallback_sec": 0.65,
|
||||
"dup_call_window_sec": 0.25,
|
||||
"dup_asr_repeat_window_sec": 0.9
|
||||
}
|
||||
}
|
||||
|
||||
@ -46,7 +46,7 @@ class Brain:
|
||||
self._lock = asyncio.Lock()
|
||||
|
||||
# Sub-modules are injected after construction so imports stay lazy.
|
||||
self._voice = None # voice.gemini_client.GeminiVoiceClient
|
||||
self._voice = None # gemini.client.GeminiVoiceClient
|
||||
self._audio_mgr = None # voice.audio_manager.AudioManager
|
||||
self._arm = None # motion.arm_controller.ArmController
|
||||
self._macro_rec = None # motion.macro_recorder.MacroRecorder
|
||||
|
||||
71
core/persona.py
Normal file
71
core/persona.py
Normal file
@ -0,0 +1,71 @@
|
||||
"""Active-persona selection — which script file Gemini loads as its system
|
||||
prompt.
|
||||
|
||||
The operator can keep several persona variants in scripts/ (e.g.
|
||||
``sanad_script.txt``, ``sanad_script_v1.txt``, ``sanad_script_v2.txt``) and pick
|
||||
which one is live. The selection is a single basename stored in
|
||||
``data/active_persona.txt``; the DEFAULT (and reset target) is always the
|
||||
configured persona (``sanad_script.txt``). The Gemini child resolves this at
|
||||
session start, so a new selection takes effect on the next voice (re)connect.
|
||||
|
||||
A missing/blank/stale pointer transparently falls back to the default, so this
|
||||
can never break the voice — worst case it loads ``sanad_script.txt``.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from Project.Sanad.config import DATA_DIR, SCRIPTS_DIR
|
||||
|
||||
ACTIVE_PERSONA_FILE = DATA_DIR / "active_persona.txt"
|
||||
|
||||
|
||||
def default_persona_name() -> str:
|
||||
"""The configured default persona filename (core.script_files.persona)."""
|
||||
try:
|
||||
from Project.Sanad.core.config_loader import section as _section
|
||||
name = (_section("core", "script_files") or {}).get("persona")
|
||||
return (name or "sanad_script.txt").strip() or "sanad_script.txt"
|
||||
except Exception:
|
||||
return "sanad_script.txt"
|
||||
|
||||
|
||||
def active_persona_name() -> str:
|
||||
"""Selected persona basename — the chosen variant if set AND still exists,
|
||||
otherwise the default. Never raises."""
|
||||
default = default_persona_name()
|
||||
try:
|
||||
sel = ACTIVE_PERSONA_FILE.read_text(encoding="utf-8").strip()
|
||||
except Exception:
|
||||
sel = ""
|
||||
if sel:
|
||||
cand = SCRIPTS_DIR / Path(sel).name # basename only — no traversal
|
||||
if cand.is_file():
|
||||
return cand.name
|
||||
return default
|
||||
|
||||
|
||||
def active_persona_path() -> Path:
|
||||
"""Full path to the persona script Gemini should load right now."""
|
||||
return SCRIPTS_DIR / active_persona_name()
|
||||
|
||||
|
||||
def set_active_persona(name: str | None) -> str:
|
||||
"""Persist the selected persona basename. Passing None/"" or the default
|
||||
name clears the pointer (revert to default). Returns the effective active
|
||||
name. Raises FileNotFoundError if a non-default name doesn't exist."""
|
||||
nm = (Path(str(name)).name if name else "").strip()
|
||||
default = default_persona_name()
|
||||
if not nm or nm == default:
|
||||
try:
|
||||
ACTIVE_PERSONA_FILE.unlink()
|
||||
except FileNotFoundError:
|
||||
pass
|
||||
except Exception:
|
||||
pass
|
||||
return default
|
||||
if not (SCRIPTS_DIR / nm).is_file():
|
||||
raise FileNotFoundError(nm)
|
||||
DATA_DIR.mkdir(parents=True, exist_ok=True)
|
||||
ACTIVE_PERSONA_FILE.write_text(nm, encoding="utf-8")
|
||||
return nm
|
||||
@ -50,9 +50,16 @@ _REST_ROUTES: list[tuple[str, str, str]] = [
|
||||
("live_voice", "/api/live-voice", "live-voice"),
|
||||
("live_subprocess", "/api/live-subprocess", "live-subprocess"),
|
||||
("typed_replay", "/api/typed-replay", "typed-replay"),
|
||||
("recognition", "/api/recognition", "recognition"),
|
||||
("zones", "/api/zones", "zones"),
|
||||
("temp_monitor", "/api/temp", "temperature"),
|
||||
("controller", "/api/controller", "controller"),
|
||||
("mask", "/api/mask", "mask"),
|
||||
("mask_social", "/api/mask", "mask-social"),
|
||||
("navigation", "/api/nav", "navigation"),
|
||||
]
|
||||
|
||||
_WS_ROUTES: list[str] = ["log_stream"]
|
||||
_WS_ROUTES: list[str] = ["log_stream", "motor_temps", "terminal"]
|
||||
|
||||
_loaded_routes: list[str] = []
|
||||
_failed_routes: dict[str, str] = {}
|
||||
@ -109,7 +116,13 @@ async def root():
|
||||
if index.exists():
|
||||
from fastapi.responses import HTMLResponse
|
||||
try:
|
||||
return HTMLResponse(index.read_text(encoding="utf-8"))
|
||||
# no-store so the browser always re-fetches the dashboard HTML/JS
|
||||
# after a deploy — otherwise stale cached JS keeps calling old
|
||||
# endpoints (e.g. /nav/* instead of /api/nav/*) and 404s.
|
||||
return HTMLResponse(
|
||||
index.read_text(encoding="utf-8"),
|
||||
headers={"Cache-Control": "no-store, must-revalidate"},
|
||||
)
|
||||
except OSError as exc:
|
||||
return {"error": f"Could not read index.html: {exc}"}
|
||||
return {
|
||||
|
||||
66
dashboard/routes/_arbiter.py
Normal file
66
dashboard/routes/_arbiter.py
Normal file
@ -0,0 +1,66 @@
|
||||
"""In-process arbitration between Nav2 (web_nav3) and the manual LocoController.
|
||||
|
||||
Both stacks can drive the G1's legs via different command paths:
|
||||
- Nav2 (web_nav3) publishes cmd_vel from a navigation goal/mission.
|
||||
- LocoController issues LocoClient.Move()/step() from the Controller tab and
|
||||
Gemini movement dispatch.
|
||||
|
||||
The documented hazard is "two stacks must never both drive the legs at once".
|
||||
This module is a tiny thread-safe gate that lets ONE commander own the legs at a
|
||||
time. controller.py sets loco_active for arm/move/step and refuses when nav is
|
||||
active; navigation.py sets nav_active for goto/missions/run and refuses when loco
|
||||
is active. The E-STOP / cancel paths clear the relevant flag.
|
||||
|
||||
Pure in-process state (no DDS, no HTTP) — both routers share this single module
|
||||
instance, so the flags are coherent across the dashboard process.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import threading
|
||||
|
||||
_lock = threading.Lock()
|
||||
_loco_active = False
|
||||
_nav_active = False
|
||||
|
||||
|
||||
def loco_active() -> bool:
|
||||
with _lock:
|
||||
return _loco_active
|
||||
|
||||
|
||||
def nav_active() -> bool:
|
||||
with _lock:
|
||||
return _nav_active
|
||||
|
||||
|
||||
def acquire_loco() -> bool:
|
||||
"""Claim the legs for manual loco. Returns False if Nav2 holds them."""
|
||||
global _loco_active
|
||||
with _lock:
|
||||
if _nav_active:
|
||||
return False
|
||||
_loco_active = True
|
||||
return True
|
||||
|
||||
|
||||
def release_loco() -> None:
|
||||
global _loco_active
|
||||
with _lock:
|
||||
_loco_active = False
|
||||
|
||||
|
||||
def acquire_nav() -> bool:
|
||||
"""Claim the legs for Nav2. Returns False if manual loco holds them."""
|
||||
global _nav_active
|
||||
with _lock:
|
||||
if _loco_active:
|
||||
return False
|
||||
_nav_active = True
|
||||
return True
|
||||
|
||||
|
||||
def release_nav() -> None:
|
||||
global _nav_active
|
||||
with _lock:
|
||||
_nav_active = False
|
||||
@ -160,7 +160,13 @@ async def audio_status():
|
||||
"g1_speaker_muted": g1_muted,
|
||||
"g1_current_volume": _g1_current_volume,
|
||||
"g1_user_volume": _g1_user_volume,
|
||||
"g1_available": _g1_audio_client is not None or (_g1_init_error == ""),
|
||||
# Only report available once an AudioClient has actually been
|
||||
# built — reporting True before any init attempt made the UI
|
||||
# advertise G1 speaker controls that then 503 on first use.
|
||||
# `g1_init_error` surfaces *why* it's unavailable (or "" if
|
||||
# init was never attempted yet).
|
||||
"g1_available": _g1_audio_client is not None,
|
||||
"g1_init_error": _g1_init_error,
|
||||
"sink": sink,
|
||||
"source": source,
|
||||
"current": cur,
|
||||
@ -312,7 +318,9 @@ async def get_g1_volume():
|
||||
"""
|
||||
def _do():
|
||||
return {
|
||||
"available": _g1_audio_client is not None or (_g1_init_error == ""),
|
||||
# True only after an AudioClient was actually constructed —
|
||||
# `init_error` (below) explains an unavailable/never-tried state.
|
||||
"available": _g1_audio_client is not None,
|
||||
"current_volume": _g1_current_volume,
|
||||
"user_volume": _g1_user_volume,
|
||||
"muted": _g1_current_volume == 0,
|
||||
@ -343,35 +351,48 @@ async def set_g1_volume(payload: G1VolumePayload):
|
||||
if not 0 <= level <= 100:
|
||||
raise HTTPException(400, "level must be 0..100")
|
||||
|
||||
# 1) G1 chest speaker (DDS) — best-effort so it works even when an
|
||||
# external sink (JBL) is the active output.
|
||||
code = None
|
||||
client = _get_g1_audio_client()
|
||||
if client is None:
|
||||
raise HTTPException(
|
||||
503,
|
||||
f"G1 AudioClient unavailable: {_g1_init_error or 'unknown'}",
|
||||
)
|
||||
try:
|
||||
with _g1_audio_lock:
|
||||
code = client.SetVolume(level)
|
||||
_g1_current_volume = level
|
||||
if level > 0:
|
||||
# Only update the "preferred unmuted" level when the
|
||||
# user is setting a non-zero volume. Setting 0 is a
|
||||
# mute, which shouldn't overwrite their preference.
|
||||
_g1_user_volume = level
|
||||
except Exception as exc:
|
||||
raise HTTPException(500, f"SetVolume failed: {exc}")
|
||||
if client is not None:
|
||||
try:
|
||||
with _g1_audio_lock:
|
||||
code = client.SetVolume(level)
|
||||
_g1_current_volume = level
|
||||
except Exception as exc:
|
||||
log.warning("G1 SetVolume failed: %s", exc)
|
||||
if level > 0:
|
||||
_g1_user_volume = level
|
||||
|
||||
# 2) The ACTIVE profile's PulseAudio sink (JBL / Anker / …). Target the
|
||||
# RESOLVED sink from the saved selection, NOT @DEFAULT_SINK@ — the PA
|
||||
# default can be a different sink (e.g. the chest platform-sound) even
|
||||
# when the JBL is the selected output, so @DEFAULT_SINK@ would move the
|
||||
# wrong sink and the slider would appear to do nothing on the JBL.
|
||||
pa_applied = False
|
||||
try:
|
||||
sink = (ad.load_state() or {}).get("sink") or "@DEFAULT_SINK@"
|
||||
_pactl(["set-sink-volume", sink, "%d%%" % level])
|
||||
if level > 0:
|
||||
_pactl(["set-sink-mute", sink, "0"])
|
||||
pa_applied = True
|
||||
except Exception as exc:
|
||||
log.warning("PA set-sink-volume failed: %s", exc)
|
||||
|
||||
if client is None and not pa_applied:
|
||||
raise HTTPException(503, "No speaker available (G1 + PulseAudio both failed)")
|
||||
|
||||
# Persist the user's preferred level (not the current) so a
|
||||
# subsequent mute-then-restart restores to the preferred level
|
||||
_save_persisted_g1_volume(_g1_user_volume)
|
||||
log.info("G1 volume → %d (user_pref=%d, rc=%s)",
|
||||
level, _g1_user_volume, code)
|
||||
log.info("volume → %d (g1_rc=%s, pa=%s, user_pref=%d)",
|
||||
level, code, pa_applied, _g1_user_volume)
|
||||
return {
|
||||
"ok": True,
|
||||
"current_volume": level,
|
||||
"user_volume": _g1_user_volume,
|
||||
"muted": level == 0,
|
||||
"return_code": code,
|
||||
"pa_applied": pa_applied,
|
||||
"persisted": True,
|
||||
}
|
||||
return await asyncio.to_thread(_do)
|
||||
@ -471,5 +492,474 @@ async def apply_audio():
|
||||
audio_mgr.refresh_devices()
|
||||
except Exception:
|
||||
pass
|
||||
# Hot-swap the live Gemini voice to the selected profile too, so picking
|
||||
# a device (e.g. the JBL) moves BOTH record playback AND the live voice
|
||||
# to it — without dropping the session. Best-effort; no-op if not running.
|
||||
try:
|
||||
from Project.Sanad.main import live_sub
|
||||
pid = (ad.load_state() or {}).get("profile_id")
|
||||
if (pid and live_sub is not None and hasattr(live_sub, "send_profile")
|
||||
and hasattr(live_sub, "is_running") and live_sub.is_running()):
|
||||
live_sub.send_profile(pid, reason="dashboard audio Apply")
|
||||
except Exception:
|
||||
pass
|
||||
# Restore the user's SAVED volume to the selected sink (USB/BT speakers
|
||||
# like the JBL otherwise come back at a low PulseAudio default). Use the
|
||||
# saved level, NOT a forced 100%, so the slider/sink keep the user's
|
||||
# choice across selects + restarts. Target the resolved sink.
|
||||
try:
|
||||
sink = (ad.load_state() or {}).get("sink") or "@DEFAULT_SINK@"
|
||||
_pactl(["set-sink-volume", sink, "%d%%" % _g1_user_volume])
|
||||
if _g1_user_volume > 0:
|
||||
_pactl(["set-sink-mute", sink, "0"])
|
||||
except Exception:
|
||||
pass
|
||||
return result
|
||||
return await asyncio.to_thread(_do)
|
||||
|
||||
|
||||
# ─────────────────────── Reset endpoints (Pulse + USB) ───────────────────────
|
||||
#
|
||||
# Two distinct recovery paths for the dashboard's audio panel:
|
||||
#
|
||||
# POST /api/audio/reset — SOFT: restart pulseaudio / pipewire-pulse.
|
||||
# Fixes Pulse-side state (stuck profile, lost default sink, crashed
|
||||
# module). Cannot recover a kernel-side missing USB capture descriptor
|
||||
# — snd-usb-audio parses those at probe time and Pulse can't influence
|
||||
# that. Use for "devices look weird" failures.
|
||||
#
|
||||
# POST /api/audio/usb-reset — HARD: unbind+rebind snd-usb-audio scoped
|
||||
# to the Anker VID:PID. Forces snd-usb-audio to re-parse UAC1
|
||||
# descriptors → input profile reappears even after the firmware/USB
|
||||
# handshake dropped it. Use for "Anker mic missing from pactl" — the
|
||||
# symptom soft-reset cannot fix.
|
||||
#
|
||||
# Both gate with module-level locks (no concurrent reset), refuse while Live
|
||||
# Gemini is running or a record is mid-playback, and return structured
|
||||
# before/after diagnostics so the dashboard can show meaningful toasts.
|
||||
|
||||
_RESET_LOCK = threading.Lock()
|
||||
_USB_RESET_LOCK = threading.Lock()
|
||||
|
||||
# Anker PowerConf A3321 — used both for VID:PID matching in sysfs and for
|
||||
# logging. Change here if you add support for a different USB conference
|
||||
# device (Hollyland etc).
|
||||
_USB_RESET_TARGETS = (
|
||||
{"vid": "291a", "pid": "3301", "label": "Anker PowerConf"},
|
||||
)
|
||||
|
||||
|
||||
def _refuse_if_busy() -> None:
|
||||
"""Raise HTTPException(409) if Live Gemini is active or a record is playing.
|
||||
|
||||
Used by both reset endpoints — a userspace audio restart mid-stream
|
||||
leaves the active session in a broken state (PortAudio handle pointing
|
||||
at a dead Pulse, in-flight write() raises, etc.). Cheaper to refuse
|
||||
than to recover.
|
||||
"""
|
||||
try:
|
||||
from Project.Sanad.main import live_sub
|
||||
except Exception:
|
||||
live_sub = None
|
||||
if live_sub is not None:
|
||||
try:
|
||||
st = live_sub.status() or {}
|
||||
except Exception:
|
||||
st = {}
|
||||
state = (st.get("state") or "").lower()
|
||||
if st.get("running") or state not in ("", "stopped", "error"):
|
||||
raise HTTPException(
|
||||
409, f"Stop Live Gemini before resetting audio (state={state or '?'}).",
|
||||
)
|
||||
|
||||
try:
|
||||
from Project.Sanad.main import audio_mgr
|
||||
except Exception:
|
||||
audio_mgr = None
|
||||
if audio_mgr is not None and hasattr(audio_mgr, "playback_status"):
|
||||
try:
|
||||
ps = audio_mgr.playback_status() or {}
|
||||
if ps.get("playing"):
|
||||
raise HTTPException(
|
||||
409, "Stop the active playback before resetting audio.",
|
||||
)
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
def _detect_pa_flavour() -> str:
|
||||
"""Return 'pipewire' if pipewire-pulse is the active daemon, else 'pulse'."""
|
||||
try:
|
||||
r = subprocess.run(
|
||||
["pgrep", "-x", "pipewire-pulse"],
|
||||
check=False, capture_output=True, text=True, timeout=1.0,
|
||||
)
|
||||
if r.returncode == 0 and (r.stdout or "").strip():
|
||||
return "pipewire"
|
||||
except (FileNotFoundError, subprocess.SubprocessError):
|
||||
pass
|
||||
return "pulse"
|
||||
|
||||
|
||||
def _kill_audio_daemon(flavour: str) -> dict:
|
||||
"""Issue the restart command for the detected daemon. Non-zero exit is a
|
||||
soft warning (some installs return 1 when there's no daemon to kill)."""
|
||||
if flavour == "pipewire":
|
||||
cmd = ["systemctl", "--user", "restart", "pipewire-pulse.service"]
|
||||
else:
|
||||
cmd = ["pulseaudio", "-k"]
|
||||
try:
|
||||
r = subprocess.run(cmd, check=False, capture_output=True,
|
||||
text=True, timeout=5.0)
|
||||
info = {"cmd": " ".join(cmd), "returncode": r.returncode,
|
||||
"stderr": (r.stderr or "").strip()[:300]}
|
||||
if r.returncode != 0:
|
||||
log.warning("audio reset: %s exited %d (%s)",
|
||||
cmd[0], r.returncode, info["stderr"])
|
||||
return info
|
||||
except FileNotFoundError as exc:
|
||||
return {"cmd": " ".join(cmd), "returncode": -1,
|
||||
"stderr": f"binary missing: {exc}"}
|
||||
except subprocess.TimeoutExpired:
|
||||
return {"cmd": " ".join(cmd), "returncode": -1,
|
||||
"stderr": "timeout (>5s)"}
|
||||
|
||||
|
||||
def _wait_for_pactl(deadline_s: float = 5.0, interval_s: float = 0.2) -> bool:
|
||||
"""Poll `pactl info` until it returns 0 or the deadline expires."""
|
||||
import time as _time
|
||||
end = _time.monotonic() + deadline_s
|
||||
while _time.monotonic() < end:
|
||||
if ad.pactl_available():
|
||||
return True
|
||||
_time.sleep(interval_s)
|
||||
return False
|
||||
|
||||
|
||||
@router.post("/reset")
|
||||
async def reset_audio_subsystem():
|
||||
"""SOFT reset — restart pulseaudio/pipewire-pulse and re-resolve devices.
|
||||
|
||||
Use when devices look stuck, pactl is unavailable, or the wrong sink
|
||||
is being selected. **Does NOT recover a kernel-side missing USB capture
|
||||
descriptor** — for that symptom use /api/audio/usb-reset.
|
||||
"""
|
||||
if os.geteuid() == 0:
|
||||
raise HTTPException(
|
||||
403, "Refusing to reset audio as root — Sanad must run as the "
|
||||
"unitree user so the per-user PulseAudio session is reachable.",
|
||||
)
|
||||
if not _RESET_LOCK.acquire(blocking=False):
|
||||
raise HTTPException(429, "Reset already in progress.")
|
||||
try:
|
||||
_refuse_if_busy()
|
||||
log.info(
|
||||
"audio reset requested (uid=%s PULSE_RUNTIME_PATH=%s XDG_RUNTIME_DIR=%s)",
|
||||
os.geteuid(),
|
||||
os.environ.get("PULSE_RUNTIME_PATH") or "-",
|
||||
os.environ.get("XDG_RUNTIME_DIR") or "-",
|
||||
)
|
||||
try:
|
||||
from Project.Sanad.main import audio_mgr
|
||||
except Exception:
|
||||
audio_mgr = None
|
||||
|
||||
def _do() -> dict:
|
||||
before = {"pactl_available": ad.pactl_available(),
|
||||
"selection": ad.current_selection()}
|
||||
|
||||
# Quiesce AudioManager so the next play_wav rebinds cleanly.
|
||||
pya_closed = False
|
||||
if audio_mgr is not None:
|
||||
play_lock = getattr(audio_mgr, "play_lock", None)
|
||||
acquired = False
|
||||
if play_lock is not None:
|
||||
acquired = play_lock.acquire(timeout=2.0)
|
||||
try:
|
||||
try:
|
||||
audio_mgr.close()
|
||||
pya_closed = True
|
||||
except Exception as exc:
|
||||
log.warning("audio reset: audio_mgr.close failed: %s", exc)
|
||||
finally:
|
||||
if acquired and play_lock is not None:
|
||||
play_lock.release()
|
||||
|
||||
flavour = _detect_pa_flavour()
|
||||
kill_info = _kill_audio_daemon(flavour)
|
||||
came_back = _wait_for_pactl(deadline_s=5.0)
|
||||
if not came_back and flavour == "pulse":
|
||||
# autospawn may be disabled — try an explicit start.
|
||||
try:
|
||||
subprocess.run(["pulseaudio", "--start"], check=False,
|
||||
capture_output=True, text=True, timeout=3.0)
|
||||
except (FileNotFoundError, subprocess.SubprocessError) as exc:
|
||||
log.warning("audio reset: pulseaudio --start failed: %s", exc)
|
||||
came_back = _wait_for_pactl(deadline_s=2.0)
|
||||
if not came_back:
|
||||
raise HTTPException(500, {
|
||||
"error": "audio daemon did not return within ~7s",
|
||||
"flavour": flavour, "kill": kill_info,
|
||||
})
|
||||
|
||||
apply_result: dict = {}
|
||||
try:
|
||||
apply_result = ad.apply_current_selection() or {}
|
||||
except Exception as exc:
|
||||
log.warning("audio reset: apply_current_selection failed: %s", exc)
|
||||
apply_result = {"error": str(exc)}
|
||||
|
||||
if audio_mgr is not None:
|
||||
try:
|
||||
import pyaudio
|
||||
audio_mgr.pya = pyaudio.PyAudio()
|
||||
audio_mgr.refresh_devices()
|
||||
except Exception as exc:
|
||||
log.error("audio reset: PyAudio re-init failed: %s", exc)
|
||||
raise HTTPException(
|
||||
500, f"PortAudio re-init failed after daemon restart: {exc}")
|
||||
|
||||
after_sel = ad.current_selection() or {}
|
||||
detected = ad.detect_plugged_profiles() or []
|
||||
after = {
|
||||
"pactl_available": ad.pactl_available(),
|
||||
"selection": after_sel,
|
||||
"detected_profiles": [p.get("profile", {}).get("id") for p in detected],
|
||||
}
|
||||
return {
|
||||
"ok": True, "best_effort": True, "flavour": flavour,
|
||||
"kill": kill_info, "pya_reinitialized": pya_closed,
|
||||
"apply_result": apply_result,
|
||||
"input_recovered": bool(after_sel.get("source")),
|
||||
"output_recovered": bool(after_sel.get("sink")),
|
||||
"before": before, "after": after,
|
||||
"hint": ("Soft reset only fixes Pulse-side state. If "
|
||||
"input_recovered is False, try POST /api/audio/usb-reset "
|
||||
"or physically replug the dongle."),
|
||||
}
|
||||
return await asyncio.to_thread(_do)
|
||||
finally:
|
||||
_RESET_LOCK.release()
|
||||
|
||||
|
||||
def _find_usb_devices_by_vid_pid(vid: str, pid: str) -> list[str]:
|
||||
"""Return sysfs bus-id strings (e.g. '1-3') for every USB device whose
|
||||
idVendor/idProduct match. Empty list when nothing matches.
|
||||
|
||||
We read /sys/bus/usb/devices/* — every USB *device* (not interface) has
|
||||
idVendor/idProduct files. Interfaces (paths with a colon, e.g. '1-3:1.1')
|
||||
do not, so they're naturally skipped.
|
||||
"""
|
||||
import glob
|
||||
hits: list[str] = []
|
||||
for path in glob.glob("/sys/bus/usb/devices/*"):
|
||||
name = os.path.basename(path)
|
||||
if ":" in name:
|
||||
continue
|
||||
try:
|
||||
with open(os.path.join(path, "idVendor")) as f:
|
||||
v = f.read().strip().lower()
|
||||
with open(os.path.join(path, "idProduct")) as f:
|
||||
p = f.read().strip().lower()
|
||||
except OSError:
|
||||
continue
|
||||
if v == vid.lower() and p == pid.lower():
|
||||
hits.append(name)
|
||||
return hits
|
||||
|
||||
|
||||
def _snd_usb_interfaces_for_device(bus_id: str) -> list[str]:
|
||||
"""For USB device `bus_id` (e.g. '1-3'), return all interface names that
|
||||
are currently bound to the snd-usb-audio driver (e.g. ['1-3:1.0']).
|
||||
|
||||
Used so we unbind ONLY the audio interfaces and don't touch HID / HUB
|
||||
interfaces on the same composite device.
|
||||
"""
|
||||
import glob
|
||||
bound: list[str] = []
|
||||
base = f"/sys/bus/usb/devices/{bus_id}"
|
||||
for iface in glob.glob(f"{base}/{bus_id}:*"):
|
||||
driver_link = os.path.join(iface, "driver")
|
||||
if not os.path.islink(driver_link):
|
||||
continue
|
||||
try:
|
||||
driver = os.path.basename(os.readlink(driver_link))
|
||||
except OSError:
|
||||
continue
|
||||
if driver == "snd-usb-audio":
|
||||
bound.append(os.path.basename(iface))
|
||||
return bound
|
||||
|
||||
|
||||
def _write_sysfs(path: str, value: str) -> tuple[bool, str]:
|
||||
"""Write `value` to a sysfs file. Returns (success, error_message).
|
||||
|
||||
Writes to /sys/bus/usb/drivers/snd-usb-audio/{bind,unbind} usually
|
||||
require root. If permission denied, the caller should fall back to
|
||||
invoking shell_scripts/reset_anker_usb.sh via sudo (one-time sudoers
|
||||
setup documented in that script's header).
|
||||
"""
|
||||
try:
|
||||
with open(path, "w") as f:
|
||||
f.write(value)
|
||||
return True, ""
|
||||
except PermissionError as exc:
|
||||
return False, f"permission denied: {path} ({exc})"
|
||||
except OSError as exc:
|
||||
return False, f"write failed: {path} ({exc})"
|
||||
|
||||
|
||||
@router.post("/usb-reset")
|
||||
async def usb_reset_anker():
|
||||
"""HARD reset — unbind+rebind snd-usb-audio for the Anker (VID:PID
|
||||
291a:3301). Forces the kernel to re-parse the USB Audio Class
|
||||
descriptors, which is the only way to recover a missing capture profile
|
||||
on this Jetson without a physical replug.
|
||||
|
||||
Tries two paths:
|
||||
1. Direct sysfs write (no sudo) — works if a udev rule has set
|
||||
`audio` group ownership / world-write on the snd-usb-audio bind
|
||||
files, or if Sanad runs as root (it shouldn't).
|
||||
2. Fallback to `sudo shell_scripts/reset_anker_usb.sh` — works after
|
||||
a one-time sudoers entry; see that script's header for setup.
|
||||
|
||||
Refuses while Live Gemini or a record playback is in flight (same
|
||||
guard as the soft reset).
|
||||
"""
|
||||
if not _USB_RESET_LOCK.acquire(blocking=False):
|
||||
raise HTTPException(429, "USB reset already in progress.")
|
||||
try:
|
||||
_refuse_if_busy()
|
||||
|
||||
# Find candidate Anker USB devices currently enumerated.
|
||||
candidates: list[dict] = []
|
||||
for tgt in _USB_RESET_TARGETS:
|
||||
for bus_id in _find_usb_devices_by_vid_pid(tgt["vid"], tgt["pid"]):
|
||||
candidates.append({"bus_id": bus_id, **tgt})
|
||||
if not candidates:
|
||||
wanted = ", ".join(
|
||||
"{}:{}".format(t["vid"], t["pid"]) for t in _USB_RESET_TARGETS
|
||||
)
|
||||
raise HTTPException(
|
||||
404,
|
||||
f"No matching USB device found (looked for {wanted}). "
|
||||
"Plug the Anker dongle and try again.",
|
||||
)
|
||||
|
||||
log.info("usb reset: candidates=%s", candidates)
|
||||
|
||||
def _do() -> dict:
|
||||
before_detected = [
|
||||
p.get("profile", {}).get("id")
|
||||
for p in (ad.detect_plugged_profiles() or [])
|
||||
]
|
||||
results: list[dict] = []
|
||||
for cand in candidates:
|
||||
bus = cand["bus_id"]
|
||||
ifaces = _snd_usb_interfaces_for_device(bus)
|
||||
attempt = {"bus_id": bus, "label": cand["label"],
|
||||
"snd_interfaces": ifaces, "method": None,
|
||||
"ok": False, "error": ""}
|
||||
if not ifaces:
|
||||
attempt["error"] = ("no snd-usb-audio interfaces bound "
|
||||
"to this device — already unbound or "
|
||||
"kernel didn't claim it")
|
||||
results.append(attempt)
|
||||
continue
|
||||
|
||||
# ─── Path 1: direct sysfs write ───
|
||||
unbind_path = "/sys/bus/usb/drivers/snd-usb-audio/unbind"
|
||||
bind_path = "/sys/bus/usb/drivers/snd-usb-audio/bind"
|
||||
direct_ok = True
|
||||
direct_err = ""
|
||||
for iface in ifaces:
|
||||
ok, err = _write_sysfs(unbind_path, iface)
|
||||
if not ok:
|
||||
direct_ok = False
|
||||
direct_err = err
|
||||
break
|
||||
if direct_ok:
|
||||
import time as _time
|
||||
_time.sleep(0.5)
|
||||
for iface in ifaces:
|
||||
ok, err = _write_sysfs(bind_path, iface)
|
||||
if not ok:
|
||||
direct_ok = False
|
||||
direct_err = err
|
||||
break
|
||||
if direct_ok:
|
||||
attempt.update({"method": "direct-sysfs", "ok": True})
|
||||
results.append(attempt)
|
||||
continue
|
||||
|
||||
# ─── Path 2: sudo helper script ───
|
||||
from pathlib import Path as _Path
|
||||
helper = (_Path(__file__).resolve().parent.parent.parent
|
||||
/ "shell_scripts" / "reset_anker_usb.sh")
|
||||
if not helper.exists():
|
||||
attempt.update({"method": "direct-sysfs",
|
||||
"error": f"{direct_err}; helper not present "
|
||||
f"at {helper}"})
|
||||
results.append(attempt)
|
||||
continue
|
||||
try:
|
||||
r = subprocess.run(
|
||||
["sudo", "-n", str(helper), bus],
|
||||
check=False, capture_output=True, text=True, timeout=10.0,
|
||||
)
|
||||
attempt["method"] = "sudo-helper"
|
||||
if r.returncode == 0:
|
||||
attempt["ok"] = True
|
||||
else:
|
||||
attempt["error"] = (
|
||||
f"sudo helper exited {r.returncode}: "
|
||||
f"{(r.stderr or r.stdout or '').strip()[:300]}"
|
||||
)
|
||||
except subprocess.TimeoutExpired:
|
||||
attempt["error"] = "sudo helper timed out (>10s)"
|
||||
except FileNotFoundError as exc:
|
||||
attempt["error"] = f"sudo not available: {exc}"
|
||||
results.append(attempt)
|
||||
|
||||
# Settle, then re-detect
|
||||
import time as _time
|
||||
_time.sleep(1.0)
|
||||
try:
|
||||
ad.apply_current_selection()
|
||||
except Exception:
|
||||
pass
|
||||
try:
|
||||
from Project.Sanad.main import audio_mgr
|
||||
if audio_mgr is not None and hasattr(audio_mgr, "refresh_devices"):
|
||||
audio_mgr.refresh_devices()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
after_detected = [
|
||||
p.get("profile", {}).get("id")
|
||||
for p in (ad.detect_plugged_profiles() or [])
|
||||
]
|
||||
any_ok = any(r.get("ok") for r in results)
|
||||
mic_now = any(
|
||||
"anker" in (p.get("profile", {}).get("id") or "").lower()
|
||||
for p in (ad.detect_plugged_profiles() or [])
|
||||
)
|
||||
|
||||
return {
|
||||
"ok": any_ok,
|
||||
"candidates": results,
|
||||
"before_detected_profiles": before_detected,
|
||||
"after_detected_profiles": after_detected,
|
||||
"input_recovered": mic_now,
|
||||
"hint": (
|
||||
"If ok is False, the unbind/rebind path needs sudo. "
|
||||
"Run `bash shell_scripts/reset_anker_usb.sh --setup-sudoers` "
|
||||
"once on the robot to install the sudoers entry, then retry."
|
||||
) if not any_ok else None,
|
||||
}
|
||||
|
||||
return await asyncio.to_thread(_do)
|
||||
finally:
|
||||
_USB_RESET_LOCK.release()
|
||||
|
||||
338
dashboard/routes/controller.py
Normal file
338
dashboard/routes/controller.py
Normal file
@ -0,0 +1,338 @@
|
||||
"""Controller tab — manual dashboard locomotion control (N2 Phase 1/2).
|
||||
|
||||
Routes live under /api/controller. All WRITE actions (move / step / postures /
|
||||
modes / MotionSwitcher) require the in-memory "Enable movement" arm flag and
|
||||
return 409 when disarmed. Reads (/status, /joints, /msc, /status/summary),
|
||||
E-STOP and the arm toggle are ALWAYS available.
|
||||
|
||||
`/status/summary` is the aggregate the dashboard polls for the global subsystem
|
||||
status strip (Camera / Face / Place / Movement). It is kept under /api/controller
|
||||
(final path /api/controller/status/summary) so no second router is needed; note
|
||||
/api/status (no /summary) is already used by the SPA, so the suffix matters.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Query
|
||||
from pydantic import BaseModel
|
||||
|
||||
from Project.Sanad.config import BASE_DIR
|
||||
from Project.Sanad.core.logger import get_logger
|
||||
from Project.Sanad.vision import recognition_state
|
||||
|
||||
from Project.Sanad.dashboard.routes import _arbiter
|
||||
|
||||
log = get_logger("controller_routes")
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
STATE_PATH = BASE_DIR / "data" / ".recognition_state.json"
|
||||
|
||||
|
||||
# ── lazy subsystem accessors ────────────────────────────────
|
||||
|
||||
def _get_loco():
|
||||
try:
|
||||
from Project.Sanad.main import loco_controller # type: ignore
|
||||
return loco_controller
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
def _get_camera():
|
||||
try:
|
||||
from Project.Sanad.main import camera # type: ignore
|
||||
return camera
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
def _get_live_sub():
|
||||
try:
|
||||
from Project.Sanad.main import live_sub # type: ignore
|
||||
return live_sub
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
def _get_dispatch():
|
||||
try:
|
||||
from Project.Sanad.main import movement_dispatch # type: ignore
|
||||
return movement_dispatch
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
def _require_loco():
|
||||
lc = _get_loco()
|
||||
if lc is None:
|
||||
raise HTTPException(503, "Locomotion controller subsystem unavailable.")
|
||||
return lc
|
||||
|
||||
|
||||
def _require_armed(lc):
|
||||
if not lc.is_armed():
|
||||
raise HTTPException(409, "Movement is disarmed. Enable movement first.")
|
||||
|
||||
|
||||
def _claim_loco():
|
||||
"""Arbitration gate: refuse a leg command while a Nav2 goal owns the legs."""
|
||||
if not _arbiter.acquire_loco():
|
||||
raise HTTPException(
|
||||
409, "Navigation (Nav2) is active. Cancel the nav goal before manual movement."
|
||||
)
|
||||
|
||||
|
||||
def _cancel_nav():
|
||||
"""Cancel any in-flight Nav2 goal and clear the nav arbitration flag.
|
||||
|
||||
Used by E-STOP so the global stop halts the legs no matter which stack is
|
||||
driving them. Calls the nav client in-process (no HTTP self-call); never
|
||||
raises into the caller.
|
||||
"""
|
||||
try:
|
||||
from Project.Sanad.dashboard.routes.navigation import _CLIENT as _nav_client
|
||||
if _nav_client is not None:
|
||||
_nav_client.cancel()
|
||||
except Exception:
|
||||
log.exception("estop nav cancel failed")
|
||||
finally:
|
||||
_arbiter.release_nav()
|
||||
|
||||
|
||||
# ── reads ───────────────────────────────────────────────────
|
||||
|
||||
@router.get("/status")
|
||||
async def get_status():
|
||||
lc = _require_loco()
|
||||
return await asyncio.to_thread(lc.status)
|
||||
|
||||
|
||||
@router.get("/joints")
|
||||
async def get_joints():
|
||||
lc = _require_loco()
|
||||
return await asyncio.to_thread(lc.joints)
|
||||
|
||||
|
||||
@router.get("/msc")
|
||||
async def get_msc():
|
||||
lc = _require_loco()
|
||||
return await asyncio.to_thread(lc.msc_check)
|
||||
|
||||
|
||||
# ── arm flag / E-STOP (always available) ────────────────────
|
||||
|
||||
@router.post("/arm")
|
||||
async def set_arm(on: bool = Query(...)):
|
||||
lc = _require_loco()
|
||||
if on:
|
||||
# Refuse to arm manual loco while Nav2 owns the legs.
|
||||
_claim_loco()
|
||||
try:
|
||||
res = await asyncio.to_thread(lc.arm_movement)
|
||||
except Exception:
|
||||
_arbiter.release_loco()
|
||||
raise
|
||||
return res
|
||||
res = await asyncio.to_thread(lc.disarm_movement)
|
||||
_arbiter.release_loco()
|
||||
return res
|
||||
|
||||
|
||||
@router.post("/gemini-movement")
|
||||
async def set_gemini_movement(on: bool = Query(...)):
|
||||
"""Enable / disable Gemini voice-driven locomotion (N2 Phase 3 gate).
|
||||
|
||||
Writes recognition_state.movement_enabled — SEPARATE from the manual arm
|
||||
flag. The Gemini child announces the toggle (spoken), and the parent
|
||||
MovementDispatcher starts/stops acting on confirmation phrases. Default OFF.
|
||||
"""
|
||||
st = await asyncio.to_thread(recognition_state.mutate, STATE_PATH,
|
||||
movement_enabled=bool(on))
|
||||
# Enabling Gemini movement also clears any E-STOP latch on the dispatcher.
|
||||
if on:
|
||||
md = _get_dispatch()
|
||||
if md is not None:
|
||||
try:
|
||||
md.clear_estop()
|
||||
except Exception:
|
||||
log.exception("clear_estop failed")
|
||||
log.info("gemini-movement %s", "ON" if on else "OFF")
|
||||
return {"ok": True, "movement_enabled": st.movement_enabled}
|
||||
|
||||
|
||||
@router.post("/estop")
|
||||
async def estop():
|
||||
lc = _require_loco()
|
||||
res = await asyncio.to_thread(lc.estop)
|
||||
# Full stop: drop the manual arm flag AND latch the voice dispatcher off, so
|
||||
# no source (teleop, step, or voice dispatch) can keep driving the robot. The
|
||||
# dispatcher latch is used instead of flipping movement_enabled so the Gemini
|
||||
# child does not deliver a spoken "movement disabled" line during an E-STOP.
|
||||
try:
|
||||
await asyncio.to_thread(lc.disarm_movement)
|
||||
except Exception:
|
||||
log.exception("estop disarm failed")
|
||||
md = _get_dispatch()
|
||||
if md is not None:
|
||||
try:
|
||||
md.emergency_stop()
|
||||
except Exception:
|
||||
log.exception("estop dispatcher latch failed")
|
||||
# Cancel any in-flight Nav2 goal too: the legs have exactly one commander,
|
||||
# and an E-STOP must halt the legs whether loco or Nav2 is driving them.
|
||||
await asyncio.to_thread(_cancel_nav)
|
||||
_arbiter.release_loco()
|
||||
return {"ok": True, **res}
|
||||
|
||||
|
||||
@router.post("/stop")
|
||||
async def stop():
|
||||
lc = _require_loco()
|
||||
# Allowed even when disarmed — StopMove is always safe.
|
||||
res = await asyncio.to_thread(lc.stop_move)
|
||||
return res
|
||||
|
||||
|
||||
# ── movement (armed) ────────────────────────────────────────
|
||||
|
||||
class MoveBody(BaseModel):
|
||||
vx: float = 0.0
|
||||
vy: float = 0.0
|
||||
vyaw: float = 0.0
|
||||
run: bool = False
|
||||
|
||||
|
||||
@router.post("/move")
|
||||
async def move(body: MoveBody):
|
||||
lc = _require_loco()
|
||||
_require_armed(lc)
|
||||
_claim_loco()
|
||||
return await asyncio.to_thread(lc.move, body.vx, body.vy, body.vyaw, body.run)
|
||||
|
||||
|
||||
@router.post("/step")
|
||||
async def step(dir: str = Query(...)):
|
||||
lc = _require_loco()
|
||||
_require_armed(lc)
|
||||
_claim_loco()
|
||||
res = await asyncio.to_thread(lc.step, dir)
|
||||
if not res.get("ok"):
|
||||
raise HTTPException(400, res.get("reason", "step failed"))
|
||||
return res
|
||||
|
||||
|
||||
# ── modes / postures (armed) ────────────────────────────────
|
||||
|
||||
@router.post("/mode/prep")
|
||||
async def mode_prep():
|
||||
lc = _require_loco()
|
||||
_require_armed(lc)
|
||||
return await asyncio.to_thread(lc.prep_mode)
|
||||
|
||||
|
||||
@router.post("/mode/ready")
|
||||
async def mode_ready():
|
||||
lc = _require_loco()
|
||||
_require_armed(lc)
|
||||
return await asyncio.to_thread(lc.ready_start_mode)
|
||||
|
||||
|
||||
@router.post("/posture/{name}")
|
||||
async def posture(name: str):
|
||||
lc = _require_loco()
|
||||
_require_armed(lc)
|
||||
res = await asyncio.to_thread(lc.posture, name)
|
||||
if not res.get("ok") and res.get("reason"):
|
||||
raise HTTPException(400, res["reason"])
|
||||
return res
|
||||
|
||||
|
||||
@router.post("/balance")
|
||||
async def balance(mode: int = Query(...)):
|
||||
lc = _require_loco()
|
||||
_require_armed(lc)
|
||||
return await asyncio.to_thread(lc.set_balance_mode, mode)
|
||||
|
||||
|
||||
@router.post("/height")
|
||||
async def height(h: float = Query(...)):
|
||||
lc = _require_loco()
|
||||
_require_armed(lc)
|
||||
return await asyncio.to_thread(lc.set_stand_height, h)
|
||||
|
||||
|
||||
# ── MotionSwitcher / reconnect (armed) ──────────────────────
|
||||
|
||||
@router.post("/msc/select-ai")
|
||||
async def msc_select_ai():
|
||||
lc = _require_loco()
|
||||
_require_armed(lc)
|
||||
return await asyncio.to_thread(lc.msc_select_ai)
|
||||
|
||||
|
||||
@router.post("/msc/release")
|
||||
async def msc_release():
|
||||
lc = _require_loco()
|
||||
_require_armed(lc)
|
||||
return await asyncio.to_thread(lc.msc_release)
|
||||
|
||||
|
||||
@router.post("/reconnect")
|
||||
async def reconnect():
|
||||
lc = _require_loco()
|
||||
_require_armed(lc)
|
||||
return await asyncio.to_thread(lc.reconnect)
|
||||
|
||||
|
||||
# ── aggregate subsystem summary (always available) ──────────
|
||||
|
||||
@router.get("/status/summary")
|
||||
async def status_summary():
|
||||
"""Live on/off state for the header status strip. Never raises."""
|
||||
try:
|
||||
st = recognition_state.read(STATE_PATH)
|
||||
except Exception:
|
||||
st = recognition_state.RecognitionState()
|
||||
|
||||
cam = _get_camera()
|
||||
camera_running = False
|
||||
try:
|
||||
camera_running = bool(cam is not None and cam.is_running())
|
||||
except Exception:
|
||||
camera_running = False
|
||||
|
||||
lc = _get_loco()
|
||||
movement_armed = False
|
||||
try:
|
||||
movement_armed = bool(lc is not None and lc.is_armed())
|
||||
except Exception:
|
||||
movement_armed = False
|
||||
|
||||
sub = _get_live_sub()
|
||||
gemini_running = False
|
||||
try:
|
||||
runner = getattr(sub, "is_running", None)
|
||||
gemini_running = bool(callable(runner) and runner())
|
||||
except Exception:
|
||||
gemini_running = False
|
||||
|
||||
# Effective Gemini-movement = the file flag AND not latched off by an E-STOP.
|
||||
md = _get_dispatch()
|
||||
estopped = False
|
||||
try:
|
||||
estopped = bool(md is not None and md.is_estopped())
|
||||
except Exception:
|
||||
estopped = False
|
||||
|
||||
return {
|
||||
"vision_enabled": st.vision_enabled,
|
||||
"camera_running": camera_running,
|
||||
"face_rec_enabled": st.face_rec_enabled,
|
||||
"zone_rec_enabled": st.zone_rec_enabled,
|
||||
"movement_armed": movement_armed,
|
||||
"gemini_movement_enabled": st.movement_enabled and not estopped,
|
||||
"gemini_running": gemini_running,
|
||||
}
|
||||
@ -4,10 +4,15 @@ from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
|
||||
from fastapi import APIRouter, HTTPException
|
||||
from fastapi import APIRouter, HTTPException, Query
|
||||
|
||||
from Project.Sanad.config import BASE_DIR
|
||||
from Project.Sanad.vision import recognition_state
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
_STATE_PATH = BASE_DIR / "data" / ".recognition_state.json"
|
||||
|
||||
|
||||
def _sub_or_503():
|
||||
from Project.Sanad.main import live_sub
|
||||
@ -19,9 +24,21 @@ def _sub_or_503():
|
||||
@router.get("/status")
|
||||
async def subprocess_status():
|
||||
from Project.Sanad.main import live_sub
|
||||
# record_enabled is a live flag (recognition_state) the panel toggle drives;
|
||||
# surface it so the UI shows the current state even before a session starts.
|
||||
rec = bool(recognition_state.read(_STATE_PATH).record_enabled)
|
||||
if live_sub is None:
|
||||
return {"available": False, "state": "unavailable"}
|
||||
return live_sub.status()
|
||||
return {"available": False, "state": "unavailable", "record_enabled": rec}
|
||||
return {**live_sub.status(), "record_enabled": rec}
|
||||
|
||||
|
||||
@router.post("/record")
|
||||
async def set_record(on: bool = Query(...)):
|
||||
"""Toggle auto-recording of conversation turns to data/recordings/. Takes
|
||||
effect live (the voice child syncs its recorder) — no session restart."""
|
||||
st = await asyncio.to_thread(
|
||||
recognition_state.mutate, _STATE_PATH, record_enabled=bool(on))
|
||||
return {"ok": True, "record_enabled": st.record_enabled}
|
||||
|
||||
|
||||
@router.post("/start")
|
||||
|
||||
@ -1,14 +1,15 @@
|
||||
"""Live Voice Commands — voice-to-arm phrase trigger dispatcher.
|
||||
|
||||
Listens to LiveGeminiSubprocess user transcripts, matches against
|
||||
Listens to GeminiSubprocess user transcripts, matches against
|
||||
sanad_arm.txt phrases, and fires ARM.trigger_action_by_id.
|
||||
|
||||
Endpoints:
|
||||
POST /start begin polling transcripts
|
||||
POST /stop stop polling
|
||||
POST /deferred-mode?enabled toggle instant vs deferred trigger
|
||||
GET /status running, last heard, last action, etc.
|
||||
GET /triggers arm trigger history (log)
|
||||
POST /start begin polling transcripts
|
||||
POST /stop stop polling
|
||||
POST /deferred-mode?enabled toggle instant vs deferred trigger
|
||||
POST /trigger-enabled?enabled master gate — allow arm actions or not
|
||||
GET /status running, last heard, last action, etc.
|
||||
GET /triggers arm trigger history (log)
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@ -54,6 +55,14 @@ async def set_deferred(enabled: bool):
|
||||
return {"ok": True, "deferred_mode": loop.deferred_mode}
|
||||
|
||||
|
||||
@router.post("/trigger-enabled")
|
||||
async def set_trigger_enabled(enabled: bool):
|
||||
"""Master gate for voice → arm triggering. Default OFF."""
|
||||
loop = _loop()
|
||||
loop.set_trigger_enabled(enabled)
|
||||
return {"ok": True, "trigger_enabled": loop.trigger_enabled}
|
||||
|
||||
|
||||
@router.get("/triggers")
|
||||
async def triggers():
|
||||
loop = _loop()
|
||||
|
||||
@ -2,16 +2,46 @@
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from pathlib import Path
|
||||
|
||||
from fastapi import APIRouter, HTTPException
|
||||
from pydantic import BaseModel
|
||||
|
||||
from Project.Sanad.config import AUDIO_RECORDINGS_DIR, MOTIONS_DIR
|
||||
from Project.Sanad.core.logger import get_logger
|
||||
|
||||
log = get_logger("macros_route")
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
def _block_if_movement_armed():
|
||||
"""409 when locomotion movement is armed — arm motion is mutually exclusive
|
||||
with walking. The arm controller's motion-block is the safety net."""
|
||||
try:
|
||||
from Project.Sanad.main import loco_controller # type: ignore
|
||||
armed = loco_controller is not None and loco_controller.is_armed()
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception:
|
||||
return
|
||||
if armed:
|
||||
raise HTTPException(
|
||||
409, "Arm actions are disabled while movement is enabled. "
|
||||
"Disable movement in the Controller tab first.")
|
||||
|
||||
|
||||
class MacroName(BaseModel):
|
||||
name: str
|
||||
|
||||
|
||||
class ComboPlayPayload(BaseModel):
|
||||
audio_file: str = "" # filename under data/audio/ (or empty for none)
|
||||
motion_file: str = "" # DEPRECATED — use action_id. Still accepted for bare JSONL by filename.
|
||||
action_id: int | None = None # arm_controller action id (SDK built-in OR JSONL) — preferred
|
||||
speed: float = 1.0
|
||||
|
||||
|
||||
@router.get("/")
|
||||
async def list_macros():
|
||||
from Project.Sanad.main import macro_play
|
||||
@ -49,6 +79,7 @@ async def stop_recording():
|
||||
@router.post("/play")
|
||||
async def play_macro(payload: MacroName):
|
||||
from Project.Sanad.main import brain
|
||||
_block_if_movement_armed()
|
||||
return await brain.play_macro(payload.name)
|
||||
|
||||
|
||||
@ -58,3 +89,150 @@ async def stop_macro():
|
||||
if macro_play:
|
||||
macro_play.stop()
|
||||
return {"ok": True}
|
||||
|
||||
|
||||
# ─── Ad-hoc audio + motion combined playback ─────────────────────────
|
||||
# List the two catalogues so the dashboard can populate dropdowns, then
|
||||
# play the chosen pair in parallel (asyncio.gather) — same scheme the
|
||||
# Brain uses for `parallel`-mode skills, but ad-hoc instead of predefined.
|
||||
|
||||
@router.get("/audio-files")
|
||||
async def list_audio_files():
|
||||
"""Enumerate playable audio files under data/audio/."""
|
||||
AUDIO_RECORDINGS_DIR.mkdir(parents=True, exist_ok=True)
|
||||
files = []
|
||||
for p in sorted(AUDIO_RECORDINGS_DIR.glob("*.wav")):
|
||||
try:
|
||||
files.append({
|
||||
"name": p.name,
|
||||
"size_kb": round(p.stat().st_size / 1024, 1),
|
||||
})
|
||||
except OSError:
|
||||
continue
|
||||
return {"files": files, "dir": str(AUDIO_RECORDINGS_DIR)}
|
||||
|
||||
|
||||
@router.get("/motion-files")
|
||||
async def list_motion_files():
|
||||
"""Enumerate playable .jsonl motions under data/motions/ (thin wrapper
|
||||
so the Macro Recorder dropdown doesn't have to call the replay route)."""
|
||||
MOTIONS_DIR.mkdir(parents=True, exist_ok=True)
|
||||
files = []
|
||||
for p in sorted(MOTIONS_DIR.glob("*.jsonl")):
|
||||
try:
|
||||
files.append({
|
||||
"name": p.name,
|
||||
"size_kb": round(p.stat().st_size / 1024, 1),
|
||||
})
|
||||
except OSError:
|
||||
continue
|
||||
return {"files": files, "dir": str(MOTIONS_DIR)}
|
||||
|
||||
|
||||
@router.post("/stop-combined")
|
||||
async def stop_combined():
|
||||
"""Immediately stop any in-flight combined playback.
|
||||
|
||||
- `arm.cancel()` — breaks the replay loop and triggers the smooth
|
||||
return-to-home ramp (see `_return_home` in arm_controller.py).
|
||||
- `audio_mgr.stop_playback()` — sends AUDIO_STOP_PLAY to the G1
|
||||
chest speaker via DDS.
|
||||
Both run unconditionally so Stop works even if only one side was
|
||||
actually playing.
|
||||
"""
|
||||
from Project.Sanad.main import audio_mgr, arm
|
||||
result = {"motion_stopped": False, "audio_stopped": False}
|
||||
if arm is not None:
|
||||
try:
|
||||
arm.cancel()
|
||||
result["motion_stopped"] = True
|
||||
except Exception as exc:
|
||||
log.warning("stop-combined: arm.cancel failed: %s", exc)
|
||||
result["motion_error"] = str(exc)
|
||||
if audio_mgr is not None:
|
||||
try:
|
||||
audio_mgr.stop_playback()
|
||||
result["audio_stopped"] = True
|
||||
except Exception as exc:
|
||||
log.warning("stop-combined: audio stop failed: %s", exc)
|
||||
result["audio_error"] = str(exc)
|
||||
return {"ok": True, **result}
|
||||
|
||||
|
||||
@router.post("/play-combined")
|
||||
async def play_combined(payload: ComboPlayPayload):
|
||||
"""Fire a user-picked audio clip and arm action in parallel.
|
||||
|
||||
Motion dispatch is via `arm.trigger_by_id(action_id)` which handles
|
||||
BOTH SDK built-in actions (shake_hand, wave, …) and recorded JSONL
|
||||
replays. Audio goes through `audio_mgr.play_wav` (routed to the G1
|
||||
chest speaker via DDS). Either side may be omitted.
|
||||
"""
|
||||
from Project.Sanad.main import audio_mgr, arm
|
||||
|
||||
has_audio = bool(payload.audio_file)
|
||||
has_motion = payload.action_id is not None or bool(payload.motion_file)
|
||||
if not has_audio and not has_motion:
|
||||
raise HTTPException(400, "pick at least one of audio_file / action_id / motion_file")
|
||||
if has_motion:
|
||||
_block_if_movement_armed() # audio-only combos still allowed while armed
|
||||
|
||||
tasks = []
|
||||
result: dict = {
|
||||
"audio_file": payload.audio_file,
|
||||
"action_id": payload.action_id,
|
||||
"motion_file": payload.motion_file,
|
||||
}
|
||||
|
||||
if has_audio:
|
||||
if audio_mgr is None:
|
||||
raise HTTPException(503, "AudioManager not available")
|
||||
audio_path = (AUDIO_RECORDINGS_DIR / payload.audio_file).resolve()
|
||||
try:
|
||||
audio_path.relative_to(AUDIO_RECORDINGS_DIR.resolve())
|
||||
except ValueError:
|
||||
raise HTTPException(400, "audio_file path traversal denied")
|
||||
if not audio_path.exists():
|
||||
raise HTTPException(404, f"audio not found: {payload.audio_file}")
|
||||
|
||||
async def _play_audio():
|
||||
try:
|
||||
await asyncio.to_thread(audio_mgr.play_wav, audio_path)
|
||||
result["audio_played"] = audio_path.name
|
||||
except Exception as exc:
|
||||
log.exception("combined play: audio failed")
|
||||
result["audio_error"] = str(exc)
|
||||
tasks.append(_play_audio())
|
||||
|
||||
if has_motion:
|
||||
if arm is None:
|
||||
raise HTTPException(503, "ArmController not available")
|
||||
|
||||
async def _play_motion():
|
||||
try:
|
||||
if payload.action_id is not None:
|
||||
# SDK built-in OR JSONL — arm.trigger_by_id handles both
|
||||
await asyncio.to_thread(arm.trigger_by_id,
|
||||
int(payload.action_id),
|
||||
payload.speed)
|
||||
result["motion_played"] = f"action_id={payload.action_id}"
|
||||
else:
|
||||
# Legacy path: bare JSONL filename
|
||||
motion_path = (MOTIONS_DIR / payload.motion_file).resolve()
|
||||
try:
|
||||
motion_path.relative_to(MOTIONS_DIR.resolve())
|
||||
except ValueError:
|
||||
result["motion_error"] = "motion_file path traversal denied"
|
||||
return
|
||||
if not motion_path.exists():
|
||||
result["motion_error"] = f"motion not found: {payload.motion_file}"
|
||||
return
|
||||
await asyncio.to_thread(arm.replay_file, str(motion_path), payload.speed)
|
||||
result["motion_played"] = motion_path.name
|
||||
except Exception as exc:
|
||||
log.exception("combined play: motion failed")
|
||||
result["motion_error"] = str(exc)
|
||||
tasks.append(_play_motion())
|
||||
|
||||
await asyncio.gather(*tasks)
|
||||
return {"ok": True, **result}
|
||||
|
||||
179
dashboard/routes/mask.py
Normal file
179
dashboard/routes/mask.py
Normal file
@ -0,0 +1,179 @@
|
||||
"""Mask Face tab — Shining LED face mask control (BLE).
|
||||
|
||||
Routes live under /api/mask. Backed by the FaceController subsystem
|
||||
(face/mask_face.py), which owns a dedicated asyncio loop + BLE connection to the
|
||||
standalone Mask project's `shiningmask` library.
|
||||
|
||||
Every handler is failure-safe: if the subsystem or its library is unavailable it
|
||||
returns 503 (GET /status returns a degraded body) rather than crash the
|
||||
dashboard. FaceController raises RuntimeError for "not connected" / "face not
|
||||
started"; those map to 409. Blocking BLE calls run in a thread pool so the event
|
||||
loop stays responsive.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from typing import List, Optional
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Query
|
||||
from pydantic import BaseModel
|
||||
|
||||
from Project.Sanad.core.logger import get_logger
|
||||
|
||||
log = get_logger("mask_routes")
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
# ── lazy subsystem accessor ─────────────────────────────────
|
||||
|
||||
def _get_face():
|
||||
try:
|
||||
from Project.Sanad.main import mask_face # type: ignore
|
||||
return mask_face
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
def _require():
|
||||
mf = _get_face()
|
||||
if mf is None:
|
||||
raise HTTPException(503, "Mask face subsystem unavailable.")
|
||||
return mf
|
||||
|
||||
|
||||
def _run(fn, *args, **kwargs):
|
||||
"""Call a FaceController method, mapping its errors to HTTP status codes."""
|
||||
try:
|
||||
return fn(*args, **kwargs)
|
||||
except HTTPException:
|
||||
raise
|
||||
except RuntimeError as exc:
|
||||
raise HTTPException(409, str(exc))
|
||||
except Exception as exc: # noqa: BLE001
|
||||
log.exception("mask operation failed")
|
||||
raise HTTPException(500, str(exc))
|
||||
|
||||
|
||||
# ── status ──────────────────────────────────────────────────
|
||||
|
||||
@router.get("/status")
|
||||
async def status():
|
||||
"""Never raises — returns a degraded body if the subsystem is missing."""
|
||||
mf = _get_face()
|
||||
if mf is None:
|
||||
return {"available": False, "connected": False, "lib_available": False,
|
||||
"last_error": "mask face subsystem not constructed"}
|
||||
s = await asyncio.to_thread(mf.status)
|
||||
s["available"] = True
|
||||
return s
|
||||
|
||||
|
||||
# ── connection ──────────────────────────────────────────────
|
||||
|
||||
@router.post("/connect")
|
||||
async def connect(timeout: Optional[float] = Query(None),
|
||||
attempts: Optional[int] = Query(None)):
|
||||
mf = _require()
|
||||
return await asyncio.to_thread(_run, mf.connect, timeout, attempts)
|
||||
|
||||
|
||||
@router.post("/disconnect")
|
||||
async def disconnect():
|
||||
mf = _require()
|
||||
return await asyncio.to_thread(_run, mf.disconnect)
|
||||
|
||||
|
||||
# ── simple commands ─────────────────────────────────────────
|
||||
|
||||
@router.post("/brightness")
|
||||
async def brightness(level: int = Query(..., ge=0, le=255)):
|
||||
mf = _require()
|
||||
return await asyncio.to_thread(_run, mf.set_brightness, level)
|
||||
|
||||
|
||||
class TextBody(BaseModel):
|
||||
text: str = ""
|
||||
color: List[int] = [255, 255, 255]
|
||||
mode: Optional[int] = None
|
||||
bg: Optional[List[int]] = None # background RGB (None -> black)
|
||||
speed: Optional[int] = None # scroll speed 0-255 (None -> firmware default)
|
||||
|
||||
|
||||
@router.post("/text")
|
||||
async def text(body: TextBody):
|
||||
mf = _require()
|
||||
bg = tuple(body.bg) if body.bg else None
|
||||
return await asyncio.to_thread(_run, mf.set_text, body.text, tuple(body.color),
|
||||
body.mode, bg, body.speed)
|
||||
|
||||
|
||||
@router.post("/image")
|
||||
async def image(id: int = Query(...)):
|
||||
mf = _require()
|
||||
return await asyncio.to_thread(_run, mf.show_image, id)
|
||||
|
||||
|
||||
@router.post("/animation")
|
||||
async def animation(id: int = Query(...)):
|
||||
mf = _require()
|
||||
return await asyncio.to_thread(_run, mf.play_animation, id)
|
||||
|
||||
|
||||
@router.post("/clear")
|
||||
async def clear():
|
||||
mf = _require()
|
||||
return await asyncio.to_thread(_run, mf.clear_diy)
|
||||
|
||||
|
||||
# ── animated face ───────────────────────────────────────────
|
||||
|
||||
@router.post("/face/start")
|
||||
async def face_start(reload: bool = Query(False)):
|
||||
mf = _require()
|
||||
return await asyncio.to_thread(_run, mf.face_start, reload)
|
||||
|
||||
|
||||
@router.post("/face/stop")
|
||||
async def face_stop():
|
||||
mf = _require()
|
||||
return await asyncio.to_thread(_run, mf.face_stop)
|
||||
|
||||
|
||||
@router.post("/face/return")
|
||||
async def face_return():
|
||||
"""Resume the live animated face after a text/image/animation override."""
|
||||
mf = _require()
|
||||
return await asyncio.to_thread(_run, mf.return_face)
|
||||
|
||||
|
||||
class FaceColorBody(BaseModel):
|
||||
eye: Optional[List[int]] = None # eye/iris RGB
|
||||
mouth: Optional[List[int]] = None # mouth RGB
|
||||
sclera: Optional[List[int]] = None # white-of-the-eye RGB
|
||||
|
||||
|
||||
@router.post("/face/color")
|
||||
async def face_color(body: FaceColorBody):
|
||||
"""Recolor the animated face (re-uploads the frame set if the face is live)."""
|
||||
mf = _require()
|
||||
return await asyncio.to_thread(_run, mf.set_face_color, body.eye, body.mouth, body.sclera)
|
||||
|
||||
|
||||
@router.post("/speaking")
|
||||
async def speaking(on: bool = Query(...)):
|
||||
mf = _require()
|
||||
return await asyncio.to_thread(_run, mf.set_speaking, on)
|
||||
|
||||
|
||||
@router.post("/mouth")
|
||||
async def mouth(level: int = Query(..., ge=0, le=3)):
|
||||
mf = _require()
|
||||
return await asyncio.to_thread(_run, mf.set_mouth, level)
|
||||
|
||||
|
||||
@router.post("/expression/{name}")
|
||||
async def expression(name: str):
|
||||
mf = _require()
|
||||
return await asyncio.to_thread(_run, mf.show_expression, name)
|
||||
395
dashboard/routes/mask_social.py
Normal file
395
dashboard/routes/mask_social.py
Normal file
@ -0,0 +1,395 @@
|
||||
"""Social-media / QR display on the LED mask.
|
||||
|
||||
Renders a QR code (for a preset Instagram account) or an uploaded image onto the
|
||||
mask's 46x58 display and holds it via the FaceController's reserved scratch slot
|
||||
until the animated face is resumed. The shared helper :func:`show_social_on_mask`
|
||||
is also called from the Gemini ``[[SHOW:account]]`` relay wired in ``main.py``.
|
||||
|
||||
Routes (under /api/mask):
|
||||
POST /social/{account} -> show a preset Instagram QR
|
||||
POST /qr -> upload an image (QR or any picture) + show it
|
||||
POST /face/resume -> stop showing the scratch image, return to the face
|
||||
GET /social -> list the preset accounts
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import io
|
||||
import logging
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import re
|
||||
|
||||
from fastapi import APIRouter, File, HTTPException, Query, UploadFile
|
||||
from fastapi.responses import FileResponse
|
||||
|
||||
log = logging.getLogger("sanad.mask_social")
|
||||
router = APIRouter() # prefix "/api/mask" supplied by dashboard/app.py _REST_ROUTES
|
||||
|
||||
# Preset Instagram accounts the mask can show as a QR. The mask is a low-res
|
||||
# 46x58 panel, so a full-URL QR is dense; the black margin acts as the quiet
|
||||
# zone and we scale modules crisply (NEAREST) to give it the best chance.
|
||||
SOCIAL = {
|
||||
"bu_sunaidah": {"handle": "@bu.sunaidah",
|
||||
"url": "https://instagram.com/bu.sunaidah",
|
||||
"short": "da.gd/VMkH8J"}, # -> instagram.com/bu.sunaidah (v1 QR)
|
||||
"yslootahtech": {"handle": "@yslootahtech",
|
||||
"url": "https://instagram.com/yslootahtech",
|
||||
"short": "da.gd/Qr8RO"}, # -> instagram.com/yslootahtech (v1 QR)
|
||||
}
|
||||
|
||||
|
||||
def _ensure_mask_path() -> None:
|
||||
"""Make the flat Mask lib (colorface) importable from this route — using the
|
||||
SAME dir the FaceController resolved (the Mask lib lives outside the repo)."""
|
||||
d = os.environ.get("SANAD_MASK_DIR")
|
||||
if not d:
|
||||
try:
|
||||
from Project.Sanad.main import mask_face as _mf # type: ignore
|
||||
d = getattr(_mf, "mask_dir", None)
|
||||
except Exception:
|
||||
d = None
|
||||
if not d:
|
||||
d = str(Path(__file__).resolve().parents[2] / "Mask")
|
||||
if d and d not in sys.path:
|
||||
sys.path.insert(0, d)
|
||||
|
||||
|
||||
def _get_face():
|
||||
from Project.Sanad.main import mask_face # type: ignore
|
||||
if mask_face is None:
|
||||
raise HTTPException(status_code=503, detail="mask face unavailable")
|
||||
return mask_face
|
||||
|
||||
|
||||
_EYE_BAND = 16 # top rows reserved for the cyan eyes; the code sits below them
|
||||
|
||||
|
||||
def _compose_under_eyes(inner) -> bytes:
|
||||
"""Draw two cyan eyes across the top and place ``inner`` (a QR / image) in the
|
||||
area BELOW them, then encode for the mask. Keeps the panel looking like a face
|
||||
with a code under the eyes instead of a full-screen QR."""
|
||||
_ensure_mask_path()
|
||||
import colorface as cf
|
||||
from PIL import Image, ImageDraw
|
||||
W, H = cf.DISPLAY_W, cf.DISPLAY_H
|
||||
inner = inner.convert("RGB")
|
||||
iw, ih = inner.size
|
||||
# keep the code a small badge under the eyes (~70% of the space below them)
|
||||
target = max(20, int(min(W, H - _EYE_BAND - 1) * 0.72))
|
||||
if iw <= target and ih <= target:
|
||||
s = max(1, min(target // iw, target // ih)) # crisp integer up-scale (QR)
|
||||
nw, nh = iw * s, ih * s
|
||||
else:
|
||||
s = min(target / iw, target / ih) # scale big images down
|
||||
nw, nh = max(1, int(iw * s)), max(1, int(ih * s))
|
||||
inner = inner.resize((nw, nh), Image.NEAREST)
|
||||
canvas = Image.new("RGB", (W, H), (0, 0, 0))
|
||||
g = ImageDraw.Draw(canvas)
|
||||
eye = cf.DEFAULT_EYE
|
||||
for cx in (W // 2 - 10, W // 2 + 10): # two eyes at the top
|
||||
g.ellipse([cx - 5, 3, cx + 5, 13], fill=(255, 255, 255))
|
||||
g.ellipse([cx - 3, 5, cx + 3, 11], fill=eye)
|
||||
g.ellipse([cx - 1, 7, cx + 1, 10], fill=(0, 0, 0))
|
||||
x = (W - nw) // 2
|
||||
y = _EYE_BAND + (H - _EYE_BAND - nh) // 2
|
||||
canvas.paste(inner, (max(0, x), max(_EYE_BAND, y)))
|
||||
return cf.encode(canvas)
|
||||
|
||||
|
||||
def _qr_bytes(url: str) -> bytes:
|
||||
"""Render a QR for ``url`` FULL-SCREEN with the largest crisp (integer) module
|
||||
size the 46-wide panel allows — the only way it has any chance of scanning.
|
||||
Only a ~version-1 QR (<=17 chars) reaches ~2 px/module; longer data is denser
|
||||
and won't scan. Returns (bytes, qr_version)."""
|
||||
_ensure_mask_path()
|
||||
import qrcode
|
||||
from PIL import Image
|
||||
import colorface as cf
|
||||
W, H = cf.DISPLAY_W, cf.DISPLAY_H
|
||||
qr = qrcode.QRCode(error_correction=qrcode.constants.ERROR_CORRECT_L,
|
||||
box_size=1, border=1)
|
||||
qr.add_data(url)
|
||||
qr.make(fit=True)
|
||||
q = qr.make_image(fill_color=(255, 255, 255),
|
||||
back_color=(0, 0, 0)).convert("RGB")
|
||||
scale = max(1, min(W, H) // max(1, q.width)) # largest integer that fits
|
||||
if scale > 1:
|
||||
q = q.resize((q.width * scale, q.width * scale), Image.NEAREST)
|
||||
canvas = Image.new("RGB", (W, H), (0, 0, 0))
|
||||
canvas.paste(q, ((W - q.width) // 2, (H - q.height) // 2))
|
||||
return cf.encode(canvas)
|
||||
|
||||
|
||||
def _image_bytes(img) -> bytes:
|
||||
"""Show an uploaded QR/image FULL-SCREEN, crisp (NEAREST) — best effort."""
|
||||
_ensure_mask_path()
|
||||
import colorface as cf
|
||||
from PIL import Image
|
||||
W, H = cf.DISPLAY_W, cf.DISPLAY_H
|
||||
s = min(W, H)
|
||||
img = img.convert("RGB").resize((s, s), Image.NEAREST)
|
||||
canvas = Image.new("RGB", (W, H), (0, 0, 0))
|
||||
canvas.paste(img, ((W - s) // 2, (H - s) // 2))
|
||||
return cf.encode(canvas)
|
||||
|
||||
|
||||
def show_social_on_mask(account: str) -> dict:
|
||||
"""Show the account's **scannable** QR on the mask — a version-1 QR made from
|
||||
a short (da.gd) link that redirects to the Instagram profile. Shared by the
|
||||
dashboard button and the Gemini ``show_social`` tool. Raises for an unknown
|
||||
account; propagates FaceController errors (e.g. not connected)."""
|
||||
acc = SOCIAL.get(str(account).strip().lower())
|
||||
if not acc:
|
||||
raise HTTPException(status_code=404, detail="unknown account")
|
||||
data = _qr_bytes(acc.get("short") or acc["url"]) # v1 short link -> scannable
|
||||
mf = _get_face()
|
||||
res = mf.show_scratch_image(data)
|
||||
log.info("showing scannable social QR on mask: %s (%s)", acc["handle"], acc.get("short"))
|
||||
return {"ok": True, "handle": acc["handle"], "scannable": True, **(res or {})}
|
||||
|
||||
|
||||
@router.get("/social")
|
||||
async def list_social():
|
||||
return {"accounts": [{"id": k, "handle": v["handle"]} for k, v in SOCIAL.items()]}
|
||||
|
||||
|
||||
def _friendly(exc: Exception) -> HTTPException:
|
||||
"""Map FaceController errors to clean HTTP responses (esp. the common
|
||||
'mask not connected' — usually the mask is off / far / held by the phone app)."""
|
||||
if isinstance(exc, HTTPException):
|
||||
return exc
|
||||
msg = str(exc)
|
||||
if "not connected" in msg or "not started" in msg or "MASK" in msg:
|
||||
return HTTPException(status_code=503, detail=(
|
||||
"Mask not connected — power it on, bring it close to the robot, and "
|
||||
"free it from the phone app."))
|
||||
log.exception("mask scratch op failed")
|
||||
return HTTPException(status_code=500, detail="%s: %s" % (type(exc).__name__, msg))
|
||||
|
||||
|
||||
@router.post("/social/{account}")
|
||||
async def show_social(account: str):
|
||||
try:
|
||||
return await asyncio.to_thread(show_social_on_mask, account)
|
||||
except Exception as exc:
|
||||
raise _friendly(exc)
|
||||
|
||||
|
||||
@router.post("/qr")
|
||||
async def upload_qr(file: UploadFile = File(...)):
|
||||
"""Upload an image (a QR you generated, or any picture) and show it on the mask."""
|
||||
raw = await file.read()
|
||||
if not raw:
|
||||
raise HTTPException(status_code=400, detail="empty upload")
|
||||
from PIL import Image
|
||||
try:
|
||||
img = Image.open(io.BytesIO(raw))
|
||||
img.load()
|
||||
except Exception:
|
||||
raise HTTPException(status_code=400, detail="not a valid image")
|
||||
try:
|
||||
data = await asyncio.to_thread(_image_bytes, img)
|
||||
mf = _get_face()
|
||||
return await asyncio.to_thread(mf.show_scratch_image, data)
|
||||
except Exception as exc:
|
||||
raise _friendly(exc)
|
||||
|
||||
|
||||
@router.post("/face/resume")
|
||||
async def resume_face():
|
||||
"""Stop showing the scratch image and resume the animated face."""
|
||||
mf = _get_face()
|
||||
return await asyncio.to_thread(mf.set_expression, None)
|
||||
|
||||
|
||||
@router.post("/face/mouth")
|
||||
async def face_mouth(hidden: bool = Query(...)):
|
||||
"""Show (hidden=false) or hide (hidden=true) the mouth on the animated face."""
|
||||
mf = _get_face()
|
||||
return await asyncio.to_thread(mf.set_mouth_hidden, hidden)
|
||||
|
||||
|
||||
@router.post("/link")
|
||||
async def face_link(on: bool = Query(...)):
|
||||
"""Link (on=true) / unlink (on=false) Gemini <-> the mask.
|
||||
|
||||
ON connects the mask + lets Gemini drive its emotions/social.
|
||||
OFF tears the link down (no BLE churn) and Gemini stops touching the mask.
|
||||
Default state is OFF. Runs in a thread — a link-on may briefly block while it
|
||||
makes its first connect attempt."""
|
||||
mf = _get_face()
|
||||
return await asyncio.to_thread(mf.set_gemini_linked, on)
|
||||
|
||||
|
||||
# ── saved QR library ────────────────────────────────────────────────
|
||||
# Upload QR/images, save them by name, list/show/delete them. Stored as PNGs
|
||||
# under data/qr_codes so they persist across restarts.
|
||||
|
||||
_QR_DIR = None
|
||||
|
||||
|
||||
def _qr_dir() -> Path:
|
||||
global _QR_DIR
|
||||
if _QR_DIR is None:
|
||||
try:
|
||||
from Project.Sanad.config import BASE_DIR
|
||||
base = Path(BASE_DIR)
|
||||
except Exception:
|
||||
base = Path(__file__).resolve().parents[2]
|
||||
_QR_DIR = base / "data" / "qr_codes"
|
||||
_QR_DIR.mkdir(parents=True, exist_ok=True)
|
||||
return _QR_DIR
|
||||
|
||||
|
||||
def _safe_name(name: str) -> str:
|
||||
n = re.sub(r"[^A-Za-z0-9_.-]", "_", (name or "").strip())[:40].strip("._")
|
||||
return n or "qr"
|
||||
|
||||
|
||||
@router.post("/qr/save")
|
||||
async def qr_save(name: str = Query(...), file: UploadFile = File(...)):
|
||||
"""Save an uploaded QR/image into the library under ``name``."""
|
||||
raw = await file.read()
|
||||
if not raw:
|
||||
raise HTTPException(status_code=400, detail="empty upload")
|
||||
from PIL import Image
|
||||
try:
|
||||
img = Image.open(io.BytesIO(raw))
|
||||
img.load()
|
||||
except Exception:
|
||||
raise HTTPException(status_code=400, detail="not a valid image")
|
||||
sn = _safe_name(name)
|
||||
await asyncio.to_thread(img.convert("RGB").save, str(_qr_dir() / (sn + ".png")))
|
||||
return {"ok": True, "name": sn}
|
||||
|
||||
|
||||
@router.post("/qr/save_link")
|
||||
async def qr_save_link(name: str = Query(...), url: str = Query(...)):
|
||||
"""Generate a QR from ``url`` and save it to the library. Returns the QR
|
||||
version + whether it's short enough to actually scan on the mask (version 1)."""
|
||||
u = (url or "").strip()
|
||||
if not u:
|
||||
raise HTTPException(status_code=400, detail="empty url")
|
||||
_ensure_mask_path()
|
||||
import qrcode
|
||||
qr = qrcode.QRCode(error_correction=qrcode.constants.ERROR_CORRECT_L,
|
||||
box_size=10, border=2)
|
||||
qr.add_data(u)
|
||||
qr.make(fit=True)
|
||||
img = qr.make_image(fill_color=(255, 255, 255),
|
||||
back_color=(0, 0, 0)).convert("RGB")
|
||||
sn = _safe_name(name or u)
|
||||
await asyncio.to_thread(img.save, str(_qr_dir() / (sn + ".png")))
|
||||
return {"ok": True, "name": sn, "version": qr.version,
|
||||
"scannable_on_mask": qr.version <= 1,
|
||||
"note": ("scannable" if qr.version <= 1 else
|
||||
"too dense to scan on the mask — use a shorter link")}
|
||||
|
||||
|
||||
@router.get("/qr/library")
|
||||
async def qr_library():
|
||||
"""List the saved QR names."""
|
||||
return {"qr": sorted(p.stem for p in _qr_dir().glob("*.png"))}
|
||||
|
||||
|
||||
@router.get("/qr/thumb/{name}")
|
||||
async def qr_thumb(name: str):
|
||||
"""Serve a saved QR image (for the dashboard thumbnail)."""
|
||||
p = _qr_dir() / (_safe_name(name) + ".png")
|
||||
if not p.exists():
|
||||
raise HTTPException(status_code=404, detail="not found")
|
||||
return FileResponse(str(p), media_type="image/png")
|
||||
|
||||
|
||||
@router.post("/qr/show/{name}")
|
||||
async def qr_show(name: str):
|
||||
"""Show a saved QR (under the eyes) on the mask."""
|
||||
p = _qr_dir() / (_safe_name(name) + ".png")
|
||||
if not p.exists():
|
||||
raise HTTPException(status_code=404, detail="not found")
|
||||
from PIL import Image
|
||||
try:
|
||||
img = Image.open(p)
|
||||
data = await asyncio.to_thread(_image_bytes, img)
|
||||
mf = _get_face()
|
||||
return await asyncio.to_thread(mf.show_scratch_image, data)
|
||||
except Exception as exc:
|
||||
raise _friendly(exc)
|
||||
|
||||
|
||||
@router.delete("/qr/{name}")
|
||||
async def qr_delete(name: str):
|
||||
"""Delete a saved QR from the library."""
|
||||
p = _qr_dir() / (_safe_name(name) + ".png")
|
||||
if p.exists():
|
||||
p.unlink()
|
||||
return {"ok": True, "deleted": _safe_name(name)}
|
||||
|
||||
|
||||
# ── saved TEXT library ──────────────────────────────────────────────
|
||||
# Save words/phrases and scroll any of them across the mask on demand.
|
||||
|
||||
_TEXT_DIR = None
|
||||
|
||||
|
||||
def _text_dir() -> Path:
|
||||
global _TEXT_DIR
|
||||
if _TEXT_DIR is None:
|
||||
try:
|
||||
from Project.Sanad.config import BASE_DIR
|
||||
base = Path(BASE_DIR)
|
||||
except Exception:
|
||||
base = Path(__file__).resolve().parents[2]
|
||||
_TEXT_DIR = base / "data" / "mask_texts"
|
||||
_TEXT_DIR.mkdir(parents=True, exist_ok=True)
|
||||
return _TEXT_DIR
|
||||
|
||||
|
||||
@router.post("/texts/save")
|
||||
async def text_save(text: str = Query(...), name: str = Query("")):
|
||||
"""Save a word/phrase to the text library (name defaults to the text)."""
|
||||
t = (text or "").strip()[:200]
|
||||
if not t:
|
||||
raise HTTPException(status_code=400, detail="empty text")
|
||||
nm = _safe_name(name or t)
|
||||
await asyncio.to_thread((_text_dir() / (nm + ".txt")).write_text, t)
|
||||
return {"ok": True, "name": nm, "text": t}
|
||||
|
||||
|
||||
@router.get("/texts/library")
|
||||
async def text_library():
|
||||
"""List the saved texts."""
|
||||
out = []
|
||||
for p in sorted(_text_dir().glob("*.txt")):
|
||||
try:
|
||||
out.append({"name": p.stem, "text": p.read_text()[:80]})
|
||||
except Exception:
|
||||
pass
|
||||
return {"texts": out}
|
||||
|
||||
|
||||
@router.post("/texts/show/{name}")
|
||||
async def text_show(name: str):
|
||||
"""Scroll a saved text across the mask."""
|
||||
p = _text_dir() / (_safe_name(name) + ".txt")
|
||||
if not p.exists():
|
||||
raise HTTPException(status_code=404, detail="not found")
|
||||
txt = p.read_text()
|
||||
mf = _get_face()
|
||||
try:
|
||||
return await asyncio.to_thread(mf.set_text, txt, (255, 255, 255), None, None, 38)
|
||||
except Exception as exc:
|
||||
raise _friendly(exc)
|
||||
|
||||
|
||||
@router.delete("/texts/{name}")
|
||||
async def text_delete(name: str):
|
||||
"""Delete a saved text."""
|
||||
p = _text_dir() / (_safe_name(name) + ".txt")
|
||||
if p.exists():
|
||||
p.unlink()
|
||||
return {"ok": True, "deleted": _safe_name(name)}
|
||||
@ -10,6 +10,23 @@ from pydantic import BaseModel
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
def _block_if_movement_armed():
|
||||
"""409 if locomotion movement is armed — arm actions are mutually exclusive
|
||||
with walking. The arm controller's own motion-block is the safety net; this
|
||||
just gives the dashboard a clear message instead of a silent no-op."""
|
||||
try:
|
||||
from Project.Sanad.main import loco_controller # type: ignore
|
||||
armed = loco_controller is not None and loco_controller.is_armed()
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception:
|
||||
return
|
||||
if armed:
|
||||
raise HTTPException(
|
||||
409, "Arm actions are disabled while movement is enabled. "
|
||||
"Disable movement in the Controller tab first.")
|
||||
|
||||
|
||||
@router.get("/status")
|
||||
async def motion_status():
|
||||
from Project.Sanad.main import arm
|
||||
@ -33,6 +50,7 @@ async def trigger_action(payload: TriggerPayload):
|
||||
from Project.Sanad.main import arm
|
||||
if arm is None:
|
||||
raise HTTPException(503, "Arm controller not attached.")
|
||||
_block_if_movement_armed()
|
||||
|
||||
speed = max(0.1, min(payload.speed, 5.0))
|
||||
|
||||
|
||||
402
dashboard/routes/navigation.py
Normal file
402
dashboard/routes/navigation.py
Normal file
@ -0,0 +1,402 @@
|
||||
"""Navigation tab — proxy to the web_nav3 Nav2 stack.
|
||||
|
||||
Routes live under /api/nav (the prefix is applied centrally in dashboard/app.py,
|
||||
NOT here). This router is a thin HTTP proxy: it forwards dashboard requests to a
|
||||
single module-level WebNav3Client, which itself talks to the standalone web_nav3
|
||||
FastAPI service (default http://127.0.0.1:8765 + rosbridge on :9090).
|
||||
|
||||
Fault isolation, two layers:
|
||||
1. The `from ...navigation import WebNav3Client` import is GUARDED. If the
|
||||
navigation package can't be imported (missing dep, syntax error), this
|
||||
module still imports cleanly — `_CLIENT` is None and every handler degrades
|
||||
(GET /status returns {"available": False}; actions raise 503). This mirrors
|
||||
how app.py loads each router in isolation.
|
||||
2. WebNav3Client never raises into us by contract — every method returns a
|
||||
clean dict / NavStatus even when web_nav3 is unreachable — so handlers just
|
||||
forward the result. Blocking HTTP calls run off the event loop.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Query
|
||||
from pydantic import BaseModel
|
||||
|
||||
from Project.Sanad.core.logger import get_logger
|
||||
|
||||
from Project.Sanad.dashboard.routes import _arbiter
|
||||
|
||||
log = get_logger("navigation_route")
|
||||
|
||||
# Module-level router with NO prefix and NO tags — those are supplied by
|
||||
# app.include_router(prefix="/api/nav", tags=["navigation"]) at registration time.
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
# ── guarded optional import ─────────────────────────────────
|
||||
# A broken navigation package must NOT stop this route module from importing —
|
||||
# app.py would otherwise log the whole router as failed. Guard it and degrade.
|
||||
try:
|
||||
from Project.Sanad.navigation import WebNav3Client # type: ignore
|
||||
_IMPORT_ERROR: str | None = None
|
||||
except Exception as exc: # noqa: BLE001
|
||||
WebNav3Client = None # type: ignore[assignment,misc]
|
||||
_IMPORT_ERROR = f"{type(exc).__name__}: {exc}"
|
||||
log.warning("navigation client unavailable — nav routes degraded: %s", _IMPORT_ERROR)
|
||||
|
||||
|
||||
# ── config (env var -> dashboard config section -> default) ──
|
||||
def _nav_config() -> dict:
|
||||
"""Resolve nav connection config. Precedence: env var -> config -> default."""
|
||||
import os
|
||||
|
||||
from Project.Sanad.core.config_loader import section as _cfg_section
|
||||
|
||||
cfg = _cfg_section("dashboard", "navigation")
|
||||
web_nav3_url = (
|
||||
os.environ.get("WEB_NAV3_URL")
|
||||
or cfg.get("web_nav3_url")
|
||||
or "http://127.0.0.1:8765"
|
||||
)
|
||||
rosbridge_url = (
|
||||
os.environ.get("ROSBRIDGE_URL")
|
||||
or cfg.get("rosbridge_url")
|
||||
or "ws://127.0.0.1:9090"
|
||||
)
|
||||
robot = os.environ.get("SANAD_ROBOT_NAME") or cfg.get("robot") or "sanad"
|
||||
return {
|
||||
"web_nav3_url": str(web_nav3_url),
|
||||
"rosbridge_url": str(rosbridge_url),
|
||||
"robot": str(robot),
|
||||
}
|
||||
|
||||
|
||||
_CFG = _nav_config()
|
||||
|
||||
# ── single module-level client ──────────────────────────────
|
||||
# One WebNav3Client for the whole dashboard, built from config. If the import
|
||||
# was guarded out (above), or construction fails, _CLIENT stays None and every
|
||||
# handler degrades gracefully.
|
||||
if WebNav3Client is not None:
|
||||
try:
|
||||
_CLIENT = WebNav3Client(base_url=_CFG["web_nav3_url"], robot=_CFG["robot"])
|
||||
log.info("WebNav3Client ready → %s (robot=%s)", _CFG["web_nav3_url"], _CFG["robot"])
|
||||
except Exception as exc: # noqa: BLE001
|
||||
_CLIENT = None
|
||||
_IMPORT_ERROR = f"construct failed: {type(exc).__name__}: {exc}"
|
||||
log.warning("WebNav3Client construction failed — nav routes degraded: %s", exc)
|
||||
else:
|
||||
_CLIENT = None
|
||||
|
||||
|
||||
def _require():
|
||||
"""Return the live client or raise 503 (for ACTION endpoints)."""
|
||||
if _CLIENT is None:
|
||||
raise HTTPException(503, f"Navigation client unavailable. {_IMPORT_ERROR or ''}".strip())
|
||||
return _CLIENT
|
||||
|
||||
|
||||
def _claim_nav():
|
||||
"""Arbitration gate: refuse to start a Nav2 goal while manual loco owns legs."""
|
||||
if not _arbiter.acquire_nav():
|
||||
raise HTTPException(
|
||||
409, "Manual movement (Controller) is armed. Disarm it before navigating."
|
||||
)
|
||||
|
||||
|
||||
# ── request bodies ──────────────────────────────────────────
|
||||
class _NameBody(BaseModel):
|
||||
name: str
|
||||
|
||||
|
||||
class _IdBody(BaseModel):
|
||||
id: object # mission ids may be int or str; forward as-is
|
||||
|
||||
|
||||
class _StartBody(BaseModel):
|
||||
mode: int = 2 # web_nav3 launch mode (e.g. 3 = localize against a saved map)
|
||||
db_path: str | None = None # saved map to load (None = build fresh)
|
||||
|
||||
|
||||
class _PoseBody(BaseModel):
|
||||
name: str
|
||||
x: float
|
||||
y: float
|
||||
yaw: float = 0.0
|
||||
|
||||
|
||||
class _RenameBody(BaseModel):
|
||||
old: str
|
||||
new: str
|
||||
|
||||
|
||||
# ── status (never raises — degraded body when unavailable) ──
|
||||
@router.get("/status")
|
||||
async def status():
|
||||
if _CLIENT is None:
|
||||
return {"available": False, "error": _IMPORT_ERROR}
|
||||
nav = await asyncio.to_thread(_CLIENT.status)
|
||||
# WebNav3Client.status() returns a NavStatus dataclass.
|
||||
body = nav.as_dict() if hasattr(nav, "as_dict") else dict(nav)
|
||||
body["available"] = True
|
||||
return body
|
||||
|
||||
|
||||
# ── places / navigation ─────────────────────────────────────
|
||||
@router.get("/places")
|
||||
async def places(map_name: str | None = Query(None, alias="map")):
|
||||
"""List saved places. Per-MAP when ?map=<name> is given (each map keeps
|
||||
its own places); else the legacy per-robot store."""
|
||||
client = _require()
|
||||
return await asyncio.to_thread(client.list_places, map_name)
|
||||
|
||||
|
||||
@router.post("/goto")
|
||||
async def goto(body: _NameBody):
|
||||
client = _require()
|
||||
_claim_nav()
|
||||
res = await asyncio.to_thread(client.goto, body.name)
|
||||
# A failed dispatch never drove the legs — release the gate so manual loco
|
||||
# isn't locked out by a goto that never started.
|
||||
if isinstance(res, dict) and not res.get("ok", True):
|
||||
_arbiter.release_nav()
|
||||
return res
|
||||
|
||||
|
||||
@router.post("/start")
|
||||
async def start(body: _StartBody):
|
||||
client = _require()
|
||||
return await asyncio.to_thread(client.start, body.mode, body.db_path)
|
||||
|
||||
|
||||
class _DbBody(BaseModel):
|
||||
db_path: str
|
||||
|
||||
|
||||
@router.post("/load_map")
|
||||
async def load_map(body: _DbBody):
|
||||
"""View a saved map: stop any running bringup, then localize against it."""
|
||||
client = _require()
|
||||
return await asyncio.to_thread(client.load_map, body.db_path)
|
||||
|
||||
|
||||
@router.post("/cancel")
|
||||
async def cancel():
|
||||
client = _require()
|
||||
res = await asyncio.to_thread(client.cancel)
|
||||
# WebNav3Client.cancel() is a no-op server-side (it only returns a note),
|
||||
# so releasing the arbiter without truly stopping Nav2 would let the robot
|
||||
# keep driving while manual loco re-acquires the legs (double-drive). Send a
|
||||
# REAL goal-cancel over rosbridge first, and disarm the arrival monitor so a
|
||||
# stale terminal can't fire, THEN release.
|
||||
try:
|
||||
from Project.Sanad.navigation.goal_monitor import request_cancel, disarm
|
||||
disarm()
|
||||
cancelled = await asyncio.to_thread(request_cancel)
|
||||
if isinstance(res, dict):
|
||||
res = {**res, "cancel_sent": bool(cancelled)}
|
||||
except Exception as exc: # noqa: BLE001
|
||||
log.debug("goal cancel skipped: %s", exc)
|
||||
_arbiter.release_nav()
|
||||
return res
|
||||
|
||||
|
||||
@router.post("/save_here")
|
||||
async def save_here(body: _NameBody):
|
||||
client = _require()
|
||||
return await asyncio.to_thread(client.save_here, body.name)
|
||||
|
||||
|
||||
@router.post("/save_at")
|
||||
async def save_at(body: _PoseBody, map_name: str | None = Query(None, alias="map")):
|
||||
"""Save a named place at a map coordinate (from clicking the map). Per-MAP
|
||||
when ?map=<name> given. Re-saving an existing name MOVES the place."""
|
||||
client = _require()
|
||||
return await asyncio.to_thread(client.save_at, body.name, body.x, body.y, body.yaw, map_name)
|
||||
|
||||
|
||||
@router.post("/places/delete")
|
||||
async def delete_place(body: _NameBody, map_name: str | None = Query(None, alias="map")):
|
||||
"""Delete a saved place (per-map)."""
|
||||
client = _require()
|
||||
return await asyncio.to_thread(client.delete_place, body.name, map_name)
|
||||
|
||||
|
||||
@router.post("/places/rename")
|
||||
async def rename_place(body: _RenameBody, map_name: str | None = Query(None, alias="map")):
|
||||
"""Rename a saved place (per-map)."""
|
||||
client = _require()
|
||||
return await asyncio.to_thread(client.rename_place, body.old, body.new, map_name)
|
||||
|
||||
|
||||
class _MapEditsBody(BaseModel):
|
||||
edits: list # [[world_x, world_y, value], ...] value 0=free/erase, 100=wall
|
||||
|
||||
|
||||
@router.get("/map_edits")
|
||||
async def get_map_edits(map_name: str = Query(..., alias="map")):
|
||||
"""Saved edit overlay for a map (erased points + painted walls)."""
|
||||
client = _require()
|
||||
return await asyncio.to_thread(client.get_map_edits, map_name)
|
||||
|
||||
|
||||
@router.post("/map_edits")
|
||||
async def save_map_edits(body: _MapEditsBody, map_name: str = Query(..., alias="map")):
|
||||
"""Persist a map's edit overlay (Map Editor)."""
|
||||
client = _require()
|
||||
return await asyncio.to_thread(client.save_map_edits, map_name, body.edits)
|
||||
|
||||
|
||||
class _VoiceGotoBody(BaseModel):
|
||||
place: str
|
||||
|
||||
|
||||
def _resolve_place(client, spoken: str) -> dict:
|
||||
"""Resolve a spoken place name against the ACTIVE map's places.
|
||||
|
||||
Strategy: exact (case-insensitive) → single substring candidate →
|
||||
ambiguous / unknown. Returns a dict the caller (and ultimately Gemini)
|
||||
can act on. Never raises.
|
||||
"""
|
||||
try:
|
||||
st = client.status()
|
||||
body = st.as_dict() if hasattr(st, "as_dict") else dict(st)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
return {"ok": False, "reason": "status_error", "detail": str(exc)[:160]}
|
||||
if not body.get("bringup_alive"):
|
||||
return {"ok": False, "reason": "no_map",
|
||||
"detail": "No navigation session is running — load a map first."}
|
||||
active_map = body.get("active_map")
|
||||
try:
|
||||
places = client.list_places(active_map) or []
|
||||
except Exception: # noqa: BLE001
|
||||
places = []
|
||||
names = [p.get("name") for p in places if isinstance(p, dict) and p.get("name")]
|
||||
sl = (spoken or "").strip().lower()
|
||||
if not sl:
|
||||
return {"ok": False, "reason": "no_place", "map": active_map, "places": names}
|
||||
exact = [n for n in names if n.lower() == sl]
|
||||
if exact:
|
||||
return {"ok": True, "resolved": exact[0], "map": active_map}
|
||||
subs = []
|
||||
for n in names:
|
||||
nl = n.lower()
|
||||
if sl in nl or nl in sl:
|
||||
subs.append(n)
|
||||
subs = list(dict.fromkeys(subs)) # de-dup, preserve order
|
||||
if len(subs) == 1:
|
||||
return {"ok": True, "resolved": subs[0], "map": active_map}
|
||||
if len(subs) > 1:
|
||||
return {"ok": False, "reason": "ambiguous", "candidates": subs, "map": active_map}
|
||||
return {"ok": False, "reason": "unknown_place", "candidates": names, "map": active_map}
|
||||
|
||||
|
||||
@router.get("/active")
|
||||
async def active():
|
||||
"""Navigation context for Gemini: the active map, its mode, and that map's
|
||||
place names — one call so the voice tools (list_places / where_am_i) don't
|
||||
have to guess the active map."""
|
||||
client = _require()
|
||||
st = await asyncio.to_thread(client.status)
|
||||
body = st.as_dict() if hasattr(st, "as_dict") else dict(st)
|
||||
places = []
|
||||
if body.get("bringup_alive"):
|
||||
try:
|
||||
pl = await asyncio.to_thread(client.list_places, body.get("active_map"))
|
||||
places = [p.get("name") for p in (pl or [])
|
||||
if isinstance(p, dict) and p.get("name")]
|
||||
except Exception: # noqa: BLE001
|
||||
places = []
|
||||
return {
|
||||
"map": body.get("active_map"),
|
||||
"mode": body.get("mode"),
|
||||
"mode_label": body.get("mode_label"),
|
||||
"localizing": bool(body.get("localizing")),
|
||||
"bringup_alive": bool(body.get("bringup_alive")),
|
||||
"places": places,
|
||||
}
|
||||
|
||||
|
||||
@router.post("/voice_goto")
|
||||
async def voice_goto(body: _VoiceGotoBody):
|
||||
"""Resolve a spoken place name and drive there — Gemini's navigate_to_place.
|
||||
|
||||
Arbiter-gated (claims the legs for Nav2) and arms the arrival monitor so
|
||||
Gemini later hears [NAV ARRIVED]/[NAV FAILED]. Never raises into the caller;
|
||||
returns a structured result the model can speak from.
|
||||
"""
|
||||
client = _require()
|
||||
res = await asyncio.to_thread(_resolve_place, client, body.place or "")
|
||||
if not res.get("ok"):
|
||||
return res
|
||||
# Claim the legs for Nav2 — refuse (don't raise) if manual loco is armed.
|
||||
if not _arbiter.acquire_nav():
|
||||
return {"ok": False, "reason": "manual_armed",
|
||||
"detail": "Manual movement (Controller) is armed — disarm it to navigate."}
|
||||
drive = await asyncio.to_thread(client.goto, res["resolved"])
|
||||
if isinstance(drive, dict) and not drive.get("ok", True):
|
||||
_arbiter.release_nav()
|
||||
return {"ok": False, "reason": "dispatch_failed",
|
||||
"resolved": res["resolved"], "detail": drive}
|
||||
# Arm arrival monitoring (best-effort; absence must not fail the drive).
|
||||
try:
|
||||
from Project.Sanad.navigation.goal_monitor import arm_goal
|
||||
arm_goal(res["resolved"])
|
||||
except Exception as exc: # noqa: BLE001
|
||||
log.debug("goal monitor arm skipped: %s", exc)
|
||||
return {"ok": True, "resolved": res["resolved"], "map": res.get("map")}
|
||||
|
||||
|
||||
@router.post("/goto_pose")
|
||||
async def goto_pose(body: _PoseBody):
|
||||
"""Arbiter-gate a coordinate nav goal (click-to-drive).
|
||||
|
||||
The browser publishes the actual /goal_pose over rosbridge; this only
|
||||
CLAIMS the legs for Nav2 (409 if manual loco is armed) so the two stacks
|
||||
never both drive. The frontend sends the goal only after this returns ok.
|
||||
"""
|
||||
_require()
|
||||
_claim_nav()
|
||||
# Arm the arrival monitor so this click-to-drive goal releases the arbiter
|
||||
# when it ends — without this, nav_active stays True forever after the goal
|
||||
# completes (the browser publishes the goal but never arms anything).
|
||||
try:
|
||||
from Project.Sanad.navigation.goal_monitor import arm_goal
|
||||
arm_goal(f"({body.x:.1f}, {body.y:.1f})")
|
||||
except Exception as exc: # noqa: BLE001
|
||||
log.debug("goal monitor arm skipped: %s", exc)
|
||||
return {"ok": True, "x": body.x, "y": body.y, "yaw": body.yaw}
|
||||
|
||||
|
||||
# ── maps / missions ─────────────────────────────────────────
|
||||
@router.get("/maps")
|
||||
async def maps():
|
||||
client = _require()
|
||||
return await asyncio.to_thread(client.list_maps)
|
||||
|
||||
|
||||
@router.get("/missions")
|
||||
async def missions():
|
||||
client = _require()
|
||||
return await asyncio.to_thread(client.list_missions)
|
||||
|
||||
|
||||
@router.post("/missions/run")
|
||||
async def run_mission(body: _IdBody):
|
||||
client = _require()
|
||||
_claim_nav()
|
||||
res = await asyncio.to_thread(client.run_mission, body.id)
|
||||
if isinstance(res, dict) and not res.get("ok", True):
|
||||
_arbiter.release_nav()
|
||||
return res
|
||||
|
||||
|
||||
# ── config (what the SPA needs to render links / connect) ───
|
||||
@router.get("/config")
|
||||
async def config():
|
||||
return {
|
||||
"web_nav3_url": _CFG["web_nav3_url"],
|
||||
"rosbridge_url": _CFG["rosbridge_url"],
|
||||
"robot": _CFG["robot"],
|
||||
}
|
||||
457
dashboard/routes/recognition.py
Normal file
457
dashboard/routes/recognition.py
Normal file
@ -0,0 +1,457 @@
|
||||
"""Recognition tab — camera vision + face gallery + hot toggles.
|
||||
|
||||
Single router covering:
|
||||
- Vision / Face Recognition toggles (hot — no Gemini restart needed)
|
||||
- Live camera preview (latest JPEG drop)
|
||||
- Face gallery CRUD: enroll, upload, capture, rename, delete, ZIP
|
||||
- Per-photo download + delete
|
||||
|
||||
Toggle changes write data/.recognition_state.json atomically. The Gemini
|
||||
child polls that file at 1 Hz and applies changes mid-session.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import io
|
||||
from typing import Optional
|
||||
|
||||
from fastapi import APIRouter, File, HTTPException, Query, UploadFile
|
||||
from fastapi.responses import FileResponse, Response, StreamingResponse
|
||||
from pydantic import BaseModel
|
||||
|
||||
from Project.Sanad.config import BASE_DIR
|
||||
from Project.Sanad.core.logger import get_logger
|
||||
from Project.Sanad.dashboard.routes._safe_io import check_upload_size
|
||||
from Project.Sanad.vision import recognition_state
|
||||
|
||||
log = get_logger("recognition_routes")
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
# ── paths (resolved from BASE_DIR) ──────────────────────────
|
||||
|
||||
STATE_PATH = BASE_DIR / "data" / ".recognition_state.json"
|
||||
FACES_DIR = BASE_DIR / "data" / "faces"
|
||||
|
||||
ALLOWED_IMAGE_EXTS = {".jpg", ".jpeg", ".png"}
|
||||
|
||||
|
||||
# ── helpers ─────────────────────────────────────────────────
|
||||
|
||||
def _get_camera():
|
||||
"""Lazy import to avoid circular import on dashboard load."""
|
||||
try:
|
||||
from Project.Sanad.main import camera # type: ignore
|
||||
return camera
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
def _get_gallery():
|
||||
"""Lazy import — same reason."""
|
||||
try:
|
||||
from Project.Sanad.main import gallery # type: ignore
|
||||
return gallery
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
def _bump_and_write_state(**changes) -> recognition_state.RecognitionState:
|
||||
"""Apply changes (vision_enabled, face_rec_enabled) and persist."""
|
||||
return recognition_state.mutate(STATE_PATH, **changes)
|
||||
|
||||
|
||||
def _bump_gallery_version() -> int:
|
||||
cur = recognition_state.read(STATE_PATH)
|
||||
new_version = cur.gallery_version + 1
|
||||
recognition_state.mutate(STATE_PATH, gallery_version=new_version)
|
||||
return new_version
|
||||
|
||||
|
||||
# ── state + toggles ─────────────────────────────────────────
|
||||
|
||||
@router.get("/state")
|
||||
async def get_state():
|
||||
"""Return the current toggle/camera/gallery state."""
|
||||
st = recognition_state.read(STATE_PATH)
|
||||
cam = _get_camera()
|
||||
gallery = _get_gallery()
|
||||
faces_count = 0
|
||||
photos_count = 0
|
||||
if gallery is not None:
|
||||
try:
|
||||
entries = gallery.list()
|
||||
faces_count = len(entries)
|
||||
photos_count = sum(len(e.sample_paths) for e in entries)
|
||||
except Exception:
|
||||
pass
|
||||
return {
|
||||
"vision_enabled": st.vision_enabled,
|
||||
"face_rec_enabled": st.face_rec_enabled,
|
||||
"gallery_version": st.gallery_version,
|
||||
"camera": cam.status() if cam is not None else {
|
||||
"running": False, "backend": None, "error": "camera subsystem unavailable"
|
||||
},
|
||||
"faces_count": faces_count,
|
||||
"photos_count": photos_count,
|
||||
}
|
||||
|
||||
|
||||
@router.post("/vision")
|
||||
async def set_vision(on: bool = Query(...)):
|
||||
"""Enable / disable camera vision (hot — no Gemini restart)."""
|
||||
cam = _get_camera()
|
||||
if cam is None:
|
||||
log.warning("vision toggle requested but camera subsystem unavailable")
|
||||
raise HTTPException(503, "Camera subsystem not available.")
|
||||
|
||||
if on and not cam.is_running():
|
||||
ok = cam.start()
|
||||
if not ok:
|
||||
log.warning("vision ON requested but camera.start() failed: %s",
|
||||
cam.error or "no backend")
|
||||
_bump_and_write_state(vision_enabled=False)
|
||||
raise HTTPException(503,
|
||||
f"Camera could not start (no backend). {cam.error or ''}")
|
||||
elif (not on) and cam.is_running():
|
||||
cam.stop()
|
||||
|
||||
st = _bump_and_write_state(vision_enabled=bool(on))
|
||||
log.info("vision %s (backend=%s)", "ON" if on else "OFF",
|
||||
cam.backend if cam.is_running() else "none")
|
||||
return {"ok": True, "vision_enabled": st.vision_enabled,
|
||||
"camera": cam.status()}
|
||||
|
||||
|
||||
@router.post("/face-rec")
|
||||
async def set_face_rec(on: bool = Query(...)):
|
||||
"""Enable / disable face recognition (hot — no Gemini restart).
|
||||
|
||||
The Gemini child picks the change up within ~1 s: ON re-sends the
|
||||
gallery primer and tells Gemini it can recognise people; OFF tells
|
||||
Gemini to disregard the gallery and stop identifying anyone. Both
|
||||
take effect on the live session — no reconnect needed.
|
||||
"""
|
||||
st = _bump_and_write_state(face_rec_enabled=bool(on))
|
||||
log.info("face recognition %s", "ON" if on else "OFF")
|
||||
return {"ok": True, "face_rec_enabled": st.face_rec_enabled}
|
||||
|
||||
|
||||
@router.post("/sync")
|
||||
async def sync_gallery():
|
||||
"""Bump gallery_version so the child re-sends the primer if face-rec is ON."""
|
||||
v = _bump_gallery_version()
|
||||
log.info("gallery sync requested → v.%d", v)
|
||||
return {"ok": True, "gallery_version": v}
|
||||
|
||||
|
||||
# ── live preview ────────────────────────────────────────────
|
||||
|
||||
@router.get("/frame.jpg")
|
||||
async def latest_frame():
|
||||
"""Serve the most recent camera frame straight from the daemon's
|
||||
in-memory cache (no file drop — frames are also pushed to the Gemini
|
||||
child over its stdin)."""
|
||||
cam = _get_camera()
|
||||
if cam is None:
|
||||
raise HTTPException(503, "Camera subsystem unavailable.")
|
||||
jpeg = cam.snapshot_jpeg()
|
||||
if not jpeg:
|
||||
raise HTTPException(404, "No frame captured yet.")
|
||||
return Response(
|
||||
content=jpeg,
|
||||
media_type="image/jpeg",
|
||||
headers={"Cache-Control": "no-store, must-revalidate"},
|
||||
)
|
||||
|
||||
|
||||
# ── camera resolution / quality ─────────────────────────────
|
||||
|
||||
class CameraConfigPayload(BaseModel):
|
||||
width: Optional[int] = None
|
||||
height: Optional[int] = None
|
||||
fps: Optional[int] = None
|
||||
jpeg_quality: Optional[int] = None
|
||||
|
||||
|
||||
@router.post("/camera-config")
|
||||
async def set_camera_config(payload: CameraConfigPayload):
|
||||
"""Hot-swap the camera capture profile (resolution / fps / JPEG quality).
|
||||
|
||||
If the camera is running, CameraDaemon.reconfigure() rebuilds the
|
||||
pipeline at the new profile (~0.5 s gap). If idle, the values just
|
||||
take effect on the next start. Bounds are sanity-checked here so a
|
||||
fat-fingered value can't wedge the daemon."""
|
||||
cam = _get_camera()
|
||||
if cam is None:
|
||||
raise HTTPException(503, "Camera subsystem unavailable.")
|
||||
if payload.width is not None and not (160 <= payload.width <= 1920):
|
||||
raise HTTPException(400, "width out of range (160–1920)")
|
||||
if payload.height is not None and not (120 <= payload.height <= 1080):
|
||||
raise HTTPException(400, "height out of range (120–1080)")
|
||||
if payload.fps is not None and not (1 <= payload.fps <= 60):
|
||||
raise HTTPException(400, "fps out of range (1–60)")
|
||||
if payload.jpeg_quality is not None and not (10 <= payload.jpeg_quality <= 95):
|
||||
raise HTTPException(400, "jpeg_quality out of range (10–95)")
|
||||
profile = cam.reconfigure(
|
||||
width=payload.width, height=payload.height,
|
||||
fps=payload.fps, jpeg_quality=payload.jpeg_quality,
|
||||
)
|
||||
log.info("camera reconfigured via dashboard → %s", profile)
|
||||
return {"ok": True, "profile": profile, "camera": cam.status()}
|
||||
|
||||
|
||||
# ── face gallery routes ─────────────────────────────────────
|
||||
|
||||
def _validate_image(content: bytes, filename: str | None = None) -> None:
|
||||
"""Reject non-JPEG/PNG content + oversize uploads."""
|
||||
check_upload_size(content)
|
||||
if len(content) < 16:
|
||||
raise HTTPException(400, "Image too small / empty.")
|
||||
is_jpeg = content[:3] == b"\xff\xd8\xff"
|
||||
is_png = content[:8] == b"\x89PNG\r\n\x1a\n"
|
||||
if not (is_jpeg or is_png):
|
||||
raise HTTPException(
|
||||
400,
|
||||
f"Only JPEG/PNG accepted (got {filename or 'unknown'}).",
|
||||
)
|
||||
|
||||
|
||||
def _entry_to_dict(entry) -> dict:
|
||||
photos = []
|
||||
for p in entry.sample_paths:
|
||||
try:
|
||||
photos.append({"name": p.name, "size_bytes": p.stat().st_size})
|
||||
except OSError:
|
||||
continue
|
||||
return {
|
||||
"id": entry.id,
|
||||
"name": entry.name,
|
||||
"description": entry.description,
|
||||
"added_at": entry.added_at,
|
||||
"photos": photos,
|
||||
}
|
||||
|
||||
|
||||
@router.get("/faces")
|
||||
async def list_faces():
|
||||
gallery = _get_gallery()
|
||||
if gallery is None:
|
||||
raise HTTPException(503, "Face gallery subsystem unavailable.")
|
||||
entries = gallery.list()
|
||||
return {"faces": [_entry_to_dict(e) for e in entries],
|
||||
"total": len(entries)}
|
||||
|
||||
|
||||
class RenamePayload(BaseModel):
|
||||
name: Optional[str] = None
|
||||
|
||||
|
||||
class DescribePayload(BaseModel):
|
||||
description: Optional[str] = None
|
||||
|
||||
|
||||
@router.post("/faces/enroll")
|
||||
async def enroll_from_camera(name: Optional[str] = Query(default=None),
|
||||
description: Optional[str] = Query(default=None)):
|
||||
"""Create a new face from the camera's latest snapshot."""
|
||||
gallery = _get_gallery()
|
||||
if gallery is None:
|
||||
raise HTTPException(503, "Face gallery subsystem unavailable.")
|
||||
cam = _get_camera()
|
||||
if cam is None or not cam.is_running():
|
||||
raise HTTPException(409, "Camera is not running. Toggle Vision ON first.")
|
||||
# get_fresh_frame waits briefly for a current frame so the enrolled
|
||||
# photo is the scene the user is posing for, not a stale buffer.
|
||||
jpeg = cam.get_fresh_frame(max_age_s=0.5, timeout_s=1.5)
|
||||
if not jpeg:
|
||||
raise HTTPException(409, "Camera has no frame yet. Wait a moment and retry.")
|
||||
entry = gallery.create_face(
|
||||
[jpeg],
|
||||
name=name.strip() if name else None,
|
||||
description=description.strip() if description else None,
|
||||
)
|
||||
v = _bump_gallery_version()
|
||||
log.info("enrolled face_%d via camera (name=%s, desc=%s, v.%d)",
|
||||
entry.id, name or "(unnamed)",
|
||||
"yes" if description else "no", v)
|
||||
return {"ok": True, "face": _entry_to_dict(entry)}
|
||||
|
||||
|
||||
@router.post("/faces/upload")
|
||||
async def enroll_from_upload(
|
||||
files: list[UploadFile] = File(...),
|
||||
name: Optional[str] = Query(default=None),
|
||||
description: Optional[str] = Query(default=None),
|
||||
):
|
||||
"""Create a new face from uploaded image file(s)."""
|
||||
gallery = _get_gallery()
|
||||
if gallery is None:
|
||||
raise HTTPException(503, "Face gallery subsystem unavailable.")
|
||||
if not files:
|
||||
raise HTTPException(400, "At least one image file required.")
|
||||
image_bytes: list[bytes] = []
|
||||
for f in files:
|
||||
content = await f.read()
|
||||
_validate_image(content, f.filename)
|
||||
image_bytes.append(content)
|
||||
entry = gallery.create_face(
|
||||
image_bytes,
|
||||
name=name.strip() if name else None,
|
||||
description=description.strip() if description else None,
|
||||
)
|
||||
v = _bump_gallery_version()
|
||||
log.info("enrolled face_%d via upload (%d photos, name=%s, desc=%s, v.%d)",
|
||||
entry.id, len(image_bytes), name or "(unnamed)",
|
||||
"yes" if description else "no", v)
|
||||
return {"ok": True, "face": _entry_to_dict(entry)}
|
||||
|
||||
|
||||
@router.post("/faces/{face_id}/capture")
|
||||
async def capture_to_face(face_id: int):
|
||||
"""Add a new sample (from the camera) to an existing face."""
|
||||
gallery = _get_gallery()
|
||||
if gallery is None:
|
||||
raise HTTPException(503, "Face gallery subsystem unavailable.")
|
||||
cam = _get_camera()
|
||||
if cam is None or not cam.is_running():
|
||||
raise HTTPException(409, "Camera is not running. Toggle Vision ON first.")
|
||||
jpeg = cam.get_fresh_frame(max_age_s=0.5, timeout_s=1.5)
|
||||
if not jpeg:
|
||||
raise HTTPException(409, "Camera has no frame yet.")
|
||||
try:
|
||||
fname = gallery.add_photo(face_id, jpeg)
|
||||
except FileNotFoundError as exc:
|
||||
raise HTTPException(404, str(exc))
|
||||
v = _bump_gallery_version()
|
||||
log.info("captured new photo for face_%d → %s (v.%d)", face_id, fname, v)
|
||||
return {"ok": True, "added": fname, "face": _entry_to_dict(gallery.get(face_id))}
|
||||
|
||||
|
||||
@router.post("/faces/{face_id}/upload")
|
||||
async def upload_to_face(face_id: int, files: list[UploadFile] = File(...)):
|
||||
"""Add one or more uploaded samples to an existing face."""
|
||||
gallery = _get_gallery()
|
||||
if gallery is None:
|
||||
raise HTTPException(503, "Face gallery subsystem unavailable.")
|
||||
if gallery.get(face_id) is None:
|
||||
raise HTTPException(404, f"face_{face_id} not found")
|
||||
added: list[str] = []
|
||||
for f in files:
|
||||
content = await f.read()
|
||||
_validate_image(content, f.filename)
|
||||
try:
|
||||
fname = gallery.add_photo(face_id, content)
|
||||
added.append(fname)
|
||||
except FileNotFoundError as exc:
|
||||
raise HTTPException(404, str(exc))
|
||||
v = _bump_gallery_version()
|
||||
log.info("uploaded %d photo(s) to face_%d (v.%d)", len(added), face_id, v)
|
||||
return {"ok": True, "added": added,
|
||||
"face": _entry_to_dict(gallery.get(face_id))}
|
||||
|
||||
|
||||
@router.post("/faces/{face_id}/rename")
|
||||
async def rename_face(face_id: int, payload: RenamePayload):
|
||||
gallery = _get_gallery()
|
||||
if gallery is None:
|
||||
raise HTTPException(503, "Face gallery subsystem unavailable.")
|
||||
try:
|
||||
gallery.rename(face_id, payload.name)
|
||||
except FileNotFoundError as exc:
|
||||
raise HTTPException(404, str(exc))
|
||||
v = _bump_gallery_version()
|
||||
log.info("renamed face_%d → %s (v.%d)", face_id,
|
||||
payload.name or "(unnamed)", v)
|
||||
return {"ok": True, "face": _entry_to_dict(gallery.get(face_id))}
|
||||
|
||||
|
||||
@router.post("/faces/{face_id}/describe")
|
||||
async def describe_face(face_id: int, payload: DescribePayload):
|
||||
"""Set / clear a face's free-text description. The description is
|
||||
folded into the Gemini primer turn so Gemini can reference it."""
|
||||
gallery = _get_gallery()
|
||||
if gallery is None:
|
||||
raise HTTPException(503, "Face gallery subsystem unavailable.")
|
||||
try:
|
||||
gallery.set_description(face_id, payload.description)
|
||||
except FileNotFoundError as exc:
|
||||
raise HTTPException(404, str(exc))
|
||||
v = _bump_gallery_version()
|
||||
log.info("described face_%d (%s, v.%d)", face_id,
|
||||
"set" if payload.description else "cleared", v)
|
||||
return {"ok": True, "face": _entry_to_dict(gallery.get(face_id))}
|
||||
|
||||
|
||||
@router.delete("/faces/{face_id}")
|
||||
async def delete_face(face_id: int):
|
||||
gallery = _get_gallery()
|
||||
if gallery is None:
|
||||
raise HTTPException(503, "Face gallery subsystem unavailable.")
|
||||
try:
|
||||
gallery.delete_face(face_id)
|
||||
except FileNotFoundError as exc:
|
||||
raise HTTPException(404, str(exc))
|
||||
v = _bump_gallery_version()
|
||||
log.info("deleted face_%d (v.%d)", face_id, v)
|
||||
return {"ok": True, "deleted": face_id}
|
||||
|
||||
|
||||
@router.delete("/faces/{face_id}/photo/{photo_name}")
|
||||
async def delete_photo(face_id: int, photo_name: str):
|
||||
gallery = _get_gallery()
|
||||
if gallery is None:
|
||||
raise HTTPException(503, "Face gallery subsystem unavailable.")
|
||||
# safe filename — only allow simple file names, no traversal
|
||||
if "/" in photo_name or ".." in photo_name or "\x00" in photo_name:
|
||||
raise HTTPException(400, "Invalid photo name.")
|
||||
try:
|
||||
gallery.delete_photo(face_id, photo_name)
|
||||
except FileNotFoundError as exc:
|
||||
raise HTTPException(404, str(exc))
|
||||
except ValueError as exc:
|
||||
raise HTTPException(400, str(exc))
|
||||
v = _bump_gallery_version()
|
||||
log.info("deleted photo %s from face_%d (v.%d)", photo_name, face_id, v)
|
||||
return {"ok": True, "deleted": photo_name}
|
||||
|
||||
|
||||
@router.get("/faces/{face_id}/photo/{photo_name}")
|
||||
async def get_photo(face_id: int, photo_name: str,
|
||||
download: int = Query(default=0)):
|
||||
"""Serve a single photo. Add ?download=1 for attachment disposition."""
|
||||
gallery = _get_gallery()
|
||||
if gallery is None:
|
||||
raise HTTPException(503, "Face gallery subsystem unavailable.")
|
||||
if "/" in photo_name or ".." in photo_name or "\x00" in photo_name:
|
||||
raise HTTPException(400, "Invalid photo name.")
|
||||
path = gallery.get_photo(face_id, photo_name)
|
||||
if path is None:
|
||||
raise HTTPException(404, "Photo not found.")
|
||||
media = "image/png" if path.suffix.lower() == ".png" else "image/jpeg"
|
||||
headers = {}
|
||||
if download:
|
||||
headers["Content-Disposition"] = (
|
||||
f'attachment; filename="face_{face_id}_{photo_name}"'
|
||||
)
|
||||
return FileResponse(path, media_type=media, headers=headers)
|
||||
|
||||
|
||||
@router.get("/faces/{face_id}/download.zip")
|
||||
async def download_face_zip(face_id: int):
|
||||
gallery = _get_gallery()
|
||||
if gallery is None:
|
||||
raise HTTPException(503, "Face gallery subsystem unavailable.")
|
||||
try:
|
||||
data = gallery.zip_face(face_id)
|
||||
except FileNotFoundError as exc:
|
||||
raise HTTPException(404, str(exc))
|
||||
return StreamingResponse(
|
||||
io.BytesIO(data),
|
||||
media_type="application/zip",
|
||||
headers={
|
||||
"Content-Disposition": f'attachment; filename="face_{face_id}.zip"',
|
||||
"Content-Length": str(len(data)),
|
||||
},
|
||||
)
|
||||
@ -23,6 +23,12 @@ router = APIRouter()
|
||||
RECORDS_INDEX = AUDIO_RECORDINGS_DIR / "records.json"
|
||||
_INDEX_LOCK = threading.Lock()
|
||||
|
||||
# Strong refs to fire-and-forget playback tasks. The event loop only keeps a
|
||||
# weak reference to tasks, so an unreferenced create_task() result can be
|
||||
# garbage-collected (cancelling playback) before it finishes. Mirror replay.py.
|
||||
import asyncio as _asyncio # noqa: E402
|
||||
_BG_TASKS: set[_asyncio.Task] = set()
|
||||
|
||||
|
||||
def _load_index() -> dict[str, Any]:
|
||||
if not RECORDS_INDEX.exists():
|
||||
@ -109,10 +115,65 @@ async def play_record(payload: RecordPlay):
|
||||
if not raw_path.exists():
|
||||
raise HTTPException(404, f"File not found: {raw_path.name}")
|
||||
|
||||
from Project.Sanad.main import audio_mgr
|
||||
import threading
|
||||
# Fire-and-forget on a DEDICATED daemon thread — NOT asyncio.to_thread.
|
||||
# to_thread runs on the shared default executor, which gets starved while
|
||||
# the dashboard services the live-voice child's reconnect chatter; that
|
||||
# delayed record playback by several seconds (clip silent, counter parked).
|
||||
# A dedicated thread starts immediately regardless of executor/event-loop
|
||||
# load. play_wav blocks for the clip duration and serves pause/stop via
|
||||
# _play_state; the UI stays responsive because this handler returns now.
|
||||
# Python keeps running threads alive, so no ref is needed to prevent GC.
|
||||
threading.Thread(
|
||||
target=audio_mgr.play_wav, args=(raw_path, payload.record_name),
|
||||
name="record-playback", daemon=True,
|
||||
).start()
|
||||
return {"ok": True, "record_name": payload.record_name,
|
||||
"file_kind": payload.file_kind, "path": str(raw_path)}
|
||||
|
||||
|
||||
@router.post("/pause")
|
||||
async def pause_playback():
|
||||
from Project.Sanad.main import audio_mgr
|
||||
return audio_mgr.pause_playback()
|
||||
|
||||
|
||||
@router.post("/resume")
|
||||
async def resume_playback():
|
||||
from Project.Sanad.main import audio_mgr
|
||||
return audio_mgr.resume_playback()
|
||||
|
||||
|
||||
@router.post("/seek")
|
||||
async def seek_playback(position_sec: float):
|
||||
"""Jump to a position (seconds) in the currently-playing clip — used by the
|
||||
waveform scrubber. No-op (ok=False) if nothing is playing."""
|
||||
from Project.Sanad.main import audio_mgr
|
||||
return audio_mgr.seek_playback(position_sec)
|
||||
|
||||
|
||||
@router.post("/stop")
|
||||
async def stop_playback():
|
||||
from Project.Sanad.main import audio_mgr
|
||||
import asyncio
|
||||
await asyncio.to_thread(audio_mgr.play_wav, raw_path)
|
||||
return {"ok": True, "record_name": payload.record_name, "file_kind": payload.file_kind, "path": str(raw_path)}
|
||||
await asyncio.to_thread(audio_mgr.stop_playback)
|
||||
return {"ok": True, "stopped": True}
|
||||
|
||||
|
||||
@router.get("/playback-status")
|
||||
async def playback_status():
|
||||
from Project.Sanad.main import audio_mgr
|
||||
return audio_mgr.playback_status()
|
||||
|
||||
|
||||
@router.post("/live-hold")
|
||||
async def set_live_hold(on: bool):
|
||||
"""Manual hold for the live-Gemini pause. on=True pauses the live voice and
|
||||
keeps it paused (records won't resume it) until on=False is sent. Default
|
||||
behaviour (on=False) is AUTO: records pause Gemini only for the clip."""
|
||||
from Project.Sanad.main import audio_mgr
|
||||
return {"live_hold": audio_mgr.set_live_voice_hold(on)}
|
||||
|
||||
|
||||
class RecordRename(BaseModel):
|
||||
@ -183,7 +244,12 @@ async def delete_record(payload: RecordDelete):
|
||||
deleted_files = []
|
||||
for fi in deleted_entry.get("files", {}).values():
|
||||
try:
|
||||
p = Path(fi.get("path", "")).resolve()
|
||||
# _resolve_path handles new-style basenames (resolved under
|
||||
# AUDIO_RECORDINGS_DIR) as well as legacy absolute paths.
|
||||
# A raw Path(basename) would resolve vs CWD and fall outside
|
||||
# base, so the relative_to guard would skip the unlink and the
|
||||
# WAV would be orphaned on disk. Mirror play_record/rename_record.
|
||||
p = _resolve_path(fi.get("path", "")).resolve()
|
||||
p.relative_to(base) # never delete files outside recordings dir
|
||||
except (ValueError, OSError):
|
||||
continue
|
||||
@ -194,3 +260,43 @@ async def delete_record(payload: RecordDelete):
|
||||
index["records"] = kept
|
||||
_save_index(index)
|
||||
return {"ok": True, "deleted": payload.record_name, "deleted_files": deleted_files}
|
||||
|
||||
|
||||
class RecordBulkDelete(BaseModel):
|
||||
record_names: list[str] | None = None
|
||||
all: bool = False
|
||||
|
||||
|
||||
@router.post("/delete-bulk")
|
||||
async def delete_bulk(payload: RecordBulkDelete):
|
||||
"""Delete many records in one call. all=True wipes every record; otherwise
|
||||
only those in record_names. Files are unlinked, guarded to the recordings
|
||||
dir (same safety as /delete)."""
|
||||
names = set(payload.record_names or [])
|
||||
with _INDEX_LOCK:
|
||||
index = _load_index()
|
||||
base = AUDIO_RECORDINGS_DIR.resolve()
|
||||
kept: list = []
|
||||
removed: list = []
|
||||
deleted_files = 0
|
||||
for r in index.get("records", []):
|
||||
if payload.all or r.get("record_name") in names:
|
||||
removed.append(r.get("record_name"))
|
||||
for fi in r.get("files", {}).values():
|
||||
try:
|
||||
p = _resolve_path(fi.get("path", "")).resolve()
|
||||
p.relative_to(base) # never delete outside recordings dir
|
||||
except (ValueError, OSError):
|
||||
continue
|
||||
if p.exists():
|
||||
try:
|
||||
p.unlink()
|
||||
deleted_files += 1
|
||||
except OSError:
|
||||
pass
|
||||
else:
|
||||
kept.append(r)
|
||||
index["records"] = kept
|
||||
_save_index(index)
|
||||
return {"ok": True, "deleted": removed, "deleted_count": len(removed),
|
||||
"deleted_files": deleted_files}
|
||||
|
||||
@ -21,6 +21,22 @@ log = get_logger("replay_route")
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
def _block_if_movement_armed():
|
||||
"""409 when locomotion movement is armed — arm motion (replay / teaching) is
|
||||
mutually exclusive with walking."""
|
||||
try:
|
||||
from Project.Sanad.main import loco_controller # type: ignore
|
||||
armed = loco_controller is not None and loco_controller.is_armed()
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception:
|
||||
return
|
||||
if armed:
|
||||
raise HTTPException(
|
||||
409, "Arm actions are disabled while movement is enabled. "
|
||||
"Disable movement in the Controller tab first.")
|
||||
|
||||
|
||||
# -- models --
|
||||
|
||||
class ReplayRequest(BaseModel):
|
||||
@ -94,6 +110,7 @@ _BG_TASKS: set[asyncio.Task] = set()
|
||||
async def test_replay(payload: ReplayRequest):
|
||||
"""Test-play a motion file at the given speed."""
|
||||
from Project.Sanad.main import arm
|
||||
_block_if_movement_armed()
|
||||
if arm.is_busy:
|
||||
raise HTTPException(409, "Arm is busy.")
|
||||
path = safe_path_under(MOTIONS_DIR, payload.name)
|
||||
@ -114,9 +131,16 @@ async def test_replay(payload: ReplayRequest):
|
||||
|
||||
@router.post("/cancel")
|
||||
async def cancel_replay():
|
||||
"""Stop the current replay — the smooth return-to-home runs as the
|
||||
final phase of the replay itself.
|
||||
|
||||
Matches g1_replay_v4_stable.py's behaviour: the play loop breaks on
|
||||
the cancel flag, then the same Run() function executes its
|
||||
return-home ramp + DisableSDK. No separate scheduling needed.
|
||||
"""
|
||||
from Project.Sanad.main import arm
|
||||
arm.cancel()
|
||||
return {"ok": True, "message": "Cancel signal sent."}
|
||||
return {"ok": True, "message": "Cancelled — returning to home pose smoothly."}
|
||||
|
||||
|
||||
@router.get("/status")
|
||||
@ -135,6 +159,7 @@ async def start_teaching(payload: TeachRequest):
|
||||
from Project.Sanad.main import teacher
|
||||
if teacher is None:
|
||||
raise HTTPException(503, "Teaching module not available.")
|
||||
_block_if_movement_armed()
|
||||
if teacher.is_recording:
|
||||
raise HTTPException(409, "Teaching session already active.")
|
||||
existing = MOTIONS_DIR / f"{payload.name}.jsonl"
|
||||
|
||||
@ -2,6 +2,7 @@
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
@ -9,6 +10,7 @@ from fastapi import APIRouter, HTTPException
|
||||
from pydantic import BaseModel
|
||||
|
||||
from Project.Sanad.config import SCRIPTS_DIR
|
||||
from Project.Sanad.core import persona as _persona
|
||||
from Project.Sanad.dashboard.routes._safe_io import (
|
||||
atomic_write_text, MAX_UPLOAD_BYTES,
|
||||
)
|
||||
@ -31,6 +33,8 @@ def _safe_path(name: str) -> Path:
|
||||
@router.get("/")
|
||||
async def list_scripts():
|
||||
SCRIPTS_DIR.mkdir(parents=True, exist_ok=True)
|
||||
active = _persona.active_persona_name()
|
||||
default = _persona.default_persona_name()
|
||||
items = []
|
||||
for p in sorted(SCRIPTS_DIR.iterdir(), key=lambda x: x.name.lower()):
|
||||
if not p.is_file():
|
||||
@ -40,8 +44,48 @@ async def list_scripts():
|
||||
"name": p.name,
|
||||
"size_bytes": st.st_size,
|
||||
"modified_at": datetime.fromtimestamp(st.st_mtime).isoformat(timespec="seconds"),
|
||||
"active": p.name == active, # the persona Gemini loads now
|
||||
"is_default": p.name == default, # the fallback (sanad_script.txt)
|
||||
})
|
||||
return {"path": str(SCRIPTS_DIR), "files": items}
|
||||
return {"path": str(SCRIPTS_DIR), "files": items,
|
||||
"active": active, "default": default}
|
||||
|
||||
|
||||
class ScriptActive(BaseModel):
|
||||
name: str | None = None # None / "" / the default name → revert to default
|
||||
restart: bool = False # also restart the live voice so it takes effect now
|
||||
|
||||
|
||||
@router.get("/active")
|
||||
async def get_active():
|
||||
"""Which persona Gemini will load, and the default it falls back to."""
|
||||
return {"active": _persona.active_persona_name(),
|
||||
"default": _persona.default_persona_name()}
|
||||
|
||||
|
||||
@router.post("/active")
|
||||
async def set_active(payload: ScriptActive):
|
||||
"""Select the persona script Gemini uses. With restart=true, the live voice
|
||||
session is bounced so the new persona takes effect immediately; otherwise it
|
||||
applies on the next voice (re)connect."""
|
||||
try:
|
||||
active = _persona.set_active_persona(payload.name)
|
||||
except FileNotFoundError:
|
||||
raise HTTPException(404, f"Script not found: {payload.name}")
|
||||
restarted = False
|
||||
if payload.restart:
|
||||
try:
|
||||
from Project.Sanad.main import live_sub
|
||||
if live_sub is not None and hasattr(live_sub, "start"):
|
||||
if hasattr(live_sub, "is_running") and live_sub.is_running():
|
||||
await asyncio.to_thread(live_sub.stop)
|
||||
await asyncio.sleep(1.5)
|
||||
await asyncio.to_thread(live_sub.start)
|
||||
restarted = True
|
||||
except Exception:
|
||||
pass # selection is saved regardless of restart success
|
||||
return {"ok": True, "active": active,
|
||||
"default": _persona.default_persona_name(), "restarted": restarted}
|
||||
|
||||
|
||||
class ScriptLoad(BaseModel):
|
||||
@ -116,5 +160,9 @@ async def delete_script(payload: ScriptDelete):
|
||||
path = _safe_path(payload.name)
|
||||
if not path.exists():
|
||||
raise HTTPException(404, f"Not found: {payload.name}")
|
||||
if path.name == _persona.default_persona_name():
|
||||
raise HTTPException(409, f"Cannot delete the default persona ({path.name}).")
|
||||
path.unlink()
|
||||
# If the active selection was the deleted file, resolution auto-falls-back
|
||||
# to the default — no extra cleanup needed.
|
||||
return {"ok": True, "deleted": payload.name}
|
||||
|
||||
@ -5,18 +5,24 @@ from __future__ import annotations
|
||||
import asyncio
|
||||
import os
|
||||
import platform
|
||||
import shutil
|
||||
import socket
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from fastapi import APIRouter
|
||||
from fastapi import APIRouter, HTTPException
|
||||
from pydantic import BaseModel
|
||||
|
||||
from Project.Sanad.config import (
|
||||
AUDIO_RECORDINGS_DIR,
|
||||
BASE_DIR,
|
||||
DASHBOARD_HOST,
|
||||
DASHBOARD_INTERFACE,
|
||||
DASHBOARD_PORT,
|
||||
DATA_DIR,
|
||||
DDS_NETWORK_INTERFACE,
|
||||
LOGS_DIR,
|
||||
list_network_interfaces,
|
||||
)
|
||||
from Project.Sanad.core.logger import get_logger
|
||||
@ -26,6 +32,36 @@ log = get_logger("system_route")
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
def _runtime_bind() -> tuple[str, int]:
|
||||
"""The host/port the server is ACTUALLY bound to.
|
||||
|
||||
main.py launches `uvicorn.run(_app, host=args.host, port=args.port)` with
|
||||
the CLI --host/--port (start_sanad.sh passes `--port $PORT`, default 8001),
|
||||
which can differ from the import-time DASHBOARD_HOST/DASHBOARD_PORT config
|
||||
defaults (port 8000). Reading the live argv reports the real URL instead of
|
||||
a stale config value. Falls back to the config constants when an arg is
|
||||
absent (e.g. argparse default in effect)."""
|
||||
host = DASHBOARD_HOST
|
||||
port = DASHBOARD_PORT
|
||||
argv = sys.argv
|
||||
for i, tok in enumerate(argv):
|
||||
if tok == "--host" and i + 1 < len(argv):
|
||||
host = argv[i + 1]
|
||||
elif tok.startswith("--host="):
|
||||
host = tok.split("=", 1)[1]
|
||||
elif tok == "--port" and i + 1 < len(argv):
|
||||
try:
|
||||
port = int(argv[i + 1])
|
||||
except (TypeError, ValueError):
|
||||
pass
|
||||
elif tok.startswith("--port="):
|
||||
try:
|
||||
port = int(tok.split("=", 1)[1])
|
||||
except (TypeError, ValueError):
|
||||
pass
|
||||
return host, port
|
||||
|
||||
|
||||
def _safe_status(component, name: str) -> dict[str, Any]:
|
||||
if component is None:
|
||||
return {"available": False}
|
||||
@ -90,8 +126,9 @@ async def system_info():
|
||||
except Exception:
|
||||
interfaces = []
|
||||
|
||||
# Determine the URL the dashboard is reachable at
|
||||
bound_host = DASHBOARD_HOST
|
||||
# Determine the URL the dashboard is reachable at — use the ACTUAL
|
||||
# runtime bind args (argv), not the import-time config defaults.
|
||||
bound_host, bound_port = _runtime_bind()
|
||||
if bound_host == "0.0.0.0":
|
||||
# Try to find the wlan0 IP for display purposes
|
||||
up_ifaces = [i for i in interfaces if i["is_up"] and i["ip"] and not i["ip"].startswith("127.")]
|
||||
@ -112,8 +149,8 @@ async def system_info():
|
||||
"interface": DASHBOARD_INTERFACE,
|
||||
"bound_host": bound_host,
|
||||
"display_host": display_host,
|
||||
"port": DASHBOARD_PORT,
|
||||
"url": f"http://{display_host}:{DASHBOARD_PORT}",
|
||||
"port": bound_port,
|
||||
"url": f"http://{display_host}:{bound_port}",
|
||||
},
|
||||
"dds": {
|
||||
"interface": DDS_NETWORK_INTERFACE,
|
||||
@ -131,3 +168,148 @@ async def system_info():
|
||||
}
|
||||
|
||||
return await asyncio.to_thread(_do)
|
||||
|
||||
|
||||
# ───────────────────── storage tracking + cleanup ─────────────────────
|
||||
# Categories surfaced in the Settings → Storage panel. `cleanable` ones get a
|
||||
# Clean button + are included in "Clean all"; the rest (faces/motions/zones)
|
||||
# are shown for tracking only — they're operational assets (enrollments,
|
||||
# motion configs) managed in their own tabs, not disposable clutter.
|
||||
_STORAGE_CATS = [
|
||||
("recordings", "Conversation recordings", DATA_DIR / "recordings", True),
|
||||
("records", "Named records (Typed Replay)", AUDIO_RECORDINGS_DIR, True),
|
||||
("logs", "Logs", LOGS_DIR, True),
|
||||
("faces", "Enrolled faces", DATA_DIR / "faces", False),
|
||||
("motions", "Motion replays + config", DATA_DIR / "motions", False),
|
||||
("photos", "Photos", DATA_DIR / "photos", False),
|
||||
("zones", "Vision zones", DATA_DIR / "zones", False),
|
||||
]
|
||||
_CLEANABLE = {k for k, _l, _p, c in _STORAGE_CATS if c}
|
||||
|
||||
|
||||
def _dir_stats(path: Path) -> tuple[int, int]:
|
||||
"""(total_bytes, file_count) of a dir tree. Missing dir → (0, 0)."""
|
||||
total, n = 0, 0
|
||||
try:
|
||||
for root, _dirs, files in os.walk(path):
|
||||
for f in files:
|
||||
try:
|
||||
total += os.path.getsize(os.path.join(root, f))
|
||||
n += 1
|
||||
except OSError:
|
||||
pass
|
||||
except Exception:
|
||||
pass
|
||||
return total, n
|
||||
|
||||
|
||||
def _human(b: float) -> str:
|
||||
f = float(b)
|
||||
for u in ("B", "KB", "MB", "GB", "TB"):
|
||||
if f < 1024 or u == "TB":
|
||||
return f"{f:.0f} {u}" if u == "B" else f"{f:.1f} {u}"
|
||||
f /= 1024
|
||||
return f"{f:.1f} TB"
|
||||
|
||||
|
||||
@router.get("/storage")
|
||||
async def storage_usage():
|
||||
"""Per-category data/log sizes + disk free, for the Storage panel."""
|
||||
def _do():
|
||||
cats = []
|
||||
for key, label, path, cleanable in _STORAGE_CATS:
|
||||
size, files = _dir_stats(Path(path))
|
||||
cats.append({
|
||||
"key": key, "label": label, "path": str(path),
|
||||
"size_bytes": size, "size_human": _human(size),
|
||||
"files": files, "cleanable": cleanable,
|
||||
})
|
||||
data_b, _ = _dir_stats(DATA_DIR)
|
||||
logs_b, _ = _dir_stats(LOGS_DIR)
|
||||
try:
|
||||
du = shutil.disk_usage(str(BASE_DIR))
|
||||
disk = {
|
||||
"free_human": _human(du.free), "total_human": _human(du.total),
|
||||
"used_pct": round(100.0 * (du.total - du.free) / du.total, 1),
|
||||
}
|
||||
except Exception:
|
||||
disk = {}
|
||||
return {
|
||||
"categories": cats,
|
||||
"data_bytes": data_b, "data_human": _human(data_b),
|
||||
"logs_human": _human(logs_b),
|
||||
"total_human": _human(data_b + logs_b),
|
||||
"disk": disk,
|
||||
}
|
||||
return await asyncio.to_thread(_do)
|
||||
|
||||
|
||||
class _CleanReq(BaseModel):
|
||||
target: str # recordings | records | logs | all
|
||||
|
||||
|
||||
def _clean_recordings() -> tuple[int, int]:
|
||||
d = DATA_DIR / "recordings"
|
||||
freed, n = 0, 0
|
||||
for f in list(d.glob("*.wav")) + [d / "index.json"]:
|
||||
if f.is_file():
|
||||
try:
|
||||
freed += f.stat().st_size
|
||||
f.unlink()
|
||||
n += 1
|
||||
except OSError:
|
||||
pass
|
||||
return n, freed
|
||||
|
||||
|
||||
def _clean_records() -> tuple[int, int]:
|
||||
d = AUDIO_RECORDINGS_DIR
|
||||
freed, n = 0, 0
|
||||
for f in list(d.glob("*.wav")) + [d / "records.json"]:
|
||||
if f.is_file():
|
||||
try:
|
||||
freed += f.stat().st_size
|
||||
f.unlink()
|
||||
n += 1
|
||||
except OSError:
|
||||
pass
|
||||
return n, freed
|
||||
|
||||
|
||||
def _clean_logs() -> tuple[int, int]:
|
||||
# Truncate (not delete) — active loggers hold append-mode handles, so
|
||||
# truncating to 0 clears content cleanly without losing the fd.
|
||||
freed, n = 0, 0
|
||||
for f in Path(LOGS_DIR).glob("*.log"):
|
||||
try:
|
||||
freed += f.stat().st_size
|
||||
open(f, "w").close()
|
||||
n += 1
|
||||
except OSError:
|
||||
pass
|
||||
return n, freed
|
||||
|
||||
|
||||
@router.post("/storage/clean")
|
||||
async def storage_clean(req: _CleanReq):
|
||||
"""Clean a disposable category (recordings | records | logs) or 'all'.
|
||||
Recordings/records are deleted; logs are truncated. Assets (faces, motions,
|
||||
zones) are never touched here."""
|
||||
t = (req.target or "").strip().lower()
|
||||
if t != "all" and t not in _CLEANABLE:
|
||||
raise HTTPException(400, f"target must be 'all' or one of {sorted(_CLEANABLE)}")
|
||||
|
||||
def _do():
|
||||
targets = ["recordings", "records", "logs"] if t == "all" else [t]
|
||||
fns = {"recordings": _clean_recordings, "records": _clean_records,
|
||||
"logs": _clean_logs}
|
||||
result, total = {}, 0
|
||||
for tg in targets:
|
||||
n, freed = fns[tg]()
|
||||
result[tg] = {"items": n, "freed_bytes": freed, "freed_human": _human(freed)}
|
||||
total += freed
|
||||
log.info("storage clean %s → freed %s", targets, _human(total))
|
||||
return {"ok": True, "cleaned": targets,
|
||||
"total_freed_bytes": total, "total_freed_human": _human(total),
|
||||
"result": result}
|
||||
return await asyncio.to_thread(_do)
|
||||
|
||||
81
dashboard/routes/temp_monitor.py
Normal file
81
dashboard/routes/temp_monitor.py
Normal file
@ -0,0 +1,81 @@
|
||||
"""REST endpoints backing the 3D motor-temperature dashboard (N1).
|
||||
|
||||
Serves the motor name/mesh mapping + thresholds, and a one-shot temperature
|
||||
snapshot (the front-end's initial fetch fallback). The live stream is over
|
||||
/ws/motor-temps (dashboard/websockets/motor_temps.py). The 3D view itself is
|
||||
the static page at /static/temp3d/index.html.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import time
|
||||
|
||||
from fastapi import APIRouter
|
||||
|
||||
from Project.Sanad.dashboard.temp_motor_map import (
|
||||
MOTOR_NAMES,
|
||||
MOTOR_TO_MESH,
|
||||
TEMP_HOT_THRESHOLD,
|
||||
TEMP_MAX,
|
||||
TEMP_MIN,
|
||||
TEMP_WARM_THRESHOLD,
|
||||
build_payload,
|
||||
)
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
def _get_arm():
|
||||
"""Lazy import — avoids a circular import on dashboard load."""
|
||||
try:
|
||||
from Project.Sanad.main import arm # type: ignore
|
||||
return arm
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
@router.get("/mapping")
|
||||
async def motor_mapping():
|
||||
"""Motor id → name / mesh map + the temperature gradient thresholds."""
|
||||
return {
|
||||
"motor_names": MOTOR_NAMES,
|
||||
"motor_to_mesh": MOTOR_TO_MESH,
|
||||
"thresholds": {
|
||||
"min": TEMP_MIN,
|
||||
"max": TEMP_MAX,
|
||||
"warm": TEMP_WARM_THRESHOLD,
|
||||
"hot": TEMP_HOT_THRESHOLD,
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
@router.get("/motors")
|
||||
async def motors_snapshot():
|
||||
"""One-shot motor temperature + position snapshot (Marcus payload shape)."""
|
||||
arm = _get_arm()
|
||||
temps: list = []
|
||||
positions: list = []
|
||||
if arm is not None:
|
||||
try:
|
||||
temps = arm.get_motor_temps()
|
||||
except Exception:
|
||||
temps = []
|
||||
try:
|
||||
positions = arm.get_current_q()
|
||||
except Exception:
|
||||
positions = []
|
||||
return build_payload(temps, positions, time.time())
|
||||
|
||||
|
||||
@router.get("/battery")
|
||||
async def battery_status():
|
||||
"""Live G1 battery (BMS) snapshot: state-of-charge %, voltage, current,
|
||||
charge/discharge status, pack temperature, cycles. `available=False` until
|
||||
the BMS topic (rt/lf/bmsstate) delivers its first message."""
|
||||
arm = _get_arm()
|
||||
if arm is None or not hasattr(arm, "get_battery"):
|
||||
return {"available": False}
|
||||
try:
|
||||
return arm.get_battery()
|
||||
except Exception:
|
||||
return {"available": False}
|
||||
@ -193,7 +193,7 @@ async def update_api_key(payload: ApiKeyPayload):
|
||||
raise HTTPException(500, f"Could not save config: {exc}")
|
||||
|
||||
# Hot-swap the in-memory module globals.
|
||||
# Both Project.Sanad.config AND Project.Sanad.voice.gemini_client
|
||||
# Both Project.Sanad.config AND Project.Sanad.gemini.client
|
||||
# have their OWN reference to GEMINI_API_KEY (the latter was created
|
||||
# at `from Project.Sanad.config import GEMINI_API_KEY` at import time).
|
||||
# Python's `from X import Y` binds a local name — updating config.Y
|
||||
@ -205,10 +205,10 @@ async def update_api_key(payload: ApiKeyPayload):
|
||||
log.exception("could not patch config.GEMINI_API_KEY")
|
||||
|
||||
try:
|
||||
import Project.Sanad.voice.gemini_client as _gc
|
||||
import Project.Sanad.gemini.client as _gc
|
||||
_gc.GEMINI_API_KEY = key
|
||||
except Exception:
|
||||
log.exception("could not patch gemini_client.GEMINI_API_KEY")
|
||||
log.exception("could not patch gemini.client.GEMINI_API_KEY")
|
||||
|
||||
# Disconnect any live session so reconnect uses the new key.
|
||||
from Project.Sanad.main import voice_client
|
||||
|
||||
597
dashboard/routes/zones.py
Normal file
597
dashboard/routes/zones.py
Normal file
@ -0,0 +1,597 @@
|
||||
"""Zones tab — zone → place → linked-faces management + "go here" destination.
|
||||
|
||||
Hierarchy (replaces the old flat places):
|
||||
Zone (name + description)
|
||||
└─ Place (name + description + optional reference photos + linked face ids)
|
||||
|
||||
Routes live under /api/zones. Toggle + CRUD changes write
|
||||
data/.recognition_state.json (the SAME file faces use); the Gemini child polls
|
||||
it at 1 Hz and re-primes / announces mid-session. The "go here" endpoints set a
|
||||
navigation target the robot will head to once N2 locomotion is wired — for now
|
||||
they just record the target and feed Gemini the place's reference.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import io
|
||||
from typing import Optional
|
||||
|
||||
from fastapi import APIRouter, File, HTTPException, Query, UploadFile
|
||||
from fastapi.responses import FileResponse, StreamingResponse
|
||||
from pydantic import BaseModel
|
||||
|
||||
from Project.Sanad.config import BASE_DIR
|
||||
from Project.Sanad.core.logger import get_logger
|
||||
from Project.Sanad.dashboard.routes._safe_io import check_upload_size
|
||||
from Project.Sanad.vision import recognition_state
|
||||
|
||||
log = get_logger("zones_routes")
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
STATE_PATH = BASE_DIR / "data" / ".recognition_state.json"
|
||||
|
||||
|
||||
# ── lazy subsystem accessors ────────────────────────────────
|
||||
|
||||
def _get_camera():
|
||||
try:
|
||||
from Project.Sanad.main import camera # type: ignore
|
||||
return camera
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
def _get_zone_gallery():
|
||||
try:
|
||||
from Project.Sanad.main import zone_gallery # type: ignore
|
||||
return zone_gallery
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
def _get_face_gallery():
|
||||
try:
|
||||
from Project.Sanad.main import gallery # type: ignore
|
||||
return gallery
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
def _require_zones():
|
||||
g = _get_zone_gallery()
|
||||
if g is None:
|
||||
raise HTTPException(503, "Zone gallery subsystem unavailable.")
|
||||
return g
|
||||
|
||||
|
||||
def _bump_zones_version() -> int:
|
||||
cur = recognition_state.read(STATE_PATH)
|
||||
v = cur.zones_version + 1
|
||||
recognition_state.mutate(STATE_PATH, zones_version=v)
|
||||
return v
|
||||
|
||||
|
||||
def _validate_image(content: bytes, filename: str | None = None) -> None:
|
||||
check_upload_size(content)
|
||||
if len(content) < 16:
|
||||
raise HTTPException(400, "Image too small / empty.")
|
||||
if not (content[:3] == b"\xff\xd8\xff" or content[:8] == b"\x89PNG\r\n\x1a\n"):
|
||||
raise HTTPException(400, f"Only JPEG/PNG accepted (got {filename or 'unknown'}).")
|
||||
|
||||
|
||||
def _safe_photo_name(name: str) -> None:
|
||||
if "/" in name or ".." in name or "\x00" in name:
|
||||
raise HTTPException(400, "Invalid photo name.")
|
||||
|
||||
|
||||
def _resolve_faces(face_ids: list[int]) -> list[dict]:
|
||||
"""Turn linked face ids into [{id, name}] using the face gallery."""
|
||||
g = _get_face_gallery()
|
||||
out = []
|
||||
for fid in face_ids:
|
||||
name = None
|
||||
if g is not None:
|
||||
try:
|
||||
e = g.get(fid)
|
||||
name = e.name if e else None
|
||||
except Exception:
|
||||
name = None
|
||||
out.append({"id": fid, "name": name})
|
||||
return out
|
||||
|
||||
|
||||
def _place_to_dict(p) -> dict:
|
||||
d = p.to_dict()
|
||||
d["faces"] = _resolve_faces(p.face_ids)
|
||||
return d
|
||||
|
||||
|
||||
def _zone_to_dict(z) -> dict:
|
||||
return {
|
||||
"id": z.id, "name": z.name, "description": z.description,
|
||||
"linked_map": getattr(z, "linked_map", None),
|
||||
"added_at": z.added_at,
|
||||
"places": [_place_to_dict(p) for p in z.places],
|
||||
}
|
||||
|
||||
|
||||
async def _maybe_drive_to_place(zone, place) -> Optional[dict]:
|
||||
"""If the place links a nav2 place AND its zone's map is the one currently
|
||||
localized, actually DRIVE there (arbiter-gated + arm arrival monitor).
|
||||
Returns the drive outcome, or None when the place isn't drivable (no link).
|
||||
Best-effort: never raises into the caller."""
|
||||
nav_place = getattr(place, "nav_place", None)
|
||||
linked_map = getattr(zone, "linked_map", None)
|
||||
if not nav_place or not linked_map:
|
||||
return None
|
||||
try:
|
||||
from Project.Sanad.dashboard.routes import navigation as navmod
|
||||
from Project.Sanad.dashboard.routes import _arbiter
|
||||
except Exception:
|
||||
return {"ok": False, "reason": "nav_unavailable"}
|
||||
client = getattr(navmod, "_CLIENT", None)
|
||||
if client is None:
|
||||
return {"ok": False, "reason": "nav_unavailable"}
|
||||
try:
|
||||
st = await asyncio.to_thread(client.status)
|
||||
body = st.as_dict() if hasattr(st, "as_dict") else dict(st)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
return {"ok": False, "reason": "status_error", "detail": str(exc)[:120]}
|
||||
if not body.get("bringup_alive"):
|
||||
return {"ok": False, "reason": "no_map"}
|
||||
# The robot can only drive in the currently-localized map. Require the
|
||||
# zone's linked map to match (compare on the sanitized .db stem).
|
||||
active = (body.get("active_map") or "").strip().lower()
|
||||
want = (linked_map or "").strip().lower()
|
||||
if want.endswith(".db"):
|
||||
want = want[:-3]
|
||||
if active and want and active != want:
|
||||
return {"ok": False, "reason": "wrong_map",
|
||||
"active": body.get("active_map"), "want": linked_map}
|
||||
if not _arbiter.acquire_nav():
|
||||
return {"ok": False, "reason": "manual_armed"}
|
||||
drive = await asyncio.to_thread(client.goto, nav_place)
|
||||
if isinstance(drive, dict) and not drive.get("ok", True):
|
||||
_arbiter.release_nav()
|
||||
return {"ok": False, "reason": "dispatch_failed", "detail": drive}
|
||||
try:
|
||||
from Project.Sanad.navigation.goal_monitor import arm_goal
|
||||
arm_goal(nav_place)
|
||||
except Exception:
|
||||
pass
|
||||
return {"ok": True, "resolved": nav_place}
|
||||
|
||||
|
||||
def _nav_target_dict(st, gallery) -> Optional[dict]:
|
||||
zid, pid = st.nav_target_zone_id, st.nav_target_place_id
|
||||
if not zid or not pid:
|
||||
return None
|
||||
zone_name = place_name = None
|
||||
if gallery is not None:
|
||||
try:
|
||||
z = gallery.get_zone(zid)
|
||||
zone_name = z.name if z else None
|
||||
p = gallery.get_place(zid, pid)
|
||||
place_name = p.name if p else None
|
||||
except Exception:
|
||||
pass
|
||||
return {"zone_id": zid, "place_id": pid,
|
||||
"zone_name": zone_name, "place_name": place_name}
|
||||
|
||||
|
||||
# ── state + toggle ──────────────────────────────────────────
|
||||
|
||||
@router.get("/state")
|
||||
async def get_state():
|
||||
st = recognition_state.read(STATE_PATH)
|
||||
g = _get_zone_gallery()
|
||||
zones_count = places_count = 0
|
||||
if g is not None:
|
||||
try:
|
||||
zones = g.list_zones()
|
||||
zones_count = len(zones)
|
||||
places_count = sum(len(z.places) for z in zones)
|
||||
except Exception:
|
||||
pass
|
||||
return {
|
||||
"zone_rec_enabled": st.zone_rec_enabled,
|
||||
"zones_version": st.zones_version,
|
||||
"zones_count": zones_count,
|
||||
"places_count": places_count,
|
||||
"nav_target": _nav_target_dict(st, g),
|
||||
}
|
||||
|
||||
|
||||
@router.post("/zone-rec")
|
||||
async def set_zone_rec(on: bool = Query(...)):
|
||||
"""Enable / disable the robot's knowledge of zones & places (hot)."""
|
||||
st = recognition_state.mutate(STATE_PATH, zone_rec_enabled=bool(on))
|
||||
log.info("zone recognition %s", "ON" if on else "OFF")
|
||||
return {"ok": True, "zone_rec_enabled": st.zone_rec_enabled}
|
||||
|
||||
|
||||
@router.post("/sync")
|
||||
async def sync_zones():
|
||||
v = _bump_zones_version()
|
||||
log.info("zones sync requested → v.%d", v)
|
||||
return {"ok": True, "zones_version": v}
|
||||
|
||||
|
||||
# ── zones CRUD ──────────────────────────────────────────────
|
||||
|
||||
class NamePayload(BaseModel):
|
||||
name: Optional[str] = None
|
||||
|
||||
|
||||
class DescribePayload(BaseModel):
|
||||
description: Optional[str] = None
|
||||
|
||||
|
||||
class FacesPayload(BaseModel):
|
||||
face_ids: list[int] = []
|
||||
|
||||
|
||||
class LinkMapPayload(BaseModel):
|
||||
# nav2 map .db basename (e.g. "office.db"); None/"" unlinks.
|
||||
map: Optional[str] = None
|
||||
|
||||
|
||||
class NavPlacePayload(BaseModel):
|
||||
# nav2 place name in the zone's linked map; None/"" unlinks.
|
||||
nav_place: Optional[str] = None
|
||||
|
||||
|
||||
@router.get("")
|
||||
async def list_zones():
|
||||
g = _require_zones()
|
||||
zones = g.list_zones()
|
||||
return {"zones": [_zone_to_dict(z) for z in zones], "total": len(zones)}
|
||||
|
||||
|
||||
@router.post("/create")
|
||||
async def create_zone(name: Optional[str] = Query(default=None),
|
||||
description: Optional[str] = Query(default=None)):
|
||||
g = _require_zones()
|
||||
if not (name or "").strip() and not (description or "").strip():
|
||||
raise HTTPException(400, "A zone needs at least a name or a description.")
|
||||
z = g.create_zone(name=name, description=description)
|
||||
_bump_zones_version()
|
||||
return {"ok": True, "zone": _zone_to_dict(z)}
|
||||
|
||||
|
||||
@router.post("/{zone_id}/rename")
|
||||
async def rename_zone(zone_id: int, payload: NamePayload):
|
||||
g = _require_zones()
|
||||
try:
|
||||
g.rename_zone(zone_id, payload.name)
|
||||
except FileNotFoundError as exc:
|
||||
raise HTTPException(404, str(exc))
|
||||
_bump_zones_version()
|
||||
return {"ok": True, "zone": _zone_to_dict(g.get_zone(zone_id))}
|
||||
|
||||
|
||||
@router.post("/{zone_id}/describe")
|
||||
async def describe_zone(zone_id: int, payload: DescribePayload):
|
||||
g = _require_zones()
|
||||
try:
|
||||
g.describe_zone(zone_id, payload.description)
|
||||
except FileNotFoundError as exc:
|
||||
raise HTTPException(404, str(exc))
|
||||
_bump_zones_version()
|
||||
return {"ok": True, "zone": _zone_to_dict(g.get_zone(zone_id))}
|
||||
|
||||
|
||||
@router.delete("/{zone_id}")
|
||||
async def delete_zone(zone_id: int):
|
||||
g = _require_zones()
|
||||
try:
|
||||
g.delete_zone(zone_id)
|
||||
except FileNotFoundError as exc:
|
||||
raise HTTPException(404, str(exc))
|
||||
# If the active destination was inside this zone, clear it.
|
||||
st = recognition_state.read(STATE_PATH)
|
||||
if st.nav_target_zone_id == zone_id:
|
||||
recognition_state.mutate(STATE_PATH, nav_target_zone_id=0, nav_target_place_id=0)
|
||||
_bump_zones_version()
|
||||
return {"ok": True, "deleted": zone_id}
|
||||
|
||||
|
||||
# ── places CRUD (within a zone) ─────────────────────────────
|
||||
|
||||
@router.post("/{zone_id}/places/create")
|
||||
async def create_place(
|
||||
zone_id: int,
|
||||
name: Optional[str] = Query(default=None),
|
||||
description: Optional[str] = Query(default=None),
|
||||
face_ids: list[int] = Query(default=[]),
|
||||
nav_place: Optional[str] = Query(default=None),
|
||||
files: Optional[list[UploadFile]] = File(default=None),
|
||||
):
|
||||
g = _require_zones()
|
||||
if g.get_zone(zone_id) is None:
|
||||
raise HTTPException(404, f"zone_{zone_id} not found")
|
||||
if not (name or "").strip() and not (description or "").strip():
|
||||
raise HTTPException(400, "A place needs at least a name or a description.")
|
||||
image_bytes: list[bytes] = []
|
||||
for f in (files or []):
|
||||
content = await f.read()
|
||||
if not content:
|
||||
continue
|
||||
_validate_image(content, f.filename)
|
||||
image_bytes.append(content)
|
||||
p = g.create_place(zone_id, name=name, description=description,
|
||||
face_ids=face_ids, image_bytes_list=image_bytes or None,
|
||||
nav_place=nav_place)
|
||||
_bump_zones_version()
|
||||
return {"ok": True, "place": _place_to_dict(p)}
|
||||
|
||||
|
||||
@router.post("/{zone_id}/link_map")
|
||||
async def link_zone_map(zone_id: int, payload: LinkMapPayload):
|
||||
"""Bind (or unbind) the zone to a nav2 map .db. Required before its places
|
||||
can link to that map's nav places / before Gemini Nav can drive in it."""
|
||||
g = _require_zones()
|
||||
try:
|
||||
g.set_zone_map(zone_id, payload.map)
|
||||
except FileNotFoundError as exc:
|
||||
raise HTTPException(404, str(exc))
|
||||
_bump_zones_version()
|
||||
return {"ok": True, "zone": _zone_to_dict(g.get_zone(zone_id))}
|
||||
|
||||
|
||||
@router.post("/{zone_id}/places/{place_id}/nav_link")
|
||||
async def link_place_nav(zone_id: int, place_id: int, payload: NavPlacePayload):
|
||||
"""Link (or unlink) a place to a nav2 place name in the zone's map — this is
|
||||
what makes the place drivable from voice / 'Go here'."""
|
||||
g = _require_zones()
|
||||
try:
|
||||
g.set_place_nav(zone_id, place_id, payload.nav_place)
|
||||
except FileNotFoundError as exc:
|
||||
raise HTTPException(404, str(exc))
|
||||
_bump_zones_version()
|
||||
return {"ok": True, "place": _place_to_dict(g.get_place(zone_id, place_id))}
|
||||
|
||||
|
||||
@router.post("/{zone_id}/places/{place_id}/rename")
|
||||
async def rename_place(zone_id: int, place_id: int, payload: NamePayload):
|
||||
g = _require_zones()
|
||||
try:
|
||||
g.rename_place(zone_id, place_id, payload.name)
|
||||
except FileNotFoundError as exc:
|
||||
raise HTTPException(404, str(exc))
|
||||
_bump_zones_version()
|
||||
return {"ok": True, "place": _place_to_dict(g.get_place(zone_id, place_id))}
|
||||
|
||||
|
||||
@router.post("/{zone_id}/places/{place_id}/describe")
|
||||
async def describe_place(zone_id: int, place_id: int, payload: DescribePayload):
|
||||
g = _require_zones()
|
||||
try:
|
||||
g.describe_place(zone_id, place_id, payload.description)
|
||||
except FileNotFoundError as exc:
|
||||
raise HTTPException(404, str(exc))
|
||||
_bump_zones_version()
|
||||
return {"ok": True, "place": _place_to_dict(g.get_place(zone_id, place_id))}
|
||||
|
||||
|
||||
@router.post("/{zone_id}/places/{place_id}/faces")
|
||||
async def set_place_faces(zone_id: int, place_id: int, payload: FacesPayload):
|
||||
"""Replace the set of saved faces linked to this place."""
|
||||
g = _require_zones()
|
||||
try:
|
||||
g.set_place_faces(zone_id, place_id, payload.face_ids)
|
||||
except FileNotFoundError as exc:
|
||||
raise HTTPException(404, str(exc))
|
||||
_bump_zones_version()
|
||||
return {"ok": True, "place": _place_to_dict(g.get_place(zone_id, place_id))}
|
||||
|
||||
|
||||
@router.post("/{zone_id}/places/{place_id}/capture")
|
||||
async def capture_to_place(zone_id: int, place_id: int):
|
||||
g = _require_zones()
|
||||
cam = _get_camera()
|
||||
if cam is None or not cam.is_running():
|
||||
raise HTTPException(409, "Camera is not running. Toggle Vision ON first.")
|
||||
jpeg = cam.get_fresh_frame(max_age_s=0.5, timeout_s=1.5)
|
||||
if not jpeg:
|
||||
raise HTTPException(409, "Camera has no frame yet.")
|
||||
try:
|
||||
fname = g.add_photo(zone_id, place_id, jpeg)
|
||||
except FileNotFoundError as exc:
|
||||
raise HTTPException(404, str(exc))
|
||||
_bump_zones_version()
|
||||
return {"ok": True, "added": fname, "place": _place_to_dict(g.get_place(zone_id, place_id))}
|
||||
|
||||
|
||||
@router.post("/{zone_id}/places/{place_id}/upload")
|
||||
async def upload_to_place(zone_id: int, place_id: int,
|
||||
files: list[UploadFile] = File(...)):
|
||||
g = _require_zones()
|
||||
if g.get_place(zone_id, place_id) is None:
|
||||
raise HTTPException(404, f"zone_{zone_id}/place_{place_id} not found")
|
||||
added: list[str] = []
|
||||
for f in files:
|
||||
content = await f.read()
|
||||
_validate_image(content, f.filename)
|
||||
try:
|
||||
added.append(g.add_photo(zone_id, place_id, content))
|
||||
except FileNotFoundError as exc:
|
||||
raise HTTPException(404, str(exc))
|
||||
_bump_zones_version()
|
||||
return {"ok": True, "added": added, "place": _place_to_dict(g.get_place(zone_id, place_id))}
|
||||
|
||||
|
||||
@router.delete("/{zone_id}/places/{place_id}")
|
||||
async def delete_place(zone_id: int, place_id: int):
|
||||
g = _require_zones()
|
||||
try:
|
||||
g.delete_place(zone_id, place_id)
|
||||
except FileNotFoundError as exc:
|
||||
raise HTTPException(404, str(exc))
|
||||
st = recognition_state.read(STATE_PATH)
|
||||
if st.nav_target_zone_id == zone_id and st.nav_target_place_id == place_id:
|
||||
recognition_state.mutate(STATE_PATH, nav_target_zone_id=0, nav_target_place_id=0)
|
||||
_bump_zones_version()
|
||||
return {"ok": True, "deleted": place_id}
|
||||
|
||||
|
||||
@router.delete("/{zone_id}/places/{place_id}/photo/{photo_name}")
|
||||
async def delete_place_photo(zone_id: int, place_id: int, photo_name: str):
|
||||
g = _require_zones()
|
||||
_safe_photo_name(photo_name)
|
||||
try:
|
||||
g.delete_photo(zone_id, place_id, photo_name)
|
||||
except FileNotFoundError as exc:
|
||||
raise HTTPException(404, str(exc))
|
||||
_bump_zones_version()
|
||||
return {"ok": True, "deleted": photo_name}
|
||||
|
||||
|
||||
@router.get("/{zone_id}/places/{place_id}/photo/{photo_name}")
|
||||
async def get_place_photo(zone_id: int, place_id: int, photo_name: str,
|
||||
download: int = Query(default=0)):
|
||||
g = _require_zones()
|
||||
_safe_photo_name(photo_name)
|
||||
path = g.get_photo(zone_id, place_id, photo_name)
|
||||
if path is None:
|
||||
raise HTTPException(404, "Photo not found.")
|
||||
media = "image/png" if path.suffix.lower() == ".png" else "image/jpeg"
|
||||
headers = {}
|
||||
if download:
|
||||
headers["Content-Disposition"] = (
|
||||
f'attachment; filename="zone_{zone_id}_place_{place_id}_{photo_name}"')
|
||||
return FileResponse(path, media_type=media, headers=headers)
|
||||
|
||||
|
||||
@router.get("/{zone_id}/places/{place_id}/download.zip")
|
||||
async def download_place_zip(zone_id: int, place_id: int):
|
||||
g = _require_zones()
|
||||
try:
|
||||
data = g.zip_place(zone_id, place_id)
|
||||
except FileNotFoundError as exc:
|
||||
raise HTTPException(404, str(exc))
|
||||
return StreamingResponse(
|
||||
io.BytesIO(data), media_type="application/zip",
|
||||
headers={
|
||||
"Content-Disposition": f'attachment; filename="zone_{zone_id}_place_{place_id}.zip"',
|
||||
"Content-Length": str(len(data)),
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
# ── "go here" navigation target ─────────────────────────────
|
||||
|
||||
@router.post("/{zone_id}/places/{place_id}/go")
|
||||
async def go_to_place(zone_id: int, place_id: int):
|
||||
"""Set this place as the active destination AND, if the place links a nav2
|
||||
place in this zone's (currently-localized) map, actually drive there.
|
||||
|
||||
Two effects: (1) records nav_target so the Gemini child primes on the
|
||||
reference photo + announces the destination; (2) if drivable, dispatches a
|
||||
Nav2 goal (arbiter-gated, with arrival monitoring). A place with no nav link
|
||||
is announce-only, as before."""
|
||||
g = _require_zones()
|
||||
p = g.get_place(zone_id, place_id)
|
||||
if p is None:
|
||||
raise HTTPException(404, f"zone_{zone_id}/place_{place_id} not found")
|
||||
recognition_state.mutate(STATE_PATH,
|
||||
nav_target_zone_id=zone_id,
|
||||
nav_target_place_id=place_id)
|
||||
log.info("nav target set → zone_%d/place_%d (%s)", zone_id, place_id,
|
||||
p.name or "(unnamed)")
|
||||
zone = g.get_zone(zone_id)
|
||||
drive = await _maybe_drive_to_place(zone, p)
|
||||
return {"ok": True,
|
||||
"nav_target": {"zone_id": zone_id, "place_id": place_id,
|
||||
"place_name": p.name},
|
||||
"drive": drive}
|
||||
|
||||
|
||||
@router.post("/nav/clear")
|
||||
async def clear_nav_target():
|
||||
recognition_state.mutate(STATE_PATH, nav_target_zone_id=0, nav_target_place_id=0)
|
||||
log.info("nav target cleared")
|
||||
return {"ok": True, "nav_target": None}
|
||||
|
||||
|
||||
def _resolve_map_path(client, linked_map: str) -> Optional[str]:
|
||||
"""Find the .db path for a zone's linked map name via the nav client."""
|
||||
want = (linked_map or "").strip().lower()
|
||||
want_stem = want[:-3] if want.endswith(".db") else want
|
||||
try:
|
||||
maps = client.list_maps() or []
|
||||
except Exception:
|
||||
return None
|
||||
for m in maps:
|
||||
nm = (m.get("name") or "").strip().lower()
|
||||
if nm == want or (nm[:-3] if nm.endswith(".db") else nm) == want_stem:
|
||||
return m.get("path")
|
||||
return None
|
||||
|
||||
|
||||
@router.post("/{zone_id}/gemini_nav/start")
|
||||
async def gemini_nav_start(zone_id: int):
|
||||
"""Enter 'Gemini Nav' for a zone: localize the zone's map, turn on camera +
|
||||
face + zone recognition + movement, ensure the Gemini session is live, and
|
||||
greet the user so they can converse to navigate.
|
||||
|
||||
The robot only ever runs ONE map; this loads the zone's map in localize-only
|
||||
mode (so it cannot fresh-map while driving), exactly as the user requires.
|
||||
"""
|
||||
g = _require_zones()
|
||||
zone = g.get_zone(zone_id)
|
||||
if zone is None:
|
||||
raise HTTPException(404, f"zone_{zone_id} not found")
|
||||
linked_map = getattr(zone, "linked_map", None)
|
||||
if not linked_map:
|
||||
raise HTTPException(400, "This zone has no linked nav2 map — link one first.")
|
||||
|
||||
# 1) Localize the zone's map (single bringup, mode 3 — no fresh mapping).
|
||||
loaded: dict = {"ok": False, "reason": "nav_unavailable"}
|
||||
try:
|
||||
from Project.Sanad.dashboard.routes import navigation as navmod
|
||||
client = getattr(navmod, "_CLIENT", None)
|
||||
if client is not None:
|
||||
db_path = await asyncio.to_thread(_resolve_map_path, client, linked_map)
|
||||
if db_path:
|
||||
loaded = await asyncio.to_thread(client.load_map, db_path)
|
||||
else:
|
||||
loaded = {"ok": False, "reason": "map_not_found", "map": linked_map}
|
||||
except Exception as exc: # noqa: BLE001
|
||||
loaded = {"ok": False, "reason": "load_error", "detail": str(exc)[:160]}
|
||||
|
||||
# 2) Camera + face + zone recognition + movement ON for the session.
|
||||
recognition_state.mutate(STATE_PATH,
|
||||
vision_enabled=True, face_rec_enabled=True,
|
||||
zone_rec_enabled=True, movement_enabled=True)
|
||||
_bump_zones_version()
|
||||
|
||||
# 3) Ensure the Gemini session is live, then greet (zone + drivable places).
|
||||
session_started = False
|
||||
try:
|
||||
from Project.Sanad.main import live_sub
|
||||
if live_sub is not None:
|
||||
if hasattr(live_sub, "is_running") and not live_sub.is_running():
|
||||
await asyncio.to_thread(live_sub.start)
|
||||
session_started = True
|
||||
drivable = [p.name or p.nav_place for p in zone.places
|
||||
if getattr(p, "nav_place", None)]
|
||||
zname = zone.name or f"zone {zone_id}"
|
||||
if drivable:
|
||||
placelist = ", ".join(str(x) for x in drivable)
|
||||
greet = (f"You are now in the '{zname}' zone. You can drive the "
|
||||
f"user to: {placelist}. Greet the user warmly in your "
|
||||
f"normal Khaleeji style and ask where they would like to go.")
|
||||
else:
|
||||
greet = (f"You are now in the '{zname}' zone, but no drivable "
|
||||
f"places are linked to its map yet. Greet the user and "
|
||||
f"say places still need to be linked before you can drive.")
|
||||
if hasattr(live_sub, "send_state"):
|
||||
live_sub.send_state("nav_zone", greet)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
log.warning("gemini_nav greet failed: %s", exc)
|
||||
|
||||
return {"ok": True, "zone_id": zone_id, "zone": _zone_to_dict(zone),
|
||||
"loaded": loaded, "session_started": session_started}
|
||||
File diff suppressed because it is too large
Load Diff
1058
dashboard/static/temp3d/g1/g1_29dof_rev_1_0.urdf
Normal file
1058
dashboard/static/temp3d/g1/g1_29dof_rev_1_0.urdf
Normal file
File diff suppressed because it is too large
Load Diff
BIN
dashboard/static/temp3d/g1/meshes/head_link.STL
Normal file
BIN
dashboard/static/temp3d/g1/meshes/head_link.STL
Normal file
Binary file not shown.
BIN
dashboard/static/temp3d/g1/meshes/left_ankle_pitch_link.STL
Normal file
BIN
dashboard/static/temp3d/g1/meshes/left_ankle_pitch_link.STL
Normal file
Binary file not shown.
BIN
dashboard/static/temp3d/g1/meshes/left_ankle_roll_link.STL
Normal file
BIN
dashboard/static/temp3d/g1/meshes/left_ankle_roll_link.STL
Normal file
Binary file not shown.
BIN
dashboard/static/temp3d/g1/meshes/left_elbow_link.STL
Normal file
BIN
dashboard/static/temp3d/g1/meshes/left_elbow_link.STL
Normal file
Binary file not shown.
BIN
dashboard/static/temp3d/g1/meshes/left_hand_index_0_link.STL
Normal file
BIN
dashboard/static/temp3d/g1/meshes/left_hand_index_0_link.STL
Normal file
Binary file not shown.
BIN
dashboard/static/temp3d/g1/meshes/left_hand_index_1_link.STL
Normal file
BIN
dashboard/static/temp3d/g1/meshes/left_hand_index_1_link.STL
Normal file
Binary file not shown.
BIN
dashboard/static/temp3d/g1/meshes/left_hand_middle_0_link.STL
Normal file
BIN
dashboard/static/temp3d/g1/meshes/left_hand_middle_0_link.STL
Normal file
Binary file not shown.
BIN
dashboard/static/temp3d/g1/meshes/left_hand_middle_1_link.STL
Normal file
BIN
dashboard/static/temp3d/g1/meshes/left_hand_middle_1_link.STL
Normal file
Binary file not shown.
BIN
dashboard/static/temp3d/g1/meshes/left_hand_palm_link.STL
Normal file
BIN
dashboard/static/temp3d/g1/meshes/left_hand_palm_link.STL
Normal file
Binary file not shown.
BIN
dashboard/static/temp3d/g1/meshes/left_hand_thumb_0_link.STL
Normal file
BIN
dashboard/static/temp3d/g1/meshes/left_hand_thumb_0_link.STL
Normal file
Binary file not shown.
BIN
dashboard/static/temp3d/g1/meshes/left_hand_thumb_1_link.STL
Normal file
BIN
dashboard/static/temp3d/g1/meshes/left_hand_thumb_1_link.STL
Normal file
Binary file not shown.
BIN
dashboard/static/temp3d/g1/meshes/left_hand_thumb_2_link.STL
Normal file
BIN
dashboard/static/temp3d/g1/meshes/left_hand_thumb_2_link.STL
Normal file
Binary file not shown.
BIN
dashboard/static/temp3d/g1/meshes/left_hip_pitch_link.STL
Normal file
BIN
dashboard/static/temp3d/g1/meshes/left_hip_pitch_link.STL
Normal file
Binary file not shown.
BIN
dashboard/static/temp3d/g1/meshes/left_hip_roll_link.STL
Normal file
BIN
dashboard/static/temp3d/g1/meshes/left_hip_roll_link.STL
Normal file
Binary file not shown.
BIN
dashboard/static/temp3d/g1/meshes/left_hip_yaw_link.STL
Normal file
BIN
dashboard/static/temp3d/g1/meshes/left_hip_yaw_link.STL
Normal file
Binary file not shown.
BIN
dashboard/static/temp3d/g1/meshes/left_knee_link.STL
Normal file
BIN
dashboard/static/temp3d/g1/meshes/left_knee_link.STL
Normal file
Binary file not shown.
BIN
dashboard/static/temp3d/g1/meshes/left_rubber_hand.STL
Normal file
BIN
dashboard/static/temp3d/g1/meshes/left_rubber_hand.STL
Normal file
Binary file not shown.
BIN
dashboard/static/temp3d/g1/meshes/left_shoulder_pitch_link.STL
Normal file
BIN
dashboard/static/temp3d/g1/meshes/left_shoulder_pitch_link.STL
Normal file
Binary file not shown.
BIN
dashboard/static/temp3d/g1/meshes/left_shoulder_roll_link.STL
Normal file
BIN
dashboard/static/temp3d/g1/meshes/left_shoulder_roll_link.STL
Normal file
Binary file not shown.
BIN
dashboard/static/temp3d/g1/meshes/left_shoulder_yaw_link.STL
Normal file
BIN
dashboard/static/temp3d/g1/meshes/left_shoulder_yaw_link.STL
Normal file
Binary file not shown.
BIN
dashboard/static/temp3d/g1/meshes/left_wrist_pitch_link.STL
Normal file
BIN
dashboard/static/temp3d/g1/meshes/left_wrist_pitch_link.STL
Normal file
Binary file not shown.
BIN
dashboard/static/temp3d/g1/meshes/left_wrist_roll_link.STL
Normal file
BIN
dashboard/static/temp3d/g1/meshes/left_wrist_roll_link.STL
Normal file
Binary file not shown.
Binary file not shown.
BIN
dashboard/static/temp3d/g1/meshes/left_wrist_yaw_link.STL
Normal file
BIN
dashboard/static/temp3d/g1/meshes/left_wrist_yaw_link.STL
Normal file
Binary file not shown.
BIN
dashboard/static/temp3d/g1/meshes/logo_link.STL
Normal file
BIN
dashboard/static/temp3d/g1/meshes/logo_link.STL
Normal file
Binary file not shown.
BIN
dashboard/static/temp3d/g1/meshes/pelvis.STL
Normal file
BIN
dashboard/static/temp3d/g1/meshes/pelvis.STL
Normal file
Binary file not shown.
BIN
dashboard/static/temp3d/g1/meshes/pelvis_contour_link.STL
Normal file
BIN
dashboard/static/temp3d/g1/meshes/pelvis_contour_link.STL
Normal file
Binary file not shown.
BIN
dashboard/static/temp3d/g1/meshes/right_ankle_pitch_link.STL
Normal file
BIN
dashboard/static/temp3d/g1/meshes/right_ankle_pitch_link.STL
Normal file
Binary file not shown.
BIN
dashboard/static/temp3d/g1/meshes/right_ankle_roll_link.STL
Normal file
BIN
dashboard/static/temp3d/g1/meshes/right_ankle_roll_link.STL
Normal file
Binary file not shown.
BIN
dashboard/static/temp3d/g1/meshes/right_elbow_link.STL
Normal file
BIN
dashboard/static/temp3d/g1/meshes/right_elbow_link.STL
Normal file
Binary file not shown.
BIN
dashboard/static/temp3d/g1/meshes/right_hand_index_0_link.STL
Normal file
BIN
dashboard/static/temp3d/g1/meshes/right_hand_index_0_link.STL
Normal file
Binary file not shown.
BIN
dashboard/static/temp3d/g1/meshes/right_hand_index_1_link.STL
Normal file
BIN
dashboard/static/temp3d/g1/meshes/right_hand_index_1_link.STL
Normal file
Binary file not shown.
BIN
dashboard/static/temp3d/g1/meshes/right_hand_middle_0_link.STL
Normal file
BIN
dashboard/static/temp3d/g1/meshes/right_hand_middle_0_link.STL
Normal file
Binary file not shown.
BIN
dashboard/static/temp3d/g1/meshes/right_hand_middle_1_link.STL
Normal file
BIN
dashboard/static/temp3d/g1/meshes/right_hand_middle_1_link.STL
Normal file
Binary file not shown.
BIN
dashboard/static/temp3d/g1/meshes/right_hand_palm_link.STL
Normal file
BIN
dashboard/static/temp3d/g1/meshes/right_hand_palm_link.STL
Normal file
Binary file not shown.
BIN
dashboard/static/temp3d/g1/meshes/right_hand_thumb_0_link.STL
Normal file
BIN
dashboard/static/temp3d/g1/meshes/right_hand_thumb_0_link.STL
Normal file
Binary file not shown.
BIN
dashboard/static/temp3d/g1/meshes/right_hand_thumb_1_link.STL
Normal file
BIN
dashboard/static/temp3d/g1/meshes/right_hand_thumb_1_link.STL
Normal file
Binary file not shown.
BIN
dashboard/static/temp3d/g1/meshes/right_hand_thumb_2_link.STL
Normal file
BIN
dashboard/static/temp3d/g1/meshes/right_hand_thumb_2_link.STL
Normal file
Binary file not shown.
BIN
dashboard/static/temp3d/g1/meshes/right_hip_pitch_link.STL
Normal file
BIN
dashboard/static/temp3d/g1/meshes/right_hip_pitch_link.STL
Normal file
Binary file not shown.
BIN
dashboard/static/temp3d/g1/meshes/right_hip_roll_link.STL
Normal file
BIN
dashboard/static/temp3d/g1/meshes/right_hip_roll_link.STL
Normal file
Binary file not shown.
BIN
dashboard/static/temp3d/g1/meshes/right_hip_yaw_link.STL
Normal file
BIN
dashboard/static/temp3d/g1/meshes/right_hip_yaw_link.STL
Normal file
Binary file not shown.
BIN
dashboard/static/temp3d/g1/meshes/right_knee_link.STL
Normal file
BIN
dashboard/static/temp3d/g1/meshes/right_knee_link.STL
Normal file
Binary file not shown.
BIN
dashboard/static/temp3d/g1/meshes/right_rubber_hand.STL
Normal file
BIN
dashboard/static/temp3d/g1/meshes/right_rubber_hand.STL
Normal file
Binary file not shown.
BIN
dashboard/static/temp3d/g1/meshes/right_shoulder_pitch_link.STL
Normal file
BIN
dashboard/static/temp3d/g1/meshes/right_shoulder_pitch_link.STL
Normal file
Binary file not shown.
BIN
dashboard/static/temp3d/g1/meshes/right_shoulder_roll_link.STL
Normal file
BIN
dashboard/static/temp3d/g1/meshes/right_shoulder_roll_link.STL
Normal file
Binary file not shown.
BIN
dashboard/static/temp3d/g1/meshes/right_shoulder_yaw_link.STL
Normal file
BIN
dashboard/static/temp3d/g1/meshes/right_shoulder_yaw_link.STL
Normal file
Binary file not shown.
BIN
dashboard/static/temp3d/g1/meshes/right_wrist_pitch_link.STL
Normal file
BIN
dashboard/static/temp3d/g1/meshes/right_wrist_pitch_link.STL
Normal file
Binary file not shown.
BIN
dashboard/static/temp3d/g1/meshes/right_wrist_roll_link.STL
Normal file
BIN
dashboard/static/temp3d/g1/meshes/right_wrist_roll_link.STL
Normal file
Binary file not shown.
Binary file not shown.
BIN
dashboard/static/temp3d/g1/meshes/right_wrist_yaw_link.STL
Normal file
BIN
dashboard/static/temp3d/g1/meshes/right_wrist_yaw_link.STL
Normal file
Binary file not shown.
BIN
dashboard/static/temp3d/g1/meshes/torso_constraint_L_link.STL
Normal file
BIN
dashboard/static/temp3d/g1/meshes/torso_constraint_L_link.STL
Normal file
Binary file not shown.
Binary file not shown.
BIN
dashboard/static/temp3d/g1/meshes/torso_constraint_R_link.STL
Normal file
BIN
dashboard/static/temp3d/g1/meshes/torso_constraint_R_link.STL
Normal file
Binary file not shown.
Binary file not shown.
BIN
dashboard/static/temp3d/g1/meshes/torso_link.STL
Normal file
BIN
dashboard/static/temp3d/g1/meshes/torso_link.STL
Normal file
Binary file not shown.
BIN
dashboard/static/temp3d/g1/meshes/torso_link_23dof_rev_1_0.STL
Normal file
BIN
dashboard/static/temp3d/g1/meshes/torso_link_23dof_rev_1_0.STL
Normal file
Binary file not shown.
BIN
dashboard/static/temp3d/g1/meshes/torso_link_rev_1_0.STL
Normal file
BIN
dashboard/static/temp3d/g1/meshes/torso_link_rev_1_0.STL
Normal file
Binary file not shown.
BIN
dashboard/static/temp3d/g1/meshes/waist_constraint_L.STL
Normal file
BIN
dashboard/static/temp3d/g1/meshes/waist_constraint_L.STL
Normal file
Binary file not shown.
BIN
dashboard/static/temp3d/g1/meshes/waist_constraint_R.STL
Normal file
BIN
dashboard/static/temp3d/g1/meshes/waist_constraint_R.STL
Normal file
Binary file not shown.
BIN
dashboard/static/temp3d/g1/meshes/waist_roll_link.STL
Normal file
BIN
dashboard/static/temp3d/g1/meshes/waist_roll_link.STL
Normal file
Binary file not shown.
BIN
dashboard/static/temp3d/g1/meshes/waist_roll_link_rev_1_0.STL
Normal file
BIN
dashboard/static/temp3d/g1/meshes/waist_roll_link_rev_1_0.STL
Normal file
Binary file not shown.
BIN
dashboard/static/temp3d/g1/meshes/waist_support_link.STL
Normal file
BIN
dashboard/static/temp3d/g1/meshes/waist_support_link.STL
Normal file
Binary file not shown.
BIN
dashboard/static/temp3d/g1/meshes/waist_yaw_link.STL
Normal file
BIN
dashboard/static/temp3d/g1/meshes/waist_yaw_link.STL
Normal file
Binary file not shown.
BIN
dashboard/static/temp3d/g1/meshes/waist_yaw_link_rev_1_0.STL
Normal file
BIN
dashboard/static/temp3d/g1/meshes/waist_yaw_link_rev_1_0.STL
Normal file
Binary file not shown.
1228
dashboard/static/temp3d/index.html
Normal file
1228
dashboard/static/temp3d/index.html
Normal file
File diff suppressed because it is too large
Load Diff
Some files were not shown because too many files have changed in this diff Show More
Loading…
x
Reference in New Issue
Block a user