Sanad_lite/README.md

# Sanad_lite

A small FastAPI dashboard that **speaks in the exact voices of the robots** —
Unitree G1, Unitree R1 and Agibot x2 — and lets you hold a **live spoken
conversation** with any of them from a browser.

It is a stripped fork of [Sanad](../Sanad/): the arm, macros, camera, wake
words and on-robot audio are gone. What remains is text → speech, saved
recordings, and a realtime voice tab — with **all audio in the user's own
browser**, never on the host.

Live at <https://sanadlite.yslootahtech.com> (cPanel host, see *Production*).

```
┌──────────────────────────────────────────────────────────────────────┐
│  Dashboard (FastAPI)                                                 │
│  ├─ /login              Cookie session + sign-in history             │
│  ├─ Voice & Audio       Typed Replay — type text, hear the robot     │
│  ├─ Live Gemini         Talk to a robot voice in realtime            │
│  ├─ Recordings          Saved WAVs: search, filter by robot, play    │
│  └─ Settings & Logs     Persona/rule files, log tail, sign-in log    │
└──────────────────────────────────────────────────────────────────────┘
```

| Robot | Gemini voice | Character |
|---|---|---|
| Unitree G1 | `Charon` | Sanad / Bousandah — male, Emirati |
| Unitree R1 | `Puck` | Super Dubai — male, Emirati |
| Agibot x2 | `Kore` | Muza — female, Emirati |


## The one thing this project is about

The site must sound **identical** to the robots. Sanadv3 (G1), SanadR1 (R1)
and the Agibot stack each open a Gemini **Live** session and speak the reply
audio. If the dashboard renders the same sentence through a different engine —
even one that offers the same voice *name* — it does not sound like the robot.
Measured: the REST TTS model's "Charon" sits roughly **50 Hz above** the Live
one.

So typed replay uses the robots' own path and refuses to quietly downgrade:

```
Gemini Live (same model, voice and prompt as the robot)   ← up to 6 attempts
   └─ each take is verified before it is served
REST TTS (gemini-2.5-flash-preview-tts)                   ← only if Live
                                                             returned NO audio
```

Every Live take is checked three ways before it is accepted
(`voice/typed_replay.py`):

| Gate | What it catches | Default |
|---|---|---|
| **Coverage** — output transcript vs the text you typed | a take that stops mid-sentence | `1.0` (all of it) |
| **Believable duration** — chars ÷ seconds | audio too short to contain the words | `14` chars/s Arabic, `20` Latin |
| **Pitch band** — median F0 of the take | the wrong voice / wrong engine | Charon & Puck 100–140 Hz, Kore 165–225 Hz |

A take that fails is retried. If no attempt passes, the **fullest Live take**
is served — the robot's voice with a missing word beats a clean read in the
wrong voice — and the UI shows a warning naming the gate that failed. If a take
is complete but ends early, up to 3 **continuations** are requested and joined.

### Why words used to cut off

The Live socket sends `generationComplete` *before* the last audio frames have
arrived; `turnComplete` is the real end of turn. Breaking on the first one
truncated tails and left stale frames in the socket for the next request to
pick up. `gemini/client.py` now breaks only on `turnComplete`, keeps a **3.0 s
post-generation grace** window, and drains the socket before closing.

> The robots (Sanadv3 / SanadR1 / Sanad_Package_4) still carry the original
> `generationComplete` behaviour. Porting this fix to them has not been done.

### Why every replay opens a fresh session

`fresh_session_per_replay: true` in `config/voice_config.json`. A warm session
carries conversational context and the model drifts — same text, same voice,
different tone and a weaker Emirati accent. One session per replay removed the
drift. Voices that should keep a warm session can be listed in
`warm_session_voices`.

`voice/pitch.py` is a dependency-free median-F0 estimator (autocorrelation over
48 frames, decimated to 8 kHz) — there is no numpy on the cPanel host. It was
validated to within ~4 Hz of a numpy reference, ~48 ms worst case.


## Quick start (laptop)

```bash
pip install --user fastapi 'uvicorn[standard]' itsdangerous python-multipart \
                   pydantic websockets

cd Project/Other/Sanad_lite
SANAD_DASHBOARD_HOST=127.0.0.1 python3 main.py
```

Open <http://127.0.0.1:8000>. Credentials live in `config/core_config.json`
under `auth` — see *Security* below.

Omit `SANAD_DASHBOARD_HOST` to auto-bind to `wlan0`'s IP so colleagues on the
LAN can reach `http://<your-ip>:8000`.

**A Gemini API key is required and none ships with the repo.**
`gemini_defaults.api_key` in `config/core_config.json` is deliberately empty.
Supply one by:

- pasting it in **Voice & Audio → Gemini API Key** (hot-swap, no restart), or
- `export SANAD_GEMINI_API_KEY=...` before starting, or
- filling `gemini_defaults.api_key`.

Get one at <https://aistudio.google.com/apikey>. Keys beginning `AQ.` are AI
Studio express keys — they work, with one consequence for the Live tab (below).

`pyaudio`, `torch` and `transformers` appear in `requirements.txt` but are
**not needed**; they are leftovers from full Sanad and their import failures
are swallowed at startup.


## Where the audio actually plays

Two different paths, and the distinction matters:

| Feature | Audio generated by | Audio played by |
|---|---|---|
| Typed Replay → Generate & Play | server (Gemini Live → WAV) | the viewer's browser |
| Typed Replay → Replay Last | cached WAV on the server | the viewer's browser |
| Recordings → Play / Raw / Download | server file | the viewer's browser |
| **Live Gemini** | **Google, direct to the browser** | the viewer's browser |

Server-side ALSA/PulseAudio is never touched. If you host on machine **A** and
a colleague opens `http://A:8000` from machine **B**, the sound comes out of
**B's** speakers.


## Live Gemini tab

Pick a robot, press **Connect**, and talk. Speech goes to Gemini and the reply
comes back in that robot's voice, in realtime — the same interaction as Sanad
Package 5 on the G1, but in a browser.

### Why the browser talks to Google directly

This deployment sits behind an Apache `[P]` rewrite that **cannot upgrade a
WebSocket** — measured: the same handshake answers `101` straight to uvicorn
and `404` through the proxy. A server-side relay is therefore impossible here,
so the page opens its own socket to Gemini.

The intended way to do that is an **ephemeral auth token** minted by
`/api/live/token`, so the real key never reaches the page. That path is
implemented (`v1beta` + `access_token`) but Google **refuses tokens minted from
`AQ.` express keys** — verified in all four documented forms. With such a key,
set `live.allow_direct_key: true` in `config/voice_config.json` and the server
hands the key itself to the browser (`v1alpha` + `key`).

> **While `allow_direct_key` is on, anyone who can sign in to the dashboard can
> read the Gemini API key.** It is on in this deployment, deliberately. Turn it
> off the moment the key becomes a standard `AIza...` key.

### Personas

A persona library, not a single editable prompt:

- 3 built-ins in code (`dashboard/routes/live_personas.py`) — one per robot.
- Any number of saved personas in `data/live_personas.json`: **Save**, **Save
  as new**, **Delete**, and **Use for this robot** to bind one to a voice.
- The active persona is shown above the picker and persists across restarts.
- Shipped saved personas: `unitree-g1`, `unitree-r1`, `agibot-x2`, plus `t800`
  — an English-speaking persona written for the G1 voice.

### Microphone and speaker selection

Live audio broke on external speakers and headsets for a structural reason: an
`AudioContext` is bound to whichever output was default when it was created,
and `getUserMedia({audio:true})` takes the system default input. Plug a device
in afterwards and audio keeps going to the old one — silently, with no error.

The tab now has explicit **Mic** and **Speaker** pickers:

- Capture opens the chosen `deviceId` exactly, and can be switched mid-session
  without dropping the connection.
- Playback is routed through a `MediaStreamAudioDestinationNode` into a hidden
  `<audio>` element so `setSinkId()` can move it to the chosen speaker.
- Both lists refresh on `devicechange`, and Windows' duplicate `default` /
  `communications` pseudo-devices are collapsed so each device appears once.
- **Allow microphone** re-requests permission if it was dismissed or denied;
  **Test speaker** plays a tone so output is proven before you spend credits.
- Choices are remembered in `localStorage`.

Capture uses an **AudioWorklet** (off the main thread; built from a Blob, so no
extra file has to be served) and falls back to `ScriptProcessorNode` where it
is unavailable. A compatibility line under the controls reports what the
current browser actually supports, and a diagnostics line shows
`sent / peak / received / played / mic ctx / out ctx / out sink / capture kind`
— enough to tell "not listening" from "not speaking" without a debugger.

### Browser support

Verified end to end with Playwright in both engines (`chromium` and `firefox`):
tab renders, 3 robots, model loaded, persona library populated, device lists
populated, test tone plays, **no page errors**. Also verified against real
hardware: `setSinkId` matched the selection, and a named microphone opened by
`deviceId`.

Caveats that are not code bugs:

- Bluetooth speakerphones switch from A2DP to HFP when a mic opens; the device
  may briefly disappear and return as a different endpoint. Prefer USB.
- Browser echo cancellation is weakest when output is routed to a non-default
  sink. A speakerphone with hardware AEC (e.g. Anker PowerConf) handles it; a
  plain speaker may echo — lower the volume or use the system default output.


## Recordings

Saved WAVs from **Save Last**, with:

- a **search box** matching name *or* the original text. Arabic is folded
  before matching (diacritics stripped, alef forms unified), so a search for
  `سلام` also matches `السلام` and `سَلام`;
- a **filter by robot** — G1 / R1 / Agibot — built from the `voice` recorded
  with each entry;
- a count line (`Showing N of M`), **Clear**, and per-record Play / Raw /
  Download / Rename / Delete, plus **Delete All**.

Each record stores `voice`, `voice_label`, `pitch_hz` and any `voice_warning`
from the gates, so an old file can be traced back to how it was produced.


## Settings & Logs

- Persona and rule script editor (`scripts/*.txt`).
- Live log tail. The WebSocket (`/ws/logs`) cannot cross the production proxy,
  so the UI falls back to cursor-based polling of `/api/logs/live`.
- **Sign-in history** — who signed in, when, from which device and IP
  (`data/logins.json`; the device string is parsed from the user agent, and the
  IP prefers `cf-connecting-ip` because the site is behind Cloudflare).
  Exportable as **CSV** (UTF-8 with BOM, so Excel renders Arabic) or **JSON**.


## Configuration

### Environment variables

| Var | Default | Effect |
|---|---|---|
| `SANAD_DASHBOARD_HOST` | `wlan0`'s IP | Bind address. `127.0.0.1` for localhost only. |
| `SANAD_DASHBOARD_INTERFACE` | `wlan0` | Which interface to take the IP from. |
| `SANAD_GEMINI_API_KEY` | *(empty)* | Gemini API key. |
| `PORT` | `8000` | Listen port. |

### Keys worth knowing

`config/core_config.json → gemini_defaults`

| Key | Meaning |
|---|---|
| `model_live` | Live model — `gemini-2.5-flash-native-audio-preview-12-2025`. |
| `voice_name` | Default voice (`Charon`). |
| `voice_options` | The robot ↔ voice table used by both tabs. |
| `default_system_prompt` | Emirati persona used for TTS. |
| `voice_system_prompts` | Per-voice overrides. **Charon and Puck are deliberately absent** so they keep the robots' verbatim prompt — that is what makes the site match Sanadv3/SanadR1. |
| `auth` | Dashboard username/password (cleartext). |

`config/voice_config.json`

| Key | Meaning |
|---|---|
| `typed_replay.fresh_session_per_replay` | One Live session per replay (tone stability). |
| `typed_replay.warm_session_voices` | Voices exempt from that. |
| `live.allow_direct_key` | Hand the API key to the browser for the Live tab. |

`config/gemini_config.json → client`

| Key | Meaning |
|---|---|
| `post_generation_grace_sec` | Wait after `generationComplete` for trailing audio (**3.0** — 1.2 was measurably too short and the cuts came back). |
| `output_transcription` | Ask for the spoken transcript, which the coverage gate needs. |
| `tts_preamble`, `tts_language_code` | Emirati steering for the REST fallback (`ar-AE`). |

Gate thresholds (`live_attempts`, `min_spoken_coverage`, `max_chars_per_sec_ar`,
`max_chars_per_sec_latin`, `max_continuations`, `pitch_gate`, `pitch_bands`,
`short_text_chars`) all live under `voice.typed_replay` and fall back to the
defaults in `voice/typed_replay.py`.


## API

```
GET    /                                  dashboard SPA (auth-gated, no-store)
GET    /login                             login page
POST   /api/auth/login                    {username,password} -> session cookie
POST   /api/auth/logout
GET    /api/auth/me
GET    /api/auth/logins?limit=20          sign-in history
GET    /api/auth/logins/export?format=csv|json

GET    /api/health                        {status, brain}
GET    /api/status                        brain + voice
GET    /api/system/info                   host / interfaces / subsystems

GET    /api/voice/status                  Gemini connection state
POST   /api/voice/connect | /disconnect
GET    /api/voice/api-key                 masked
POST   /api/voice/api-key                 {key} -> persisted, hot-swapped
GET    /api/voice/voices                  robot <-> voice table
POST   /api/voice/voice                   {voice} -> switch (reconnects)
POST   /api/voice/generate                legacy offline TTS route

POST   /api/typed-replay/say              {text,record,record_name}
GET    /api/typed-replay/audio/last       streams the cached WAV
POST   /api/typed-replay/replay-last
POST   /api/typed-replay/save-last
GET    /api/typed-replay/status           engine, session, voice, pitch, warning
GET    /api/typed-replay/records
GET    /api/typed-replay/records/{name}
POST   /api/typed-replay/records/{name}/rename
DELETE /api/typed-replay/records/{name}

GET    /api/records/                      saved records + voice metadata
GET    /api/records/audio/{name}?kind=speaker|raw
POST   /api/records/rename | /delete | /delete-all

GET    /api/live/config                   voices, model, active personas
GET    /api/live/personas
POST   /api/live/personas                 save / save-as-new
POST   /api/live/personas/select          bind a persona to a voice
POST   /api/live/personas/delete
POST   /api/live/token                    {voice} -> token + api_version + auth_param

GET    /api/scripts/                      persona/rule files
POST   /api/scripts/load | /save | /create | /rename | /delete
GET    /api/prompt/                       resolved system prompt
POST   /api/prompt/update | /reload
GET    /api/prompt/rule                   the Voice Rule text
POST   /api/prompt/rule

GET    /api/logs/                         available logs
GET    /api/logs/live?cursor=&limit=      polling tail (proxy-safe)
GET    /api/logs/tail/{filename}?lines=
POST   /api/logs/snapshot
GET    /api/logs/bundle                   zip of all logs
GET    /api/audio/*                       server-side device info (informational)
WS     /ws/logs                           live log stream (LAN only — see above)
```


## Layout

| Path | Contents |
|---|---|
| `main.py` | Entry point — boots subsystems, then the dashboard. |
| `config.py` | Runtime constants derived from `config/*.json`. |
| `config/` | `core`, `voice`, `gemini`, `dashboard` JSON. |
| `core/` | Brain (callback whitelist + status), skill registry, event bus, config loader, logger, asyncio compat. |
| `gemini/client.py` | Gemini Live WebSocket client + REST TTS fallback + quota detection. |
| `voice/typed_replay.py` | Generation, the three gates, continuations, records. |
| `voice/pitch.py` | Dependency-free median-F0 estimator. |
| `voice/` (rest) | `audio_manager.py`, `audio_devices.py`, `local_tts.py`, `text_utils.py` — vestigial on this deployment, degrade silently. |
| `dashboard/app.py` | FastAPI app, session middleware, auth gate, route registry, cache headers. |
| `dashboard/routes/` | `auth`, `health`, `system`, `voice`, `logs`, `audio_control`, `scripts`, `records`, `prompt`, `typed_replay`, `live`, `live_personas`. |
| `dashboard/static/` | `index.html` (SPA), `live.js` (browser Gemini client), `login.html`. |
| `dashboard/websockets/log_stream.py` | Log ring buffer + WS endpoint + `recent_since()` for polling. |
| `scripts/` | `sanad_script.txt` (persona), `sanad_rule.txt` (voice rule). |
| `data/audio/` | Saved WAVs + `records.json`. |
| `data/live_personas.json` | Persona library. |
| `data/logins.json` | Sign-in history (runtime, gitignored). |
| `shell_scripts/start_cpanel.sh` | Production launcher / self-heal (below). |
| `tests/test_smoke.py` | Brain whitelist, skill registry, atomic IO, audio devices, isolation. |


## Production (cPanel)

The host cannot run a normal service manager, so:

- **uvicorn** runs on `127.0.0.1:8000` from the CloudLinux virtualenv
  (`~/virtualenv/sanadlite/3.8`), started by `shell_scripts/start_cpanel.sh`.
- **Apache** reverse-proxies `https://sanadlite.yslootahtech.com` to it. A `[P]`
  rewrite cannot upgrade WebSockets — hence log polling and the browser-direct
  Live socket.
- **cron every 2 minutes** re-runs the launcher. It is idempotent and does a
  *real HTTP health check*, not a TCP probe: a wedged uvicorn with a frozen
  asyncio loop still holds the port and answers a TCP connect, so a
  connect-only check once reported "up" while the site was down for ~2 days. If
  the port is bound but `/api/health` does not answer, the frozen process is
  killed and relaunched.
- **Environment variables** set in cPanel's *Setup Python App* form are stored
  in `~/.cl.selector/python-selector.json` and are **not** re-exported by the
  venv's `activate`. The launcher parses and exports them itself — that is the
  store the Gemini API key must go into for a restart to keep it.
- `passenger_wsgi.py` is the alternative Passenger/WSGI entry point. It works,
  but WSGI has no WebSockets at all; the uvicorn path above is what runs.

Deployment is `scp` of changed files plus a launcher restart. Static assets are
served with a 7-day `max-age` and the page itself with `no-store`, so
`index.html` references `live.js?v=<mtime>` — **bump that query string when you
change a static file**, or browsers will keep the old one.


## Security

- `config/core_config.json → auth` holds the dashboard username and password in
  **cleartext**, and that file is committed. Change both before any deployment
  you do not control, and consider replacing the check in
  `dashboard/routes/auth.py` with a bcrypt hash.
- The session cookie is signed with a secret regenerated at every start, so a
  restart signs everyone out.
- `live.allow_direct_key` exposes the Gemini API key to any signed-in user.
- No API key is committed; `gemini_defaults.api_key` ships empty.


## Troubleshooting

| Symptom | Cause / fix |
|---|---|
| Replay sounds like a different voice | The REST fallback ran — Live returned no audio at all. Check `logs/` for `served REST TTS`. Usually quota, or a very short input. |
| "NOT the robot voice — … Hz outside …" warning | The pitch gate caught an off-voice take. Generate again; if it persists, the Live path is failing and everything is coming from REST. |
| Words cut off at the end | Should be fixed (`post_generation_grace_sec: 3.0`). If it returns, raise that value before touching anything else. |
| Tone/accent drifts between generations | A warm session leaked context — confirm `fresh_session_per_replay` is `true` and the voice is not in `warm_session_voices`. |
| Everything 503s with a quota message | `GeminiQuotaExhausted` — the key is out of credits. It fails fast and says so rather than retrying six times. |
| Live tab connects but never replies | Read the diagnostics line. `peak` near 0 = the mic is not being heard (wrong input device, or muted). `received` climbing but `played` flat = output routing. |
| Live tab: token minted but connect fails | An `AQ.` key with `allow_direct_key: false`. Ephemeral tokens do not work for those keys. |
| A UI fix "did not deploy" | Browser cache. Bump the `?v=` on the changed asset in `index.html`. |
| Arabic in the CSV export looks like mojibake | Open it through Excel's import dialog as UTF-8; the file already carries a BOM. |
| `ModuleNotFoundError: itsdangerous` / `websockets` | `pip install itsdangerous websockets` — required by SessionMiddleware and `gemini/client.py`. |
| `Failed to construct audio_mgr — pyaudio not installed` | Harmless. Nothing user-facing needs PyAudio. |
| Redirected to `/login` on every call | The server restarted and the cookie secret changed. Sign in again. |


## Known limits

- **Accent quality is not measured.** Pitch and coverage are verified
  automatically; how Emirati a take *sounds* is not. That needs a blind
  listening test.
- **Typed replay is single-user.** The session, the selected voice and
  `audio/last` are process globals — two people generating at once will
  interfere.
- **Very short inputs** (a single word) can still fall through to the REST
  engine, because the Live model sometimes returns text instead of audio for
  them.
- The `generationComplete` truncation fix has **not** been ported to the robots.


## Attribution

Internal project for **YS Lootah Robotics**. Trimmed from Sanad; the original
reuses patterns from `SanadVoice/gemini_interact` and Unitree `unitree_sdk2py`.