diff --git a/.gitignore b/.gitignore index c02264b..8439067 100644 --- a/.gitignore +++ b/.gitignore @@ -1,4 +1,13 @@ __pycache__/ *.pyc Logs/ +logs/ *.log + +# runtime state, not source +data/logins.json +data/audio/*.wav + +# local +.env +*.bak.* diff --git a/README.md b/README.md index da2ee7b..aafa923 100644 --- a/README.md +++ b/README.md @@ -1,246 +1,447 @@ # Sanad_lite -Multi-user, browser-audio fork of [Sanad](../Sanad/). The full Sanad robot -stack (arm, macros, camera, live conversation subprocess) was stripped out; -what remains is a small FastAPI dashboard for **typed-replay TTS** and -**saved-record management** where **all audio plays in each user's own -browser**, not on the host machine. +A small FastAPI dashboard that **speaks in the exact voices of the robots** — +Unitree G1, Unitree R1 and Agibot x2 — and lets you hold a **live spoken +conversation** with any of them from a browser. + +It is a stripped fork of [Sanad](../Sanad/): the arm, macros, camera, wake +words and on-robot audio are gone. What remains is text → speech, saved +recordings, and a realtime voice tab — with **all audio in the user's own +browser**, never on the host. + +Live at (cPanel host, see *Production*). ``` -┌────────────────────────────────────────────────────────────────────┐ -│ Dashboard (FastAPI) ── http://:8000 │ -│ ├─ /login Cookie-session auth │ -│ ├─ Voice & Audio Gemini API key, Typed Replay (TTS) │ -│ ├─ Recordings Saved WAVs — Play / Raw / Download / Del │ -│ │ plus "Delete All" │ -│ └─ Settings & Logs Scripts, system prompt, live log tail │ -└────────────────────────────────────────────────────────────────────┘ +┌──────────────────────────────────────────────────────────────────────┐ +│ Dashboard (FastAPI) │ +│ ├─ /login Cookie session + sign-in history │ +│ ├─ Voice & Audio Typed Replay — type text, hear the robot │ +│ ├─ Live Gemini Talk to a robot voice in realtime │ +│ ├─ Recordings Saved WAVs: search, filter by robot, play │ +│ └─ Settings & Logs Persona/rule files, log tail, sign-in log │ +└──────────────────────────────────────────────────────────────────────┘ ``` +| Robot | Gemini voice | Character | +|---|---|---| +| Unitree G1 | `Charon` | Sanad / Bousandah — male, Emirati | +| Unitree R1 | `Puck` | Super Dubai — male, Emirati | +| Agibot x2 | `Kore` | Muza — female, Emirati | -## Run on your laptop + +## The one thing this project is about + +The site must sound **identical** to the robots. Sanadv3 (G1), SanadR1 (R1) +and the Agibot stack each open a Gemini **Live** session and speak the reply +audio. If the dashboard renders the same sentence through a different engine — +even one that offers the same voice *name* — it does not sound like the robot. +Measured: the REST TTS model's "Charon" sits roughly **50 Hz above** the Live +one. + +So typed replay uses the robots' own path and refuses to quietly downgrade: + +``` +Gemini Live (same model, voice and prompt as the robot) ← up to 6 attempts + └─ each take is verified before it is served +REST TTS (gemini-2.5-flash-preview-tts) ← only if Live + returned NO audio +``` + +Every Live take is checked three ways before it is accepted +(`voice/typed_replay.py`): + +| Gate | What it catches | Default | +|---|---|---| +| **Coverage** — output transcript vs the text you typed | a take that stops mid-sentence | `1.0` (all of it) | +| **Believable duration** — chars ÷ seconds | audio too short to contain the words | `14` chars/s Arabic, `20` Latin | +| **Pitch band** — median F0 of the take | the wrong voice / wrong engine | Charon & Puck 100–140 Hz, Kore 165–225 Hz | + +A take that fails is retried. If no attempt passes, the **fullest Live take** +is served — the robot's voice with a missing word beats a clean read in the +wrong voice — and the UI shows a warning naming the gate that failed. If a take +is complete but ends early, up to 3 **continuations** are requested and joined. + +### Why words used to cut off + +The Live socket sends `generationComplete` *before* the last audio frames have +arrived; `turnComplete` is the real end of turn. Breaking on the first one +truncated tails and left stale frames in the socket for the next request to +pick up. `gemini/client.py` now breaks only on `turnComplete`, keeps a **3.0 s +post-generation grace** window, and drains the socket before closing. + +> The robots (Sanadv3 / SanadR1 / Sanad_Package_4) still carry the original +> `generationComplete` behaviour. Porting this fix to them has not been done. + +### Why every replay opens a fresh session + +`fresh_session_per_replay: true` in `config/voice_config.json`. A warm session +carries conversational context and the model drifts — same text, same voice, +different tone and a weaker Emirati accent. One session per replay removed the +drift. Voices that should keep a warm session can be listed in +`warm_session_voices`. + +`voice/pitch.py` is a dependency-free median-F0 estimator (autocorrelation over +48 frames, decimated to 8 kHz) — there is no numpy on the cPanel host. It was +validated to within ~4 Hz of a numpy reference, ~48 ms worst case. + + +## Quick start (laptop) ```bash -pip install --user \ - fastapi 'uvicorn[standard]' itsdangerous python-multipart pydantic \ - websockets +pip install --user fastapi 'uvicorn[standard]' itsdangerous python-multipart \ + pydantic websockets -cd /home/zedx/Robotics_workspace/yslootahtech/Project/Sanad_lite +cd Project/Other/Sanad_lite SANAD_DASHBOARD_HOST=127.0.0.1 python3 main.py ``` -Open and sign in with: +Open . Credentials live in `config/core_config.json` +under `auth` — see *Security* below. -> **Username:** `lkasjda213h` -> **Password:** `kj812bf@jdon` +Omit `SANAD_DASHBOARD_HOST` to auto-bind to `wlan0`'s IP so colleagues on the +LAN can reach `http://:8000`. -Setting `SANAD_DASHBOARD_HOST=127.0.0.1` keeps the server bound to -localhost; omit it to auto-bind to `wlan0`'s IP so colleagues on the LAN -can reach it at `http://:8000`. +**A Gemini API key is required and none ships with the repo.** +`gemini_defaults.api_key` in `config/core_config.json` is deliberately empty. +Supply one by: -The `websockets` package is needed because the Gemini Live TTS used by -Typed Replay opens a WebSocket to Google. Everything else (records list, -records delete-all, login, logs) works without it. +- pasting it in **Voice & Audio → Gemini API Key** (hot-swap, no restart), or +- `export SANAD_GEMINI_API_KEY=...` before starting, or +- filling `gemini_defaults.api_key`. -> **Gemini API key — required, none ships with the repo.** The `api_key` -> in `config/core_config.json` (`gemini_defaults`) is intentionally empty -> (`""`). Typed Replay / Gemini TTS won't work until you supply one: -> - paste it in the dashboard → **Voice & Audio → Gemini API Key** (hot-swap, no restart), **or** -> - `export SANAD_GEMINI_API_KEY=AIza...` before `python3 main.py`, **or** -> - set `gemini_defaults.api_key` in `config/core_config.json`. -> -> Get a key at . +Get one at . Keys beginning `AQ.` are AI +Studio express keys — they work, with one consequence for the Live tab (below). -> The other heavy deps (`pyaudio`, `transformers`, `torch`) are listed in -> `requirements.txt` but are **not required** for the lite dashboard. -> They were leftovers from the parent Sanad project and may still be -> imported lazily by `voice/audio_manager.py` / `voice/local_tts.py` -> on construction — failures are caught silently in `main.py`. +`pyaudio`, `torch` and `transformers` appear in `requirements.txt` but are +**not needed**; they are leftovers from full Sanad and their import failures +are swallowed at startup. -## Run on the server +## Where the audio actually plays -Replace the SSH/IP/path placeholders with your server's values: +Two different paths, and the distinction matters: -```bash -# 1. Install deps once on the server -ssh @ 'pip install itsdangerous fastapi "uvicorn[standard]" python-multipart pydantic websockets' +| Feature | Audio generated by | Audio played by | +|---|---|---| +| Typed Replay → Generate & Play | server (Gemini Live → WAV) | the viewer's browser | +| Typed Replay → Replay Last | cached WAV on the server | the viewer's browser | +| Recordings → Play / Raw / Download | server file | the viewer's browser | +| **Live Gemini** | **Google, direct to the browser** | the viewer's browser | -# 2. Push the lite tree -rsync -av --delete \ - --exclude=__pycache__ --exclude=logs --exclude=data \ - /home/zedx/Robotics_workspace/yslootahtech/Project/Sanad_lite/ \ - @:~/Sanad_lite/ - -# 3. Start it on the server (SSH in first, then run) -ssh @ -cd ~/Sanad_lite -python3 main.py -``` - -Then open `http://:8000` and sign in with **`lkasjda213h`** / -**`kj812bf@jdon`**. - -To leave it running after you log out, use `tmux`, `screen`, `nohup`, or -the systemd unit at `shell_scripts/sanad.service` (edit the paths inside -to match your install). +Server-side ALSA/PulseAudio is never touched. If you host on machine **A** and +a colleague opens `http://A:8000` from machine **B**, the sound comes out of +**B's** speakers. -## Login +## Live Gemini tab -Credentials are in `config/core_config.json`: -```json -"auth": { - "username": "lkasjda213h", - "password": "kj812bf@jdon" -} -``` +Pick a robot, press **Connect**, and talk. Speech goes to Gemini and the reply +comes back in that robot's voice, in realtime — the same interaction as Sanad +Package 5 on the G1, but in a browser. -Change them before any non-LAN deployment. The session cookie is signed -with a fresh secret each time `main.py` starts, so a restart logs every -user out. +### Why the browser talks to Google directly -For a stronger setup, replace the plaintext check with a bcrypt hash in -`dashboard/routes/auth.py`. +This deployment sits behind an Apache `[P]` rewrite that **cannot upgrade a +WebSocket** — measured: the same handshake answers `101` straight to uvicorn +and `404` through the proxy. A server-side relay is therefore impossible here, +so the page opens its own socket to Gemini. + +The intended way to do that is an **ephemeral auth token** minted by +`/api/live/token`, so the real key never reaches the page. That path is +implemented (`v1beta` + `access_token`) but Google **refuses tokens minted from +`AQ.` express keys** — verified in all four documented forms. With such a key, +set `live.allow_direct_key: true` in `config/voice_config.json` and the server +hands the key itself to the browser (`v1alpha` + `key`). + +> **While `allow_direct_key` is on, anyone who can sign in to the dashboard can +> read the Gemini API key.** It is on in this deployment, deliberately. Turn it +> off the moment the key becomes a standard `AIza...` key. + +### Personas + +A persona library, not a single editable prompt: + +- 3 built-ins in code (`dashboard/routes/live_personas.py`) — one per robot. +- Any number of saved personas in `data/live_personas.json`: **Save**, **Save + as new**, **Delete**, and **Use for this robot** to bind one to a voice. +- The active persona is shown above the picker and persists across restarts. +- Shipped saved personas: `unitree-g1`, `unitree-r1`, `agibot-x2`, plus `t800` + — an English-speaking persona written for the G1 voice. + +### Microphone and speaker selection + +Live audio broke on external speakers and headsets for a structural reason: an +`AudioContext` is bound to whichever output was default when it was created, +and `getUserMedia({audio:true})` takes the system default input. Plug a device +in afterwards and audio keeps going to the old one — silently, with no error. + +The tab now has explicit **Mic** and **Speaker** pickers: + +- Capture opens the chosen `deviceId` exactly, and can be switched mid-session + without dropping the connection. +- Playback is routed through a `MediaStreamAudioDestinationNode` into a hidden + `