Device selection, cross-browser Live audio, and a rewritten README
Live Gemini could not use an external speaker or headset. An AudioContext
is bound to whichever output was default when it was created, and
getUserMedia({audio:true}) takes the system default input, so plugging a
device in afterwards left audio going to the old one — silently, with no
error to explain it.
* Explicit Mic and Speaker pickers. Capture opens the chosen deviceId
exactly and can be switched mid-session; playback is routed through a
MediaStreamAudioDestinationNode into a hidden <audio> element so
setSinkId() can move it to the chosen sink. Both lists refresh on
devicechange and are remembered in localStorage.
* Windows reports each device three times (default, communications, and
the real one), so an Anker would have appeared three times with no way
to tell them apart. The pseudo-devices are now collapsed.
* Capture moved to an AudioWorklet (Blob-built, no extra file served)
with the ScriptProcessor kept as a fallback.
* A compatibility line reports what the browser actually supports, and
the diagnostics line now shows the output sink and capture kind — the
difference between "not listening" and "not speaking" without a
debugger.
Verified in both engines with Playwright (chromium PASS, firefox PASS: no
page errors, personas and device lists populated, test tone plays) and
against real hardware, where setSinkId matched the selection and a named
microphone opened by deviceId. Two engine-specific bugs fell out of that
run and are fixed here: reading AudioContext.prototype.audioWorklet invokes
the getter and throws in both browsers, which aborted init and left the tab
empty, and the init steps are now isolated so one failure cannot take the
rest down.
Also in this commit:
* README rewritten against what the code does today — the voice-fidelity
rationale and its three gates, why words used to cut off, the Live tab
and why the browser talks to Google directly, personas, device
selection, recordings search/filter, sign-in history, a current API
list, the cPanel production setup, and the known limits.
* shell_scripts/start_cpanel.sh — the launcher that actually keeps the
site up (HTTP health check, not a TCP probe) was only on the host.
* data/live_personas.json — the persona library (G1, R1, Agibot, T800)
existed only on the server; it is user-written content worth keeping.
* .gitignore covers runtime state (logins, generated WAVs, .env, backups).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kassam Dakhlalah
parent
d9b2d5427f
commit
9b7ef7d939
9
.gitignore
vendored
9
.gitignore
vendored
@ -1,4 +1,13 @@
|
||||
__pycache__/
|
||||
*.pyc
|
||||
Logs/
|
||||
logs/
|
||||
*.log
|
||||
|
||||
# runtime state, not source
|
||||
data/logins.json
|
||||
data/audio/*.wav
|
||||
|
||||
# local
|
||||
.env
|
||||
*.bak.*
|
||||
|
||||
579
README.md
579
README.md
@ -1,246 +1,447 @@
|
||||
# Sanad_lite
|
||||
|
||||
Multi-user, browser-audio fork of [Sanad](../Sanad/). The full Sanad robot
|
||||
stack (arm, macros, camera, live conversation subprocess) was stripped out;
|
||||
what remains is a small FastAPI dashboard for **typed-replay TTS** and
|
||||
**saved-record management** where **all audio plays in each user's own
|
||||
browser**, not on the host machine.
|
||||
A small FastAPI dashboard that **speaks in the exact voices of the robots** —
|
||||
Unitree G1, Unitree R1 and Agibot x2 — and lets you hold a **live spoken
|
||||
conversation** with any of them from a browser.
|
||||
|
||||
It is a stripped fork of [Sanad](../Sanad/): the arm, macros, camera, wake
|
||||
words and on-robot audio are gone. What remains is text → speech, saved
|
||||
recordings, and a realtime voice tab — with **all audio in the user's own
|
||||
browser**, never on the host.
|
||||
|
||||
Live at <https://sanadlite.yslootahtech.com> (cPanel host, see *Production*).
|
||||
|
||||
```
|
||||
┌────────────────────────────────────────────────────────────────────┐
|
||||
│ Dashboard (FastAPI) ── http://<host>:8000 │
|
||||
│ ├─ /login Cookie-session auth │
|
||||
│ ├─ Voice & Audio Gemini API key, Typed Replay (TTS) │
|
||||
│ ├─ Recordings Saved WAVs — Play / Raw / Download / Del │
|
||||
│ │ plus "Delete All" │
|
||||
│ └─ Settings & Logs Scripts, system prompt, live log tail │
|
||||
└────────────────────────────────────────────────────────────────────┘
|
||||
┌──────────────────────────────────────────────────────────────────────┐
|
||||
│ Dashboard (FastAPI) │
|
||||
│ ├─ /login Cookie session + sign-in history │
|
||||
│ ├─ Voice & Audio Typed Replay — type text, hear the robot │
|
||||
│ ├─ Live Gemini Talk to a robot voice in realtime │
|
||||
│ ├─ Recordings Saved WAVs: search, filter by robot, play │
|
||||
│ └─ Settings & Logs Persona/rule files, log tail, sign-in log │
|
||||
└──────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
| Robot | Gemini voice | Character |
|
||||
|---|---|---|
|
||||
| Unitree G1 | `Charon` | Sanad / Bousandah — male, Emirati |
|
||||
| Unitree R1 | `Puck` | Super Dubai — male, Emirati |
|
||||
| Agibot x2 | `Kore` | Muza — female, Emirati |
|
||||
|
||||
## Run on your laptop
|
||||
|
||||
## The one thing this project is about
|
||||
|
||||
The site must sound **identical** to the robots. Sanadv3 (G1), SanadR1 (R1)
|
||||
and the Agibot stack each open a Gemini **Live** session and speak the reply
|
||||
audio. If the dashboard renders the same sentence through a different engine —
|
||||
even one that offers the same voice *name* — it does not sound like the robot.
|
||||
Measured: the REST TTS model's "Charon" sits roughly **50 Hz above** the Live
|
||||
one.
|
||||
|
||||
So typed replay uses the robots' own path and refuses to quietly downgrade:
|
||||
|
||||
```
|
||||
Gemini Live (same model, voice and prompt as the robot) ← up to 6 attempts
|
||||
└─ each take is verified before it is served
|
||||
REST TTS (gemini-2.5-flash-preview-tts) ← only if Live
|
||||
returned NO audio
|
||||
```
|
||||
|
||||
Every Live take is checked three ways before it is accepted
|
||||
(`voice/typed_replay.py`):
|
||||
|
||||
| Gate | What it catches | Default |
|
||||
|---|---|---|
|
||||
| **Coverage** — output transcript vs the text you typed | a take that stops mid-sentence | `1.0` (all of it) |
|
||||
| **Believable duration** — chars ÷ seconds | audio too short to contain the words | `14` chars/s Arabic, `20` Latin |
|
||||
| **Pitch band** — median F0 of the take | the wrong voice / wrong engine | Charon & Puck 100–140 Hz, Kore 165–225 Hz |
|
||||
|
||||
A take that fails is retried. If no attempt passes, the **fullest Live take**
|
||||
is served — the robot's voice with a missing word beats a clean read in the
|
||||
wrong voice — and the UI shows a warning naming the gate that failed. If a take
|
||||
is complete but ends early, up to 3 **continuations** are requested and joined.
|
||||
|
||||
### Why words used to cut off
|
||||
|
||||
The Live socket sends `generationComplete` *before* the last audio frames have
|
||||
arrived; `turnComplete` is the real end of turn. Breaking on the first one
|
||||
truncated tails and left stale frames in the socket for the next request to
|
||||
pick up. `gemini/client.py` now breaks only on `turnComplete`, keeps a **3.0 s
|
||||
post-generation grace** window, and drains the socket before closing.
|
||||
|
||||
> The robots (Sanadv3 / SanadR1 / Sanad_Package_4) still carry the original
|
||||
> `generationComplete` behaviour. Porting this fix to them has not been done.
|
||||
|
||||
### Why every replay opens a fresh session
|
||||
|
||||
`fresh_session_per_replay: true` in `config/voice_config.json`. A warm session
|
||||
carries conversational context and the model drifts — same text, same voice,
|
||||
different tone and a weaker Emirati accent. One session per replay removed the
|
||||
drift. Voices that should keep a warm session can be listed in
|
||||
`warm_session_voices`.
|
||||
|
||||
`voice/pitch.py` is a dependency-free median-F0 estimator (autocorrelation over
|
||||
48 frames, decimated to 8 kHz) — there is no numpy on the cPanel host. It was
|
||||
validated to within ~4 Hz of a numpy reference, ~48 ms worst case.
|
||||
|
||||
|
||||
## Quick start (laptop)
|
||||
|
||||
```bash
|
||||
pip install --user \
|
||||
fastapi 'uvicorn[standard]' itsdangerous python-multipart pydantic \
|
||||
websockets
|
||||
pip install --user fastapi 'uvicorn[standard]' itsdangerous python-multipart \
|
||||
pydantic websockets
|
||||
|
||||
cd /home/zedx/Robotics_workspace/yslootahtech/Project/Sanad_lite
|
||||
cd Project/Other/Sanad_lite
|
||||
SANAD_DASHBOARD_HOST=127.0.0.1 python3 main.py
|
||||
```
|
||||
|
||||
Open <http://127.0.0.1:8000> and sign in with:
|
||||
Open <http://127.0.0.1:8000>. Credentials live in `config/core_config.json`
|
||||
under `auth` — see *Security* below.
|
||||
|
||||
> **Username:** `lkasjda213h`
|
||||
> **Password:** `kj812bf@jdon`
|
||||
Omit `SANAD_DASHBOARD_HOST` to auto-bind to `wlan0`'s IP so colleagues on the
|
||||
LAN can reach `http://<your-ip>:8000`.
|
||||
|
||||
Setting `SANAD_DASHBOARD_HOST=127.0.0.1` keeps the server bound to
|
||||
localhost; omit it to auto-bind to `wlan0`'s IP so colleagues on the LAN
|
||||
can reach it at `http://<your-ip>:8000`.
|
||||
**A Gemini API key is required and none ships with the repo.**
|
||||
`gemini_defaults.api_key` in `config/core_config.json` is deliberately empty.
|
||||
Supply one by:
|
||||
|
||||
The `websockets` package is needed because the Gemini Live TTS used by
|
||||
Typed Replay opens a WebSocket to Google. Everything else (records list,
|
||||
records delete-all, login, logs) works without it.
|
||||
- pasting it in **Voice & Audio → Gemini API Key** (hot-swap, no restart), or
|
||||
- `export SANAD_GEMINI_API_KEY=...` before starting, or
|
||||
- filling `gemini_defaults.api_key`.
|
||||
|
||||
> **Gemini API key — required, none ships with the repo.** The `api_key`
|
||||
> in `config/core_config.json` (`gemini_defaults`) is intentionally empty
|
||||
> (`""`). Typed Replay / Gemini TTS won't work until you supply one:
|
||||
> - paste it in the dashboard → **Voice & Audio → Gemini API Key** (hot-swap, no restart), **or**
|
||||
> - `export SANAD_GEMINI_API_KEY=AIza...` before `python3 main.py`, **or**
|
||||
> - set `gemini_defaults.api_key` in `config/core_config.json`.
|
||||
>
|
||||
> Get a key at <https://aistudio.google.com/apikey>.
|
||||
Get one at <https://aistudio.google.com/apikey>. Keys beginning `AQ.` are AI
|
||||
Studio express keys — they work, with one consequence for the Live tab (below).
|
||||
|
||||
> The other heavy deps (`pyaudio`, `transformers`, `torch`) are listed in
|
||||
> `requirements.txt` but are **not required** for the lite dashboard.
|
||||
> They were leftovers from the parent Sanad project and may still be
|
||||
> imported lazily by `voice/audio_manager.py` / `voice/local_tts.py`
|
||||
> on construction — failures are caught silently in `main.py`.
|
||||
`pyaudio`, `torch` and `transformers` appear in `requirements.txt` but are
|
||||
**not needed**; they are leftovers from full Sanad and their import failures
|
||||
are swallowed at startup.
|
||||
|
||||
|
||||
## Run on the server
|
||||
## Where the audio actually plays
|
||||
|
||||
Replace the SSH/IP/path placeholders with your server's values:
|
||||
Two different paths, and the distinction matters:
|
||||
|
||||
```bash
|
||||
# 1. Install deps once on the server
|
||||
ssh <user>@<server-ip> 'pip install itsdangerous fastapi "uvicorn[standard]" python-multipart pydantic websockets'
|
||||
| Feature | Audio generated by | Audio played by |
|
||||
|---|---|---|
|
||||
| Typed Replay → Generate & Play | server (Gemini Live → WAV) | the viewer's browser |
|
||||
| Typed Replay → Replay Last | cached WAV on the server | the viewer's browser |
|
||||
| Recordings → Play / Raw / Download | server file | the viewer's browser |
|
||||
| **Live Gemini** | **Google, direct to the browser** | the viewer's browser |
|
||||
|
||||
# 2. Push the lite tree
|
||||
rsync -av --delete \
|
||||
--exclude=__pycache__ --exclude=logs --exclude=data \
|
||||
/home/zedx/Robotics_workspace/yslootahtech/Project/Sanad_lite/ \
|
||||
<user>@<server-ip>:~/Sanad_lite/
|
||||
|
||||
# 3. Start it on the server (SSH in first, then run)
|
||||
ssh <user>@<server-ip>
|
||||
cd ~/Sanad_lite
|
||||
python3 main.py
|
||||
```
|
||||
|
||||
Then open `http://<server-ip>:8000` and sign in with **`lkasjda213h`** /
|
||||
**`kj812bf@jdon`**.
|
||||
|
||||
To leave it running after you log out, use `tmux`, `screen`, `nohup`, or
|
||||
the systemd unit at `shell_scripts/sanad.service` (edit the paths inside
|
||||
to match your install).
|
||||
Server-side ALSA/PulseAudio is never touched. If you host on machine **A** and
|
||||
a colleague opens `http://A:8000` from machine **B**, the sound comes out of
|
||||
**B's** speakers.
|
||||
|
||||
|
||||
## Login
|
||||
## Live Gemini tab
|
||||
|
||||
Credentials are in `config/core_config.json`:
|
||||
```json
|
||||
"auth": {
|
||||
"username": "lkasjda213h",
|
||||
"password": "kj812bf@jdon"
|
||||
}
|
||||
```
|
||||
Pick a robot, press **Connect**, and talk. Speech goes to Gemini and the reply
|
||||
comes back in that robot's voice, in realtime — the same interaction as Sanad
|
||||
Package 5 on the G1, but in a browser.
|
||||
|
||||
Change them before any non-LAN deployment. The session cookie is signed
|
||||
with a fresh secret each time `main.py` starts, so a restart logs every
|
||||
user out.
|
||||
### Why the browser talks to Google directly
|
||||
|
||||
For a stronger setup, replace the plaintext check with a bcrypt hash in
|
||||
`dashboard/routes/auth.py`.
|
||||
This deployment sits behind an Apache `[P]` rewrite that **cannot upgrade a
|
||||
WebSocket** — measured: the same handshake answers `101` straight to uvicorn
|
||||
and `404` through the proxy. A server-side relay is therefore impossible here,
|
||||
so the page opens its own socket to Gemini.
|
||||
|
||||
The intended way to do that is an **ephemeral auth token** minted by
|
||||
`/api/live/token`, so the real key never reaches the page. That path is
|
||||
implemented (`v1beta` + `access_token`) but Google **refuses tokens minted from
|
||||
`AQ.` express keys** — verified in all four documented forms. With such a key,
|
||||
set `live.allow_direct_key: true` in `config/voice_config.json` and the server
|
||||
hands the key itself to the browser (`v1alpha` + `key`).
|
||||
|
||||
> **While `allow_direct_key` is on, anyone who can sign in to the dashboard can
|
||||
> read the Gemini API key.** It is on in this deployment, deliberately. Turn it
|
||||
> off the moment the key becomes a standard `AIza...` key.
|
||||
|
||||
### Personas
|
||||
|
||||
A persona library, not a single editable prompt:
|
||||
|
||||
- 3 built-ins in code (`dashboard/routes/live_personas.py`) — one per robot.
|
||||
- Any number of saved personas in `data/live_personas.json`: **Save**, **Save
|
||||
as new**, **Delete**, and **Use for this robot** to bind one to a voice.
|
||||
- The active persona is shown above the picker and persists across restarts.
|
||||
- Shipped saved personas: `unitree-g1`, `unitree-r1`, `agibot-x2`, plus `t800`
|
||||
— an English-speaking persona written for the G1 voice.
|
||||
|
||||
### Microphone and speaker selection
|
||||
|
||||
Live audio broke on external speakers and headsets for a structural reason: an
|
||||
`AudioContext` is bound to whichever output was default when it was created,
|
||||
and `getUserMedia({audio:true})` takes the system default input. Plug a device
|
||||
in afterwards and audio keeps going to the old one — silently, with no error.
|
||||
|
||||
The tab now has explicit **Mic** and **Speaker** pickers:
|
||||
|
||||
- Capture opens the chosen `deviceId` exactly, and can be switched mid-session
|
||||
without dropping the connection.
|
||||
- Playback is routed through a `MediaStreamAudioDestinationNode` into a hidden
|
||||
`<audio>` element so `setSinkId()` can move it to the chosen speaker.
|
||||
- Both lists refresh on `devicechange`, and Windows' duplicate `default` /
|
||||
`communications` pseudo-devices are collapsed so each device appears once.
|
||||
- **Allow microphone** re-requests permission if it was dismissed or denied;
|
||||
**Test speaker** plays a tone so output is proven before you spend credits.
|
||||
- Choices are remembered in `localStorage`.
|
||||
|
||||
Capture uses an **AudioWorklet** (off the main thread; built from a Blob, so no
|
||||
extra file has to be served) and falls back to `ScriptProcessorNode` where it
|
||||
is unavailable. A compatibility line under the controls reports what the
|
||||
current browser actually supports, and a diagnostics line shows
|
||||
`sent / peak / received / played / mic ctx / out ctx / out sink / capture kind`
|
||||
— enough to tell "not listening" from "not speaking" without a debugger.
|
||||
|
||||
### Browser support
|
||||
|
||||
Verified end to end with Playwright in both engines (`chromium` and `firefox`):
|
||||
tab renders, 3 robots, model loaded, persona library populated, device lists
|
||||
populated, test tone plays, **no page errors**. Also verified against real
|
||||
hardware: `setSinkId` matched the selection, and a named microphone opened by
|
||||
`deviceId`.
|
||||
|
||||
Caveats that are not code bugs:
|
||||
|
||||
- Bluetooth speakerphones switch from A2DP to HFP when a mic opens; the device
|
||||
may briefly disappear and return as a different endpoint. Prefer USB.
|
||||
- Browser echo cancellation is weakest when output is routed to a non-default
|
||||
sink. A speakerphone with hardware AEC (e.g. Anker PowerConf) handles it; a
|
||||
plain speaker may echo — lower the volume or use the system default output.
|
||||
|
||||
|
||||
## Audio architecture — who plays what, where
|
||||
## Recordings
|
||||
|
||||
| Action | Where audio plays |
|
||||
Saved WAVs from **Save Last**, with:
|
||||
|
||||
- a **search box** matching name *or* the original text. Arabic is folded
|
||||
before matching (diacritics stripped, alef forms unified), so a search for
|
||||
`سلام` also matches `السلام` and `سَلام`;
|
||||
- a **filter by robot** — G1 / R1 / Agibot — built from the `voice` recorded
|
||||
with each entry;
|
||||
- a count line (`Showing N of M`), **Clear**, and per-record Play / Raw /
|
||||
Download / Rename / Delete, plus **Delete All**.
|
||||
|
||||
Each record stores `voice`, `voice_label`, `pitch_hz` and any `voice_warning`
|
||||
from the gates, so an old file can be traced back to how it was produced.
|
||||
|
||||
|
||||
## Settings & Logs
|
||||
|
||||
- Persona and rule script editor (`scripts/*.txt`).
|
||||
- Live log tail. The WebSocket (`/ws/logs`) cannot cross the production proxy,
|
||||
so the UI falls back to cursor-based polling of `/api/logs/live`.
|
||||
- **Sign-in history** — who signed in, when, from which device and IP
|
||||
(`data/logins.json`; the device string is parsed from the user agent, and the
|
||||
IP prefers `cf-connecting-ip` because the site is behind Cloudflare).
|
||||
Exportable as **CSV** (UTF-8 with BOM, so Excel renders Arabic) or **JSON**.
|
||||
|
||||
|
||||
## Configuration
|
||||
|
||||
### Environment variables
|
||||
|
||||
| Var | Default | Effect |
|
||||
|---|---|---|
|
||||
| `SANAD_DASHBOARD_HOST` | `wlan0`'s IP | Bind address. `127.0.0.1` for localhost only. |
|
||||
| `SANAD_DASHBOARD_INTERFACE` | `wlan0` | Which interface to take the IP from. |
|
||||
| `SANAD_GEMINI_API_KEY` | *(empty)* | Gemini API key. |
|
||||
| `PORT` | `8000` | Listen port. |
|
||||
|
||||
### Keys worth knowing
|
||||
|
||||
`config/core_config.json → gemini_defaults`
|
||||
|
||||
| Key | Meaning |
|
||||
|---|---|
|
||||
| Recordings → **Play** | each viewing user's browser |
|
||||
| Recordings → **Raw** | each viewing user's browser |
|
||||
| Recordings → **Download** | saves WAV to viewing user's device |
|
||||
| Recordings → **Delete All** | wipes `data/audio/*.wav` on the server |
|
||||
| Voice & Audio → **Typed Replay → Generate & Play** | each viewing user's browser |
|
||||
| Voice & Audio → **Typed Replay → Replay Last** | each viewing user's browser |
|
||||
| `model_live` | Live model — `gemini-2.5-flash-native-audio-preview-12-2025`. |
|
||||
| `voice_name` | Default voice (`Charon`). |
|
||||
| `voice_options` | The robot ↔ voice table used by both tabs. |
|
||||
| `default_system_prompt` | Emirati persona used for TTS. |
|
||||
| `voice_system_prompts` | Per-voice overrides. **Charon and Puck are deliberately absent** so they keep the robots' verbatim prompt — that is what makes the site match Sanadv3/SanadR1. |
|
||||
| `auth` | Dashboard username/password (cleartext). |
|
||||
|
||||
Server-side ALSA / PulseAudio is **not** touched for any of the above.
|
||||
Both audio paths use the same pattern:
|
||||
`config/voice_config.json`
|
||||
|
||||
1. Server generates / loads the WAV bytes.
|
||||
2. Server returns them as `audio/wav` from an HTTP endpoint
|
||||
(`/api/records/audio/{name}` or `/api/typed-replay/audio/last`).
|
||||
3. Browser fetches the response into `new Audio(url)` and calls `.play()`.
|
||||
| Key | Meaning |
|
||||
|---|---|
|
||||
| `typed_replay.fresh_session_per_replay` | One Live session per replay (tone stability). |
|
||||
| `typed_replay.warm_session_voices` | Voices exempt from that. |
|
||||
| `live.allow_direct_key` | Hand the API key to the browser for the Live tab. |
|
||||
|
||||
So if you host the dashboard on machine **A** and a colleague on machine
|
||||
**B** opens `http://A:8000` and clicks Play, the sound comes out of **B's**
|
||||
speakers. Machine A stays silent.
|
||||
`config/gemini_config.json → client`
|
||||
|
||||
| Key | Meaning |
|
||||
|---|---|
|
||||
| `post_generation_grace_sec` | Wait after `generationComplete` for trailing audio (**3.0** — 1.2 was measurably too short and the cuts came back). |
|
||||
| `output_transcription` | Ask for the spoken transcript, which the coverage gate needs. |
|
||||
| `tts_preamble`, `tts_language_code` | Emirati steering for the REST fallback (`ar-AE`). |
|
||||
|
||||
Gate thresholds (`live_attempts`, `min_spoken_coverage`, `max_chars_per_sec_ar`,
|
||||
`max_chars_per_sec_latin`, `max_continuations`, `pitch_gate`, `pitch_bands`,
|
||||
`short_text_chars`) all live under `voice.typed_replay` and fall back to the
|
||||
defaults in `voice/typed_replay.py`.
|
||||
|
||||
|
||||
## Directory layout
|
||||
## API
|
||||
|
||||
```
|
||||
GET / dashboard SPA (auth-gated, no-store)
|
||||
GET /login login page
|
||||
POST /api/auth/login {username,password} -> session cookie
|
||||
POST /api/auth/logout
|
||||
GET /api/auth/me
|
||||
GET /api/auth/logins?limit=20 sign-in history
|
||||
GET /api/auth/logins/export?format=csv|json
|
||||
|
||||
GET /api/health {status, brain}
|
||||
GET /api/status brain + voice
|
||||
GET /api/system/info host / interfaces / subsystems
|
||||
|
||||
GET /api/voice/status Gemini connection state
|
||||
POST /api/voice/connect | /disconnect
|
||||
GET /api/voice/api-key masked
|
||||
POST /api/voice/api-key {key} -> persisted, hot-swapped
|
||||
GET /api/voice/voices robot <-> voice table
|
||||
POST /api/voice/voice {voice} -> switch (reconnects)
|
||||
POST /api/voice/generate legacy offline TTS route
|
||||
|
||||
POST /api/typed-replay/say {text,record,record_name}
|
||||
GET /api/typed-replay/audio/last streams the cached WAV
|
||||
POST /api/typed-replay/replay-last
|
||||
POST /api/typed-replay/save-last
|
||||
GET /api/typed-replay/status engine, session, voice, pitch, warning
|
||||
GET /api/typed-replay/records
|
||||
GET /api/typed-replay/records/{name}
|
||||
POST /api/typed-replay/records/{name}/rename
|
||||
DELETE /api/typed-replay/records/{name}
|
||||
|
||||
GET /api/records/ saved records + voice metadata
|
||||
GET /api/records/audio/{name}?kind=speaker|raw
|
||||
POST /api/records/rename | /delete | /delete-all
|
||||
|
||||
GET /api/live/config voices, model, active personas
|
||||
GET /api/live/personas
|
||||
POST /api/live/personas save / save-as-new
|
||||
POST /api/live/personas/select bind a persona to a voice
|
||||
POST /api/live/personas/delete
|
||||
POST /api/live/token {voice} -> token + api_version + auth_param
|
||||
|
||||
GET /api/scripts/ persona/rule files
|
||||
POST /api/scripts/load | /save | /create | /rename | /delete
|
||||
GET /api/prompt/ resolved system prompt
|
||||
POST /api/prompt/update | /reload
|
||||
GET /api/prompt/rule the Voice Rule text
|
||||
POST /api/prompt/rule
|
||||
|
||||
GET /api/logs/ available logs
|
||||
GET /api/logs/live?cursor=&limit= polling tail (proxy-safe)
|
||||
GET /api/logs/tail/{filename}?lines=
|
||||
POST /api/logs/snapshot
|
||||
GET /api/logs/bundle zip of all logs
|
||||
GET /api/audio/* server-side device info (informational)
|
||||
WS /ws/logs live log stream (LAN only — see above)
|
||||
```
|
||||
|
||||
|
||||
## Layout
|
||||
|
||||
| Path | Contents |
|
||||
|---|---|
|
||||
| `main.py` | Entry point — boots subsystems + dashboard. |
|
||||
| `config.py` | Runtime constants derived from `config/*_config.json`. |
|
||||
| `config/` | Per-subsystem JSON: `core`, `voice`, `gemini`, `dashboard`. |
|
||||
| `core/` | Brain (callback whitelist + status), skill registry, event bus, config loader, logger. |
|
||||
| `gemini/` | `client.py` — Gemini Live WebSocket client used by typed_replay for one-shot TTS calls. |
|
||||
| `voice/` | `typed_replay.py` (server generates, browser plays), `audio_manager.py` (host PyAudio — only used to share a PyAudio instance with typed_replay; degrades gracefully if PyAudio is missing), `local_tts.py` (offline SpeechT5 — unused in the lite UI but kept for the `/api/voice/generate` legacy route), `audio_devices.py`, `text_utils.py`. |
|
||||
| `dashboard/` | `app.py` (FastAPI + SessionMiddleware + auth gate), `routes/*.py`, `static/index.html`, `static/login.html`. |
|
||||
| `dashboard/routes/` | `auth.py`, `health.py`, `system.py`, `voice.py`, `logs.py`, `audio_control.py`, `scripts.py`, `records.py`, `prompt.py`, `typed_replay.py`, plus `websockets/log_stream.py`. |
|
||||
| `scripts/` | `sanad_script.txt` (persona), `sanad_rule.txt` (rules). |
|
||||
| `data/audio/` | Generated WAVs from Typed Replay → Save Last. Wiped by "Delete All". |
|
||||
| `data/motions/` | Persisted dashboard settings (Gemini API key, G1 volume) — back-compat path. |
|
||||
| `logs/` | Per-module rotating logs. |
|
||||
| `tests/` | `test_smoke.py` — Brain whitelist, skill registry, wake-phrase matching, atomic IO, audio devices, isolation. |
|
||||
| `main.py` | Entry point — boots subsystems, then the dashboard. |
|
||||
| `config.py` | Runtime constants derived from `config/*.json`. |
|
||||
| `config/` | `core`, `voice`, `gemini`, `dashboard` JSON. |
|
||||
| `core/` | Brain (callback whitelist + status), skill registry, event bus, config loader, logger, asyncio compat. |
|
||||
| `gemini/client.py` | Gemini Live WebSocket client + REST TTS fallback + quota detection. |
|
||||
| `voice/typed_replay.py` | Generation, the three gates, continuations, records. |
|
||||
| `voice/pitch.py` | Dependency-free median-F0 estimator. |
|
||||
| `voice/` (rest) | `audio_manager.py`, `audio_devices.py`, `local_tts.py`, `text_utils.py` — vestigial on this deployment, degrade silently. |
|
||||
| `dashboard/app.py` | FastAPI app, session middleware, auth gate, route registry, cache headers. |
|
||||
| `dashboard/routes/` | `auth`, `health`, `system`, `voice`, `logs`, `audio_control`, `scripts`, `records`, `prompt`, `typed_replay`, `live`, `live_personas`. |
|
||||
| `dashboard/static/` | `index.html` (SPA), `live.js` (browser Gemini client), `login.html`. |
|
||||
| `dashboard/websockets/log_stream.py` | Log ring buffer + WS endpoint + `recent_since()` for polling. |
|
||||
| `scripts/` | `sanad_script.txt` (persona), `sanad_rule.txt` (voice rule). |
|
||||
| `data/audio/` | Saved WAVs + `records.json`. |
|
||||
| `data/live_personas.json` | Persona library. |
|
||||
| `data/logins.json` | Sign-in history (runtime, gitignored). |
|
||||
| `shell_scripts/start_cpanel.sh` | Production launcher / self-heal (below). |
|
||||
| `tests/test_smoke.py` | Brain whitelist, skill registry, atomic IO, audio devices, isolation. |
|
||||
|
||||
|
||||
## Runtime env vars
|
||||
## Production (cPanel)
|
||||
|
||||
| Var | Values | Default | Effect |
|
||||
|---|---|---|---|
|
||||
| `SANAD_DASHBOARD_HOST` | IP or hostname | wlan0's IP | Override the bind address. Use `127.0.0.1` for localhost-only. |
|
||||
| `SANAD_DASHBOARD_INTERFACE` | iface name | `wlan0` | Pick which interface's IP to auto-bind to. |
|
||||
| `SANAD_GEMINI_API_KEY` | string | `""` (empty) | Gemini API key. No key ships in the repo — set this, paste one in the dashboard (**Voice & Audio → Gemini API Key**), or fill `gemini_defaults.api_key` in `config/core_config.json`. |
|
||||
The host cannot run a normal service manager, so:
|
||||
|
||||
- **uvicorn** runs on `127.0.0.1:8000` from the CloudLinux virtualenv
|
||||
(`~/virtualenv/sanadlite/3.8`), started by `shell_scripts/start_cpanel.sh`.
|
||||
- **Apache** reverse-proxies `https://sanadlite.yslootahtech.com` to it. A `[P]`
|
||||
rewrite cannot upgrade WebSockets — hence log polling and the browser-direct
|
||||
Live socket.
|
||||
- **cron every 2 minutes** re-runs the launcher. It is idempotent and does a
|
||||
*real HTTP health check*, not a TCP probe: a wedged uvicorn with a frozen
|
||||
asyncio loop still holds the port and answers a TCP connect, so a
|
||||
connect-only check once reported "up" while the site was down for ~2 days. If
|
||||
the port is bound but `/api/health` does not answer, the frozen process is
|
||||
killed and relaunched.
|
||||
- **Environment variables** set in cPanel's *Setup Python App* form are stored
|
||||
in `~/.cl.selector/python-selector.json` and are **not** re-exported by the
|
||||
venv's `activate`. The launcher parses and exports them itself — that is the
|
||||
store the Gemini API key must go into for a restart to keep it.
|
||||
- `passenger_wsgi.py` is the alternative Passenger/WSGI entry point. It works,
|
||||
but WSGI has no WebSockets at all; the uvicorn path above is what runs.
|
||||
|
||||
Deployment is `scp` of changed files plus a launcher restart. Static assets are
|
||||
served with a 7-day `max-age` and the page itself with `no-store`, so
|
||||
`index.html` references `live.js?v=<mtime>` — **bump that query string when you
|
||||
change a static file**, or browsers will keep the old one.
|
||||
|
||||
|
||||
## What was stripped vs Sanad (full)
|
||||
## Security
|
||||
|
||||
Removed because the lite dashboard never needed them:
|
||||
|
||||
- **Motion / arm:** `motion/`, `scripts/sanad_arm.txt`, `config/motion_config.json`, `dashboard/routes/{motion,macros,replay,skills}.py`.
|
||||
- **Live voice conversation:** `voice/sanad_voice.py`, `voice/audio_io.py`, `voice/live_voice_loop.py`, `voice/wake_phrase_manager.py`, `voice/model_script.py`, `voice/model_subprocess.py`, `gemini/subprocess.py`, `gemini/script.py`, `dashboard/routes/{live_voice,live_subprocess,wake_phrases}.py`.
|
||||
- **Offline brain:** `local/` (LLM, STT, TTS, VAD), `config/local_config.json`.
|
||||
- **Camera / vision:** `dashboard/routes/vision.py` and all `/api/vision/*` endpoints, the camera tab UI.
|
||||
- **Examples / demos:** `examples/`.
|
||||
- **Tabs:** Operations, Motion & Replay, Camera & Vision (deprecated), Live Voice Commands card, Wake Phrase Manager card, Live Gemini Process card.
|
||||
|
||||
Added by lite:
|
||||
|
||||
- **Login page + session cookie auth** (`dashboard/routes/auth.py`, `dashboard/static/login.html`, `SessionMiddleware`).
|
||||
- **Browser-side audio streaming** — `GET /api/records/audio/{name}?kind={speaker,raw}` and `GET /api/typed-replay/audio/last`.
|
||||
- **Download button** on each saved record.
|
||||
- **Delete All button** that wipes every WAV under `data/audio/`.
|
||||
- `config/core_config.json → auth` holds the dashboard username and password in
|
||||
**cleartext**, and that file is committed. Change both before any deployment
|
||||
you do not control, and consider replacing the check in
|
||||
`dashboard/routes/auth.py` with a bcrypt hash.
|
||||
- The session cookie is signed with a secret regenerated at every start, so a
|
||||
restart signs everyone out.
|
||||
- `live.allow_direct_key` exposes the Gemini API key to any signed-in user.
|
||||
- No API key is committed; `gemini_defaults.api_key` ships empty.
|
||||
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | Fix |
|
||||
| Symptom | Cause / fix |
|
||||
|---|---|
|
||||
| `ModuleNotFoundError: itsdangerous` at startup | `pip install itsdangerous` — required by Starlette's `SessionMiddleware`. |
|
||||
| `ModuleNotFoundError: websockets` when generating typed-replay audio | `pip install websockets` — `gemini/client.py` uses it. |
|
||||
| Redirected to `/login` on every API call | Session cookie cleared on server restart by design — sign in again. |
|
||||
| `Failed to construct audio_mgr — pyaudio not installed` warning at startup | Harmless on a laptop. `voice/audio_manager.py` requires PyAudio + portaudio headers; not needed for any user-facing button. Install with `sudo apt install portaudio19-dev && pip install pyaudio` if you want it gone. |
|
||||
| ALSA / PortAudio noise at startup (`pcm_dmix.c`, `Cannot connect to JACK`) | Pre-init probe of PortAudio inside `pyaudio.PyAudio()`. Cosmetic — the lite dashboard never actually opens an ALSA stream. To silence it, drop PyAudio entirely (uninstall + add a `_safe_import` guard for `voice.audio_manager`). |
|
||||
| `Gemini TTS attempt N returned no audio — parts: …` then 503 | Gemini Live is non-deterministic on short Arabic snippets — it sometimes returns reasoning text instead of audio. The retry chain in `voice/typed_replay.py:generate_audio` tries 3 prompt variants. Lengthen the text or add diacritics if it persists. |
|
||||
| `cannot import name 'X' from 'Project.Sanad.main'` | A route is trying to import a global that lite removed. Add a `try/except ImportError` in that route or drop the route from `dashboard/app.py:_REST_ROUTES`. |
|
||||
| Replay sounds like a different voice | The REST fallback ran — Live returned no audio at all. Check `logs/` for `served REST TTS`. Usually quota, or a very short input. |
|
||||
| "NOT the robot voice — … Hz outside …" warning | The pitch gate caught an off-voice take. Generate again; if it persists, the Live path is failing and everything is coming from REST. |
|
||||
| Words cut off at the end | Should be fixed (`post_generation_grace_sec: 3.0`). If it returns, raise that value before touching anything else. |
|
||||
| Tone/accent drifts between generations | A warm session leaked context — confirm `fresh_session_per_replay` is `true` and the voice is not in `warm_session_voices`. |
|
||||
| Everything 503s with a quota message | `GeminiQuotaExhausted` — the key is out of credits. It fails fast and says so rather than retrying six times. |
|
||||
| Live tab connects but never replies | Read the diagnostics line. `peak` near 0 = the mic is not being heard (wrong input device, or muted). `received` climbing but `played` flat = output routing. |
|
||||
| Live tab: token minted but connect fails | An `AQ.` key with `allow_direct_key: false`. Ephemeral tokens do not work for those keys. |
|
||||
| A UI fix "did not deploy" | Browser cache. Bump the `?v=` on the changed asset in `index.html`. |
|
||||
| Arabic in the CSV export looks like mojibake | Open it through Excel's import dialog as UTF-8; the file already carries a BOM. |
|
||||
| `ModuleNotFoundError: itsdangerous` / `websockets` | `pip install itsdangerous websockets` — required by SessionMiddleware and `gemini/client.py`. |
|
||||
| `Failed to construct audio_mgr — pyaudio not installed` | Harmless. Nothing user-facing needs PyAudio. |
|
||||
| Redirected to `/login` on every call | The server restarted and the cookie secret changed. Sign in again. |
|
||||
|
||||
|
||||
## Endpoints
|
||||
## Known limits
|
||||
|
||||
```
|
||||
GET / → / dashboard (auth-gated)
|
||||
GET /login → login page
|
||||
POST /api/auth/login → {username,password} → set cookie
|
||||
POST /api/auth/logout → clear cookie
|
||||
GET /api/auth/me → {authenticated, user}
|
||||
|
||||
GET /api/health → {status, brain}
|
||||
GET /api/status → {brain, voice}
|
||||
GET /api/system/info → host / interfaces / subsystems
|
||||
|
||||
GET /api/voice/status → Gemini connection state
|
||||
POST /api/voice/connect → connect Gemini Live socket
|
||||
POST /api/voice/disconnect → disconnect
|
||||
GET /api/voice/api-key → masked current key
|
||||
POST /api/voice/api-key → {key} → persist new key
|
||||
|
||||
POST /api/typed-replay/say → {text,record,record_name} → generates, caches
|
||||
GET /api/typed-replay/audio/last → streams cached WAV (browser plays it)
|
||||
POST /api/typed-replay/replay-last → bumps replay counter (audio still client-side)
|
||||
POST /api/typed-replay/save-last → persists cached generation to records
|
||||
GET /api/typed-replay/status → engine + session state
|
||||
GET /api/typed-replay/records → list
|
||||
DELETE /api/typed-replay/records/{name} → delete one
|
||||
POST /api/typed-replay/records/{name}/rename
|
||||
|
||||
GET /api/records/ → list saved records
|
||||
GET /api/records/audio/{name}?kind=... → stream a record's WAV
|
||||
POST /api/records/delete → {record_name} → delete one
|
||||
POST /api/records/delete-all → wipe data/audio/*.wav + reset index
|
||||
|
||||
GET /api/scripts/ → list persona/rule files
|
||||
POST /api/scripts/load → {name} → file contents
|
||||
POST /api/scripts/save → {name,content}
|
||||
POST /api/scripts/create → {name,content}
|
||||
POST /api/scripts/delete → {name}
|
||||
|
||||
GET /api/prompt/ → resolved system prompt
|
||||
POST /api/prompt/update → {content}
|
||||
POST /api/prompt/reload → re-read from disk
|
||||
|
||||
GET /api/logs/{module}/tail → last N log lines
|
||||
POST /api/logs/snapshot → save snapshot bundle
|
||||
GET /api/logs/bundle → download all logs as a zip
|
||||
GET /api/audio/status → mic/spk mute state (server-side, informational)
|
||||
WS /ws/logs → live log stream
|
||||
```
|
||||
- **Accent quality is not measured.** Pitch and coverage are verified
|
||||
automatically; how Emirati a take *sounds* is not. That needs a blind
|
||||
listening test.
|
||||
- **Typed replay is single-user.** The session, the selected voice and
|
||||
`audio/last` are process globals — two people generating at once will
|
||||
interfere.
|
||||
- **Very short inputs** (a single word) can still fall through to the REST
|
||||
engine, because the Live model sometimes returns text instead of audio for
|
||||
them.
|
||||
- The `generationComplete` truncation fix has **not** been ported to the robots.
|
||||
|
||||
|
||||
## License / attribution
|
||||
## Attribution
|
||||
|
||||
Internal project for YS Lootah Technology. Trimmed from Sanad — original
|
||||
Sanad reuses patterns from `SanadVoice/gemini_interact` and Unitree
|
||||
`unitree_sdk2py`.
|
||||
Internal project for **YS Lootah Robotics**. Trimmed from Sanad; the original
|
||||
reuses patterns from `SanadVoice/gemini_interact` and Unitree `unitree_sdk2py`.
|
||||
|
||||
@ -228,6 +228,16 @@
|
||||
<label>Robot</label>
|
||||
<div id="live-voices" class="row" style="gap:.3rem;flex-wrap:wrap;margin:.2rem 0 .6rem"></div>
|
||||
|
||||
<div class="row" style="gap:.35rem;align-items:center;flex-wrap:wrap;margin-bottom:.35rem">
|
||||
<label style="font-size:.7rem;color:var(--dim);min-width:52px">Mic</label>
|
||||
<select id="live-mic" onchange="liveSetMic(this.value)" style="flex:1 1 220px;max-width:340px"></select>
|
||||
<label style="font-size:.7rem;color:var(--dim);min-width:52px">Speaker</label>
|
||||
<select id="live-out" onchange="liveSetOutput(this.value)" style="flex:1 1 220px;max-width:340px"></select>
|
||||
<button class="btn btn-ghost btn-sm" onclick="liveListDevices()">Refresh devices</button>
|
||||
</div>
|
||||
<!-- Playback is routed through this element so setSinkId() can send it to
|
||||
the chosen speaker; an AudioContext alone always uses the default. -->
|
||||
<audio id="live-audio-out" autoplay style="display:none"></audio>
|
||||
<div class="row" style="gap:.4rem;align-items:center;flex-wrap:wrap;margin-bottom:.4rem">
|
||||
<button class="btn btn-ghost btn-sm" onclick="liveRequestMic()">Allow microphone</button>
|
||||
<button class="btn btn-ghost btn-sm" onclick="liveTestSpeaker()">Test speaker</button>
|
||||
@ -238,6 +248,7 @@
|
||||
<span style="font-size:.68rem;color:var(--dim)">model: <span id="live-model">--</span></span>
|
||||
</div>
|
||||
|
||||
<div id="live-compat" style="font-size:.66rem;color:var(--dim);margin:.1rem 0"></div>
|
||||
<div id="live-diag" style="font-size:.68rem;color:var(--dim);margin:.1rem 0 .5rem;font-family:monospace"></div>
|
||||
|
||||
<label>Conversation</label>
|
||||
@ -729,6 +740,6 @@ async function setVoice(voice,b){
|
||||
refreshStatus();refreshAudio();refreshTR();refreshApiKey();refreshVoices();refreshRule();refreshRecords();connectLogs();refreshLogins();if(window.liveInit)liveInit();
|
||||
setInterval(refreshStatus,5000);
|
||||
</script>
|
||||
<script src="/static/live.js?v=1787937410"></script>
|
||||
<script src="/static/live.js?v=1788376613"></script>
|
||||
</body>
|
||||
</html>
|
||||
|
||||
@ -22,6 +22,11 @@
|
||||
let playHead = 0; // when the next chunk should start, in playCtx time
|
||||
let connected = false;
|
||||
let sentChunks = 0, recvFrames = 0, playedChunks = 0, micPeak = 0, diagTimer = null;
|
||||
let playDest = null; // MediaStreamAudioDestinationNode
|
||||
let workletNode = null; // AudioWorklet capture node
|
||||
let captureKind = ''; // 'worklet' | 'scriptprocessor'
|
||||
let micId = localStorage.getItem('live.micId') || '';
|
||||
let outId = localStorage.getItem('live.outId') || '';
|
||||
let cfg = null; // {voices:[{voice,label,persona}], model}
|
||||
let selectedVoice = 'Charon';
|
||||
|
||||
@ -40,9 +45,12 @@
|
||||
if (!el) return;
|
||||
const mic = micCtx ? micCtx.state : '-';
|
||||
const play = playCtx ? playCtx.state : '-';
|
||||
const outEl = $('live-audio-out');
|
||||
const route = outId ? ('sink ' + (outEl && outEl.sinkId ? outEl.sinkId.slice(0, 8) : '?')) : 'default';
|
||||
el.textContent = `sent ${sentChunks} chunks (peak ${micPeak.toFixed(3)}) · `
|
||||
+ `received ${recvFrames} frames · played ${playedChunks} · `
|
||||
+ `mic ctx ${mic} · out ctx ${play}`;
|
||||
+ `mic ctx ${mic} · out ctx ${play} · out ${route}`
|
||||
+ (captureKind ? ` · capture ${captureKind}` : '');
|
||||
}
|
||||
|
||||
function addLine(who, text) {
|
||||
@ -240,7 +248,7 @@
|
||||
for (let i = 0; i < pcm.length; i++) ch[i] = pcm[i] / 32768;
|
||||
const src = playCtx.createBufferSource();
|
||||
src.buffer = buf;
|
||||
src.connect(playCtx.destination);
|
||||
src.connect(outputNode());
|
||||
// Queue chunks back-to-back so speech does not overlap or gap.
|
||||
const now = playCtx.currentTime;
|
||||
if (playHead < now) playHead = now;
|
||||
@ -253,9 +261,7 @@
|
||||
if (connected) return;
|
||||
status('Requesting microphone…');
|
||||
try {
|
||||
micStream = await navigator.mediaDevices.getUserMedia({
|
||||
audio: { channelCount: 1, echoCancellation: true, noiseSuppression: true },
|
||||
});
|
||||
micStream = await navigator.mediaDevices.getUserMedia({ audio: micConstraint() });
|
||||
} catch (e) {
|
||||
status('Microphone denied — the browser must allow it', 'err');
|
||||
return;
|
||||
@ -343,6 +349,62 @@
|
||||
};
|
||||
}
|
||||
|
||||
// Worklet source, inlined as a Blob so no extra file needs serving. It only
|
||||
// forwards raw frames to the main thread; resampling stays in one place.
|
||||
const WORKLET_SRC = `
|
||||
class Cap extends AudioWorkletProcessor {
|
||||
process(inputs) {
|
||||
const ch = inputs[0] && inputs[0][0];
|
||||
if (ch && ch.length) this.port.postMessage(ch.slice(0));
|
||||
return true;
|
||||
}
|
||||
}
|
||||
registerProcessor('cap', Cap);
|
||||
`;
|
||||
|
||||
function sendFrame(raw) {
|
||||
if (!connected || !ws || ws.readyState !== WebSocket.OPEN) return;
|
||||
let peak = 0;
|
||||
for (let i = 0; i < raw.length; i++) { const a = Math.abs(raw[i]); if (a > peak) peak = a; }
|
||||
if (peak > micPeak) micPeak = peak;
|
||||
const down = downsample(raw, micCtx.sampleRate, SEND_RATE);
|
||||
const pcm = floatToPcm16(down);
|
||||
sentChunks++;
|
||||
ws.send(JSON.stringify({
|
||||
realtimeInput: {
|
||||
mediaChunks: [{
|
||||
mimeType: 'audio/pcm;rate=' + SEND_RATE,
|
||||
data: b64FromBytes(new Uint8Array(pcm.buffer)),
|
||||
}],
|
||||
},
|
||||
}));
|
||||
}
|
||||
|
||||
async function startMicWorklet(source) {
|
||||
const url = URL.createObjectURL(new Blob([WORKLET_SRC], { type: 'application/javascript' }));
|
||||
try {
|
||||
await micCtx.audioWorklet.addModule(url);
|
||||
} finally {
|
||||
URL.revokeObjectURL(url);
|
||||
}
|
||||
workletNode = new AudioWorkletNode(micCtx, 'cap');
|
||||
workletNode.port.onmessage = (e) => sendFrame(e.data);
|
||||
source.connect(workletNode);
|
||||
// Firefox will not run a worklet that has no downstream connection.
|
||||
const mute = micCtx.createGain();
|
||||
mute.gain.value = 0;
|
||||
workletNode.connect(mute).connect(micCtx.destination);
|
||||
return 'worklet';
|
||||
}
|
||||
|
||||
function startMicScriptProcessor(source) {
|
||||
processor = micCtx.createScriptProcessor(4096, 1, 1);
|
||||
processor.onaudioprocess = (ev) => sendFrame(ev.inputBuffer.getChannelData(0));
|
||||
source.connect(processor);
|
||||
processor.connect(micCtx.destination);
|
||||
return 'scriptprocessor';
|
||||
}
|
||||
|
||||
function startMic() {
|
||||
micCtx = new (window.AudioContext || window.webkitAudioContext)();
|
||||
// A context created outside a user gesture starts suspended and its
|
||||
@ -354,27 +416,13 @@
|
||||
if (diagTimer) clearInterval(diagTimer);
|
||||
diagTimer = setInterval(diag, 1000);
|
||||
const source = micCtx.createMediaStreamSource(micStream);
|
||||
processor = micCtx.createScriptProcessor(4096, 1, 1);
|
||||
source.connect(processor);
|
||||
processor.connect(micCtx.destination);
|
||||
processor.onaudioprocess = (ev) => {
|
||||
if (!connected || !ws || ws.readyState !== WebSocket.OPEN) return;
|
||||
const raw = ev.inputBuffer.getChannelData(0);
|
||||
let peak = 0;
|
||||
for (let i = 0; i < raw.length; i++) { const a = Math.abs(raw[i]); if (a > peak) peak = a; }
|
||||
if (peak > micPeak) micPeak = peak;
|
||||
const down = downsample(raw, micCtx.sampleRate, SEND_RATE);
|
||||
const pcm = floatToPcm16(down);
|
||||
sentChunks++;
|
||||
ws.send(JSON.stringify({
|
||||
realtimeInput: {
|
||||
mediaChunks: [{
|
||||
mimeType: 'audio/pcm;rate=' + SEND_RATE,
|
||||
data: b64FromBytes(new Uint8Array(pcm.buffer)),
|
||||
}],
|
||||
},
|
||||
}));
|
||||
};
|
||||
if (micCtx.audioWorklet && typeof AudioWorkletNode === 'function') {
|
||||
startMicWorklet(source)
|
||||
.then((k) => { captureKind = k; })
|
||||
.catch(() => { captureKind = startMicScriptProcessor(source); });
|
||||
} else {
|
||||
captureKind = startMicScriptProcessor(source);
|
||||
}
|
||||
}
|
||||
|
||||
function stop() {
|
||||
@ -382,15 +430,135 @@
|
||||
if (diagTimer) { clearInterval(diagTimer); diagTimer = null; }
|
||||
diag();
|
||||
try { if (processor) processor.disconnect(); } catch (_) {}
|
||||
try { if (workletNode) { workletNode.port.onmessage = null; workletNode.disconnect(); } } catch (_) {}
|
||||
try { if (micCtx) micCtx.close(); } catch (_) {}
|
||||
try { if (micStream) micStream.getTracks().forEach(t => t.stop()); } catch (_) {}
|
||||
try { if (ws && ws.readyState <= 1) ws.close(); } catch (_) {}
|
||||
processor = micCtx = micStream = ws = null;
|
||||
processor = workletNode = micCtx = micStream = ws = null;
|
||||
captureKind = '';
|
||||
playHead = 0;
|
||||
const b = $('live-connect');
|
||||
if (b) { b.textContent = 'Connect'; b.classList.add('btn-primary'); b.classList.remove('btn-danger'); }
|
||||
}
|
||||
|
||||
// ── device selection ────────────────────────────────────────────
|
||||
function outputNode() {
|
||||
// Route through a MediaStream + <audio> when a specific speaker is chosen,
|
||||
// because only a media element can be pointed at a sink. With no choice,
|
||||
// go straight to the context's own destination.
|
||||
const el = $('live-audio-out');
|
||||
if (!outId || !el || typeof el.setSinkId !== 'function') return playCtx.destination;
|
||||
if (!playDest) {
|
||||
playDest = playCtx.createMediaStreamDestination();
|
||||
el.srcObject = playDest.stream;
|
||||
el.play().catch(() => {});
|
||||
}
|
||||
return playDest;
|
||||
}
|
||||
|
||||
window.liveCompat = () => {
|
||||
const el = $('live-compat');
|
||||
if (!el) return;
|
||||
const audioEl = document.createElement('audio');
|
||||
const bits = [
|
||||
['getUserMedia', !!(navigator.mediaDevices && navigator.mediaDevices.getUserMedia)],
|
||||
['WebSocket', typeof WebSocket === 'function'],
|
||||
// `in` only tests for the property; reading AudioContext.prototype
|
||||
// .audioWorklet invokes the getter on the prototype and throws
|
||||
// ("Illegal invocation" / "does not implement interface").
|
||||
['AudioWorklet', !!(window.AudioContext && ('audioWorklet' in AudioContext.prototype))],
|
||||
['speaker choice', typeof audioEl.setSinkId === 'function'],
|
||||
['device list', !!(navigator.mediaDevices && navigator.mediaDevices.enumerateDevices)],
|
||||
];
|
||||
el.innerHTML = bits.map(([name, ok]) =>
|
||||
`<span style="color:${ok ? '#4ade80' : '#f87171'}">${name}${ok ? ' ok' : ' missing'}</span>`
|
||||
).join(' · ');
|
||||
// Everything except the speaker picker is required; that one degrades to
|
||||
// the system default output rather than failing.
|
||||
return bits.filter(b => !b[1]).map(b => b[0]);
|
||||
};
|
||||
|
||||
window.liveListDevices = async () => {
|
||||
const micSel = $('live-mic'), outSel = $('live-out');
|
||||
if (!micSel || !outSel) return;
|
||||
let devs = [];
|
||||
try { devs = await navigator.mediaDevices.enumerateDevices(); } catch (e) { return; }
|
||||
// Labels are empty until microphone permission has been granted once.
|
||||
const needPerm = devs.some(d => d.kind === 'audioinput' && !d.label);
|
||||
// Windows reports each device three times: the real one plus the
|
||||
// 'default' and 'communications' pseudo-entries, whose labels are prefixed
|
||||
// ("Default - Anker PowerConf"). Drop 'communications', keep the real
|
||||
// devices, and strip the prefixes so one speaker appears once.
|
||||
const clean = (list) => list
|
||||
.filter(d => d.deviceId !== 'communications' && d.deviceId !== 'default')
|
||||
.map(d => ({ deviceId: d.deviceId,
|
||||
label: (d.label || '').replace(/^(Default|Communications)\s*-\s*/i, '') }));
|
||||
const mics = clean(devs.filter(d => d.kind === 'audioinput'));
|
||||
const outs = clean(devs.filter(d => d.kind === 'audiooutput'));
|
||||
micSel.innerHTML = '<option value="">System default</option>' + mics.map((d, i) =>
|
||||
`<option value="${esc(d.deviceId)}">${esc(d.label || ('Microphone ' + (i + 1)))}</option>`).join('');
|
||||
const canPick = $('live-audio-out') && typeof $('live-audio-out').setSinkId === 'function';
|
||||
outSel.innerHTML = '<option value="">System default</option>' + (canPick ? outs.map((d, i) =>
|
||||
`<option value="${esc(d.deviceId)}">${esc(d.label || ('Speaker ' + (i + 1)))}</option>`).join('') : '');
|
||||
if (!canPick) {
|
||||
outSel.title = 'This browser cannot choose an output device; it follows the system default.';
|
||||
}
|
||||
if (micId) micSel.value = micId;
|
||||
if (outId) outSel.value = outId;
|
||||
if (needPerm) permNote('allow the microphone once to see device names', '');
|
||||
};
|
||||
|
||||
window.liveSetMic = async (id) => {
|
||||
micId = id || '';
|
||||
localStorage.setItem('live.micId', micId);
|
||||
if (connected) {
|
||||
// Reopen capture on the new device without dropping the session.
|
||||
try {
|
||||
if (processor) processor.disconnect();
|
||||
if (workletNode) { workletNode.port.onmessage = null; workletNode.disconnect(); workletNode = null; }
|
||||
if (micCtx) await micCtx.close();
|
||||
if (micStream) micStream.getTracks().forEach(t => t.stop());
|
||||
micStream = await navigator.mediaDevices.getUserMedia({ audio: micConstraint() });
|
||||
startMic();
|
||||
permNote('microphone switched', 'ok');
|
||||
} catch (e) {
|
||||
permNote('could not switch microphone: ' + ((e && e.message) || e), 'err');
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
window.liveSetOutput = async (id) => {
|
||||
outId = id || '';
|
||||
localStorage.setItem('live.outId', outId);
|
||||
const el = $('live-audio-out');
|
||||
if (!el) return;
|
||||
try {
|
||||
if (!playCtx) playCtx = new (window.AudioContext || window.webkitAudioContext)();
|
||||
if (playCtx.state === 'suspended') await playCtx.resume();
|
||||
if (outId && typeof el.setSinkId === 'function') {
|
||||
// Attach the stream first, then point the element at the sink.
|
||||
outputNode();
|
||||
await el.setSinkId(outId);
|
||||
await el.play().catch(() => {});
|
||||
permNote('speaker set - press Test speaker to confirm', 'ok');
|
||||
} else if (outId) {
|
||||
permNote('this browser cannot choose an output device; set it in the system mixer', 'err');
|
||||
} else {
|
||||
// Back to default: drop the element route so audio goes direct.
|
||||
if (playDest) { try { playDest.disconnect(); } catch (e) {} playDest = null; }
|
||||
el.srcObject = null;
|
||||
permNote('speaker: system default', '');
|
||||
}
|
||||
} catch (e) {
|
||||
permNote('could not set speaker: ' + ((e && e.message) || e), 'err');
|
||||
}
|
||||
};
|
||||
|
||||
function micConstraint() {
|
||||
const base = { channelCount: 1, echoCancellation: true, noiseSuppression: true };
|
||||
return micId ? Object.assign({ deviceId: { exact: micId } }, base) : base;
|
||||
}
|
||||
|
||||
// ── microphone / speaker ────────────────────────────────────────
|
||||
function permNote(text, tone) {
|
||||
const el = $('live-perm');
|
||||
@ -430,6 +598,7 @@
|
||||
name = mic ? ' (' + mic.label + ')' : '';
|
||||
} catch (e) { /* labels need permission; ignore */ }
|
||||
permNote('microphone: allowed' + name, 'ok');
|
||||
liveListDevices();
|
||||
} catch (e) {
|
||||
// Once refused, the browser will not prompt again for this site.
|
||||
const denied = e && (e.name === 'NotAllowedError' || e.name === 'SecurityError');
|
||||
@ -447,7 +616,7 @@
|
||||
const gain = playCtx.createGain();
|
||||
osc.frequency.value = 440;
|
||||
gain.gain.value = 0.15; // audible but not startling
|
||||
osc.connect(gain).connect(playCtx.destination);
|
||||
osc.connect(gain).connect(outputNode());
|
||||
osc.start();
|
||||
osc.stop(playCtx.currentTime + 0.35);
|
||||
permNote('speaker: played a test tone (output ' + playCtx.state + ')', 'ok');
|
||||
@ -461,6 +630,12 @@
|
||||
// Independent: a failure in one must not leave the other blank.
|
||||
loadConfig().catch((e) => status('Could not load config: ' + (e && e.message || e), 'err'));
|
||||
if (window.liveRefreshPerm) liveRefreshPerm();
|
||||
try { liveCompat(); } catch (e) { console.error('compat check failed', e); }
|
||||
liveListDevices().catch(() => {});
|
||||
if (navigator.mediaDevices) {
|
||||
// Plugging in a headset mid-session must update the lists.
|
||||
navigator.mediaDevices.addEventListener('devicechange', () => liveListDevices());
|
||||
}
|
||||
loadPersonas().catch((e) => {
|
||||
window.__personaError = e;
|
||||
const note = document.getElementById('persona-note');
|
||||
|
||||
29
data/live_personas.json
Normal file
29
data/live_personas.json
Normal file
File diff suppressed because one or more lines are too long
65
shell_scripts/start_cpanel.sh
Normal file
65
shell_scripts/start_cpanel.sh
Normal file
@ -0,0 +1,65 @@
|
||||
#!/bin/bash
|
||||
# Sanad_lite uvicorn launcher. Idempotent — exits cleanly if uvicorn is
|
||||
# already SERVING on :8000. Designed to be called every 2 min by cron.
|
||||
#
|
||||
# This is the copy that actually runs in production, kept here so the repo
|
||||
# is not missing the piece that keeps the site up. It lives on the host at
|
||||
# /home2/sanadliteysloota/sanadlite/start.sh
|
||||
# and is invoked by:
|
||||
# */2 * * * * /home2/sanadliteysloota/sanadlite/start.sh
|
||||
# The absolute paths below are that account's; change them for another host.
|
||||
|
||||
# --- 1. Health check: is the app actually SERVING HTTP on :8000? ----------
|
||||
# A plain TCP-connect probe is not enough on its own: a wedged uvicorn (frozen
|
||||
# asyncio loop) still holds the port and completes the TCP handshake, so a
|
||||
# connect-only check reports "up" and never restarts a hung app (this bit us —
|
||||
# the site was down ~2 days on a bound-but-frozen process). So: first cheaply
|
||||
# check whether anything is bound (bash /dev/tcp), and if so require a real
|
||||
# HTTP response too. curl prints "000" on timeout (hung); any real status code
|
||||
# (200/303/401/...) means the event loop is alive and serving.
|
||||
if (exec 3<>/dev/tcp/127.0.0.1/8000) 2>/dev/null; then
|
||||
exec 3<&-; exec 3>&-
|
||||
CODE="$(curl -s -o /dev/null -m 8 -w '%{http_code}' http://127.0.0.1:8000/api/health 2>/dev/null)"
|
||||
if [ -n "$CODE" ] && [ "$CODE" != "000" ]; then
|
||||
exit 0 # bound AND serving → healthy, nothing to do
|
||||
fi
|
||||
# Bound but not responding → wedged. Kill the frozen instance so we can
|
||||
# relaunch cleanly. Pattern is anchored to the venv path so it only ever
|
||||
# matches our own uvicorn, never this script or the selector helper.
|
||||
echo "[start.sh] $(date) — :8000 bound but not serving (code=${CODE:-none}); killing frozen instance" >> /home2/sanadliteysloota/sanadlite/logs/uvicorn.log
|
||||
pkill -9 -f '/home2/sanadliteysloota/virtualenv/sanadlite/.*main.py' 2>/dev/null
|
||||
sleep 2
|
||||
fi
|
||||
|
||||
cd /home2/sanadliteysloota/sanadlite
|
||||
|
||||
# --- 2. Activate the venv (puts python3 on PATH, sets VIRTUAL_ENV) --------
|
||||
source /home2/sanadliteysloota/virtualenv/sanadlite/3.8/bin/activate
|
||||
|
||||
# --- 3. Source env vars set in cPanel "Setup Python App" form -------------
|
||||
# CloudLinux's Python Selector stores them in ~/.cl.selector/python-selector.json
|
||||
# but the activate script does NOT re-export them. Parse + export here.
|
||||
SELECTOR_JSON="$HOME/.cl.selector/python-selector.json"
|
||||
APP_NAME="sanadlite"
|
||||
if [ -f "$SELECTOR_JSON" ]; then
|
||||
eval "$(python - <<PYEOF
|
||||
import json, shlex
|
||||
try:
|
||||
data = json.load(open("$SELECTOR_JSON"))
|
||||
except Exception:
|
||||
pass
|
||||
else:
|
||||
app = data.get("$APP_NAME") or {}
|
||||
for k, v in (app.get("env_vars") or {}).items():
|
||||
print(f"export {shlex.quote(str(k))}={shlex.quote(str(v))}")
|
||||
PYEOF
|
||||
)"
|
||||
fi
|
||||
|
||||
# --- 4. Standard runtime env ---------------------------------------------
|
||||
export SANAD_DASHBOARD_HOST=127.0.0.1
|
||||
|
||||
# --- 5. Launch uvicorn in the background ---------------------------------
|
||||
echo "[start.sh] $(date) — launching uvicorn on :8000" >> /home2/sanadliteysloota/sanadlite/logs/uvicorn.log
|
||||
nohup python main.py >> /home2/sanadliteysloota/sanadlite/logs/uvicorn.log 2>&1 &
|
||||
disown
|
||||
Loading…
x
Reference in New Issue
Block a user