# AGIBOT A3 · Voice Control A local web dashboard for making an **AGIBOT A3** humanoid speak typed text through its own built-in speaker. ``` PC Browser → Local Web App → Robot Service Layer → A3 Adapter → AGIBOT A3 → Speaker ``` It runs today, with **no robot attached**, against a built-in simulator. When the robot arrives you change the IP in one file and switch one setting. **The A3's speech interface is already implemented.** AgiBot publicly documents a native text-to-speech HTTP endpoint on the robot, and this project speaks it: ```http POST http://:59301/rpc/aimdk.protocol.TTSService/PlayTTS Content-Type: application/json {"text": "...", "priority_level": "INTERACTION_L6", "domain": "voice_control", "trace_id": "...", "is_interrupted": true} ``` Text goes in, the robot's own TTS comes out of its own speaker. **No audio is generated or transferred by the PC**, which is what makes it fast. Full detail, sources and open questions: **[docs/AGIBOT_A3_INTEGRATION.md](docs/AGIBOT_A3_INTEGRATION.md)**.
## Quick start ```bat start.bat ``` …then open **http://localhost:8000**. Or manually: ```bash pip install -r requirements.txt python backend/main.py ``` The app starts in **Mock Mode**: the dashboard, the API, the status pipeline, the history and every error path work exactly as they will with the real robot — the robot itself is simulated. **You will hear it.** The simulator speaks each utterance through this PC's speakers, so you can rehearse a demo — wording, pacing, the Stop button — before the A3 is on the network. The real audio drives the UI: *Completed* appears when the sound actually stops, and **Stop** cuts the voice mid-word. Two voice engines, set by `MOCK_VOICE_ENGINE` in `.env`: | | `system` | `gemini` | | --- | --- | --- | | Sound | robotic, built-in OS voice | **natural neural voice** | | New text | instant | ~4 s (sentence) – 8 s (paragraph) | | Repeated text | instant | **instant** — cached on disk | | Needs | nothing | `GEMINI_API_KEY` + internet | ```bash python scripts/voices.py # list both engines' voices python scripts/voices.py --demo # hear the system ones ``` ### Saved audio Every line spoken with the neural voice is saved to **`audio_library/`** as an ordinary `.wav` named after its text: ``` audio_library/ welcome-to-our-showroom-78e24feb.wav please-follow-me-to-the-first-exhibit-653c14ff.wav index.json ``` That means a line spoken once **replays instantly** — no synthesis, no network, no wait — and the files are yours: play them in any media player, drop them into a video edit, or put them on a stand's playlist. The dashboard's **Saved audio** panel lists them with play and download buttons, and history rows that already have audio get a ▶ for instant replay. **Before a live demo, warm your lines** so nothing waits on synthesis: ```bash python scripts/warm_voice.py --file demo_lines.txt python scripts/warm_voice.py --stats ``` If Gemini is unreachable — no network, bad key, quota — the simulator silently falls back to the built-in voice, so a network problem never leaves you with silence. Pick one with `MOCK_VOICE=` in `.env`; silence it entirely with `MOCK_LOCAL_AUDIO=false`. This is the *simulator's* voice — the real A3 synthesises its own speech on-board. The robot's configured voice is **"Yunxiao"** (teenager, male, multi-language). Windows ships no teenage voice, so the default here is its lighter adult male voice (**Mark**) pitched up (`MOCK_SPEECH_PITCH=3`) to read younger. It is an approximation for rehearsal, not a match. Verify the whole stack end to end at any time: ```bash python scripts/selftest.py # 31 checks: REST, WebSocket, speak, stop, errors, history ``` ### Testing the real robot path before the robot exists Mock mode tests the *application*. To exercise the actual A3 wire protocol — the URL, the JSON body, the `trace_id` round trip, byte-limit chunking, Stop — run the local stand-in: ```bash python scripts/fake_a3_server.py # implements the documented A3 contract on :59301 ``` then set `ROBOT_MODE=real`, `ROBOT_IP=127.0.0.1` in `.env`. Every utterance prints in the stand-in's console. This project's `aimdk` transport passes all 31 checks against it.
## Using the dashboard | Action | How | | --- | --- | | Speak | Type, then click **Speak** or press **Ctrl + Enter** | | Stop | **Stop** button or **Esc** | | Clear the box | **Clear** button | | Repeat something | Click any entry in **History** — it goes back in the box | | Reconnect | The ↻ button in the header | `Enter` inserts a newline and does **not** send — a half-typed sentence should never reach the robot's speaker mid-demo. The header always states the truth about the connection: green *Robot Connected*, amber *Connecting…*, red *Robot Disconnected*, plus the measured round-trip latency to the robot.
## Project layout ``` A3_text_to_speach/ │ ├── frontend/ no build step - plain ES modules, served by the backend │ ├── index.html │ ├── styles/main.css │ └── js/ │ ├── app.js wiring + speech lifecycle │ ├── api.js REST client │ ├── socket.js WebSocket client (auto-reconnect, RTT ping) │ └── ui.js all DOM rendering │ ├── backend/ │ ├── main.py entry point: app factory, static serving, error mapping │ ├── api/ │ │ ├── routes.py REST endpoints │ │ ├── websocket.py live status/lifecycle channel │ │ └── schemas.py │ ├── services/ │ │ ├── speech_service.py validation, single-flight, interrupt, latency │ │ └── history.py │ ├── robot/ │ │ ├── base.py ← the only contract the app depends on │ │ ├── factory.py mode → adapter │ │ ├── manager.py connect / health-poll / backoff reconnect │ │ ├── mock_robot.py the simulator │ │ ├── agibot_a3.py ← THE INTEGRATION POINT │ │ └── transports/ │ │ ├── aimdk_transport.py ← AgiBot's documented A3 speech RPC (default) │ │ └── http · ws · ros2 · ssh fallbacks, fully config-driven │ ├── core/ event bus, text utils, logging │ └── config/settings.py every tunable, loaded from .env │ ├── docs/ │ ├── AGIBOT_A3_INTEGRATION.md how A3 speech works + what to fill in │ └── NETWORK.md PC ↔ robot networking and troubleshooting │ ├── scripts/ │ ├── voices.py list / audition the simulator's voices │ ├── warm_voice.py pre-synthesise demo lines (instant playback) │ ├── selftest.py end-to-end test (31 checks) │ ├── discover_robot.py probe a robot IP for its speech interface │ └── fake_a3_server.py local stand-in for the A3's RPC, to test the real path │ ├── audio_library/ saved .wav files + index (instant replay) ├── pronunciation.json how the simulator says tricky names ├── .env ← YOUR CONFIGURATION (robot IP goes here) ├── .env.example documented template ├── requirements.txt └── start.bat / start.sh ``` ### Why it is layered this way Each arrow is a seam you can replace without touching the others: ``` Browser │ REST for commands, WebSocket for state Backend API (api/) │ domain objects only - no HTTP, no sockets Speech Service (services/) │ RobotAdapter interface Robot Adapter (robot/mock_robot.py | robot/agibot_a3.py) │ SpeechTransport interface Transport (robot/transports/*) │ AGIBOT A3 ``` Nothing above `robot/` knows a robot SDK exists. Adding an AgiBot X2, a second robot, or a different TTS engine means writing one adapter and adding one line to `factory.py` — the UI, the API and the service layer are untouched.
## Configuration Everything lives in **`.env`** (created from `.env.example` on first run). The source code contains no IP address, port or endpoint. ```env ROBOT_MODE=mock # mock | real ← the switch ROBOT_IP= # ← the robot's IP goes here ROBOT_PORT=59301 # AgiBot's documented A3 TTS RPC port A3_TRANSPORT=aimdk # aimdk | http | ws | ros2 | ssh ``` After editing `.env`, either restart the server or call: ```bash curl -X POST http://localhost:8000/api/config/reload ``` which rebuilds the robot adapter in place — no restart, no code change. ### Mock-mode knobs worth knowing | Setting | Effect | | --- | --- | | `MOCK_FAILURE_RATE=0.3` | 3 in 10 utterances fail — exercises the error UI | | `MOCK_FLAKY_CONNECTION=true` | the link drops periodically — exercises reconnect | | `MOCK_LOCAL_AUDIO=true` | **on by default** — actually speaks through this PC's speakers | | `MOCK_VOICE_ENGINE=gemini` | natural neural voice instead of the robotic built-in one | | `GEMINI_VOICE=Puck` | which neural voice (Puck/Fenrir read youngest + male) | | `MOCK_VOICE=Mark` | which *system* voice to use, when engine is `system` | | `MOCK_SPEECH_PITCH=3` | raise the pitch — approximates the robot's young male voice | | `MOCK_SPEECH_RATE=-2` | slow the simulator down for a noisy room (−10…10) | | `SPEECH_ALLOW_INTERRUPT=false` | a second Speak while talking is rejected instead of interrupting |
## API | Method | Path | Purpose | | --- | --- | --- | | `GET` | `/api/health` | web app liveness (always 200, even with the robot offline) | | `GET` | `/api/robot/status` | connection state, latency, uptime, config problems | | `POST` | `/api/robot/speak` | `{"text": "..."}` → speak it | | `POST` | `/api/robot/stop` | interrupt the current utterance | | `POST` | `/api/robot/reconnect` | retry the connection immediately | | `GET` | `/api/robot/diagnostics` | adapter + transport detail, for troubleshooting | | `GET` | `/api/speech/history` | recent utterances | | `DELETE` | `/api/speech/history` | clear it | | `GET` | `/api/audio` | saved clips + library stats | | `GET` | `/api/audio/{id}/file` | the `.wav` itself (play or download) | | `DELETE` | `/api/audio/{id}` | delete one clip | | `DELETE` | `/api/audio` | delete all saved audio | | `GET` | `/api/config` | non-secret configuration the UI needs | | `POST` | `/api/config/reload` | re-read `.env` and rebuild the adapter | | `WS` | `/ws` | live status, speech lifecycle, history updates | Interactive docs: **http://localhost:8000/api/docs** ```bash curl -X POST http://localhost:8000/api/robot/speak \ -H "Content-Type: application/json" \ -d "{\"text\": \"Hello, welcome to our showroom.\"}" ``` ```json { "success": true, "status": "processing", "requestId": "3c7cfe41cf7d", "ackLatencyMs": 56 } ``` **`/api/robot/speak` returns when the robot has *accepted* the utterance, not when it stops talking.** A 12-second sentence must not look like a 12-second-slow button. The rest of the lifecycle — `speaking`, `completed`, failures — arrives on the WebSocket.
## How the speed is achieved | Decision | Why | | --- | --- | | Text is sent to the robot; audio is never generated on the PC | no synthesis, no file, no upload — one small HTTP request | | One persistent connection, opened at startup and reused | no TCP/TLS handshake per utterance | | HTTP response returns at acknowledgement | button feels instant; speaking progress streams separately | | WebSocket push, zero polling | the UI changes the moment the robot does | | Health probe is separate from the speech path | a slow health check can never delay a Speak | | Short connect timeout (3 s), long request timeout (8 s) | an offline robot fails fast; a busy one is not cut off |
## Error handling | Situation | What you see | | --- | --- | | Robot offline | `Robot is offline. Check the robot IP address and network connection.` | | `ROBOT_IP` empty while `ROBOT_MODE=real` | a configuration banner naming the exact `.env` key | | Invalid IP / hostname | configuration error before any connection is attempted | | Connection timeout | fails after 3 s, UI stays responsive, retry continues in the background | | TTS failure | `Speech request failed.` + the robot's own reason in the server log | | Empty text | Speak is disabled; the API returns `400` | | Text too long | rejected with the actual limit named | | Disconnect mid-utterance | status flips immediately and the utterance is marked failed | | Backend stopped | the page says so rather than showing a stale green light | The robot being unreachable is treated as normal, not exceptional: the server keeps serving the dashboard and retries with capped exponential backoff.
## Security The browser never talks to the robot. ``` Browser → local backend (127.0.0.1) → robot ``` - The server binds to `127.0.0.1` by default — nothing is exposed to the LAN. - Robot credentials stay in `.env` on the PC; `/api/config` returns a filtered view with no tokens, passwords or key paths. - No robot port is proxied or forwarded to the page. Set `HOST=0.0.0.0` only if you deliberately want to open the dashboard from a tablet on the same network.
## Connecting the real robot Full procedure: **[docs/AGIBOT_A3_INTEGRATION.md](docs/AGIBOT_A3_INTEGRATION.md)** · Networking and troubleshooting: **[docs/NETWORK.md](docs/NETWORK.md)** Short version, once you have the robot's IP: ```bash ping 192.168.1.50 # 1. is it reachable python scripts/discover_robot.py 192.168.1.50 # 2. what does it expose ``` ```bash # 3. the decisive test - this makes the robot talk python scripts/discover_robot.py 192.168.1.50 --speak "Hello, I am Expedition A3" ``` then in `.env` — **three lines, no code changes**: ```env ROBOT_MODE=real ROBOT_IP=192.168.1.50 ROBOT_PORT=59301 ``` and restart (or `curl -X POST http://localhost:8000/api/config/reload`). > Do **not** use `10.42.10.10`. It appears throughout AgiBot's examples but is > the robot's *internal* address — your PC cannot reach it. Use the head unit's > address on your own network.
## Requirements - **Python 3.9+** (developed and tested on 3.11) - Windows, macOS or Linux - A browser from the last few years Runtime dependencies: `fastapi`, `uvicorn`, `python-dotenv`, `pydantic`, `httpx`, `websockets`. Optional: `pyttsx3` (mock audio), `paramiko` (SSH password auth), `rclpy` (ROS 2 transport — comes from a ROS 2 install, not pip).