A3_text_to_speach/README.md

# AGIBOT A3 · Voice Control

A local web dashboard for making an **AGIBOT A3** humanoid speak typed text through
its own built-in speaker.

```
PC Browser  →  Local Web App  →  Robot Service Layer  →  A3 Adapter  →  AGIBOT A3  →  Speaker
```

It runs today, with **no robot attached**, against a built-in simulator. When the
robot arrives you change the IP in one file and switch one setting.

**The A3's speech interface is already implemented.** AgiBot publicly documents a
native text-to-speech HTTP endpoint on the robot, and this project speaks it:

```http
POST http://<robot>:59301/rpc/aimdk.protocol.TTSService/PlayTTS
Content-Type: application/json
{"text": "...", "priority_level": "INTERACTION_L6", "domain": "voice_control",
 "trace_id": "...", "is_interrupted": true}
```

Text goes in, the robot's own TTS comes out of its own speaker. **No audio is
generated or transferred by the PC**, which is what makes it fast. Full detail,
sources and open questions: **[docs/AGIBOT_A3_INTEGRATION.md](docs/AGIBOT_A3_INTEGRATION.md)**.

<br>

## Quick start

```bat
start.bat
```

…then open **http://localhost:8000**.

Or manually:

```bash
pip install -r requirements.txt
python backend/main.py
```

The app starts in **Mock Mode**: the dashboard, the API, the status pipeline, the
history and every error path work exactly as they will with the real robot — the
robot itself is simulated.

**You will hear it.** The simulator speaks each utterance through this PC's
speakers, so you can rehearse a demo — wording, pacing, the Stop button — before
the A3 is on the network. The real audio drives the UI: *Completed* appears when
the sound actually stops, and **Stop** cuts the voice mid-word.

Two voice engines, set by `MOCK_VOICE_ENGINE` in `.env`:

| | `system` | `gemini` |
| --- | --- | --- |
| Sound | robotic, built-in OS voice | **natural neural voice** |
| New text | instant | ~4 s (sentence) – 8 s (paragraph) |
| Repeated text | instant | **instant** — cached on disk |
| Needs | nothing | `GEMINI_API_KEY` + internet |

```bash
python scripts/voices.py          # list both engines' voices
python scripts/voices.py --demo   # hear the system ones
```

### Saved audio

Every line spoken with the neural voice is saved to **`audio_library/`** as an
ordinary `.wav` named after its text:

```
audio_library/
    welcome-to-our-showroom-78e24feb.wav
    please-follow-me-to-the-first-exhibit-653c14ff.wav
    index.json
```

That means a line spoken once **replays instantly** — no synthesis, no network,
no wait — and the files are yours: play them in any media player, drop them into
a video edit, or put them on a stand's playlist.

The dashboard's **Saved audio** panel lists them with play and download buttons,
and history rows that already have audio get a ▶ for instant replay.

**Before a live demo, warm your lines** so nothing waits on synthesis:

```bash
python scripts/warm_voice.py --file demo_lines.txt
python scripts/warm_voice.py --stats
```

If Gemini is unreachable — no network, bad key, quota — the simulator silently
falls back to the built-in voice, so a network problem never leaves you with
silence.

Pick one with `MOCK_VOICE=` in `.env`; silence it entirely with
`MOCK_LOCAL_AUDIO=false`.

This is the *simulator's* voice — the real A3 synthesises its own speech
on-board. The robot's configured voice is **"Yunxiao"** (teenager, male,
multi-language). Windows ships no teenage voice, so the default here is its
lighter adult male voice (**Mark**) pitched up (`MOCK_SPEECH_PITCH=3`) to read
younger. It is an approximation for rehearsal, not a match.

Verify the whole stack end to end at any time:

```bash
python scripts/selftest.py      # 31 checks: REST, WebSocket, speak, stop, errors, history
```

### Testing the real robot path before the robot exists

Mock mode tests the *application*. To exercise the actual A3 wire protocol —
the URL, the JSON body, the `trace_id` round trip, byte-limit chunking, Stop —
run the local stand-in:

```bash
python scripts/fake_a3_server.py     # implements the documented A3 contract on :59301
```

then set `ROBOT_MODE=real`, `ROBOT_IP=127.0.0.1` in `.env`. Every utterance
prints in the stand-in's console. This project's `aimdk` transport passes all 31
checks against it.

<br>

## Using the dashboard

| Action | How |
| --- | --- |
| Speak | Type, then click **Speak** or press **Ctrl + Enter** |
| Stop | **Stop** button or **Esc** |
| Clear the box | **Clear** button |
| Repeat something | Click any entry in **History** — it goes back in the box |
| Reconnect | The ↻ button in the header |

`Enter` inserts a newline and does **not** send — a half-typed sentence should
never reach the robot's speaker mid-demo.

The header always states the truth about the connection: green *Robot Connected*,
amber *Connecting…*, red *Robot Disconnected*, plus the measured round-trip
latency to the robot.

<br>

## Project layout

```
A3_text_to_speach/
│
├── frontend/                  no build step - plain ES modules, served by the backend
│   ├── index.html
│   ├── styles/main.css
│   └── js/
│       ├── app.js             wiring + speech lifecycle
│       ├── api.js             REST client
│       ├── socket.js          WebSocket client (auto-reconnect, RTT ping)
│       └── ui.js              all DOM rendering
│
├── backend/
│   ├── main.py                entry point: app factory, static serving, error mapping
│   ├── api/
│   │   ├── routes.py          REST endpoints
│   │   ├── websocket.py       live status/lifecycle channel
│   │   └── schemas.py
│   ├── services/
│   │   ├── speech_service.py  validation, single-flight, interrupt, latency
│   │   └── history.py
│   ├── robot/
│   │   ├── base.py            ← the only contract the app depends on
│   │   ├── factory.py         mode → adapter
│   │   ├── manager.py         connect / health-poll / backoff reconnect
│   │   ├── mock_robot.py      the simulator
│   │   ├── agibot_a3.py       ← THE INTEGRATION POINT
│   │   └── transports/
│   │       ├── aimdk_transport.py   ← AgiBot's documented A3 speech RPC (default)
│   │       └── http · ws · ros2 · ssh   fallbacks, fully config-driven
│   ├── core/                  event bus, text utils, logging
│   └── config/settings.py     every tunable, loaded from .env
│
├── docs/
│   ├── AGIBOT_A3_INTEGRATION.md   how A3 speech works + what to fill in
│   └── NETWORK.md                 PC ↔ robot networking and troubleshooting
│
├── scripts/
│   ├── voices.py              list / audition the simulator's voices
│   ├── warm_voice.py          pre-synthesise demo lines (instant playback)
│   ├── selftest.py            end-to-end test (31 checks)
│   ├── discover_robot.py      probe a robot IP for its speech interface
│   └── fake_a3_server.py      local stand-in for the A3's RPC, to test the real path
│
├── audio_library/             saved .wav files + index (instant replay)
├── pronunciation.json         how the simulator says tricky names
├── .env                       ← YOUR CONFIGURATION (robot IP goes here)
├── .env.example               documented template
├── requirements.txt
└── start.bat / start.sh
```

### Why it is layered this way

Each arrow is a seam you can replace without touching the others:

```
Browser
   │  REST for commands, WebSocket for state
Backend API            (api/)
   │  domain objects only - no HTTP, no sockets
Speech Service         (services/)
   │  RobotAdapter interface
Robot Adapter          (robot/mock_robot.py | robot/agibot_a3.py)
   │  SpeechTransport interface
Transport              (robot/transports/*)
   │
AGIBOT A3
```

Nothing above `robot/` knows a robot SDK exists. Adding an AgiBot X2, a second
robot, or a different TTS engine means writing one adapter and adding one line to
`factory.py` — the UI, the API and the service layer are untouched.

<br>

## Configuration

Everything lives in **`.env`** (created from `.env.example` on first run). The
source code contains no IP address, port or endpoint.

```env
ROBOT_MODE=mock          # mock | real   ← the switch
ROBOT_IP=                # ← the robot's IP goes here
ROBOT_PORT=59301         # AgiBot's documented A3 TTS RPC port
A3_TRANSPORT=aimdk       # aimdk | http | ws | ros2 | ssh
```

After editing `.env`, either restart the server or call:

```bash
curl -X POST http://localhost:8000/api/config/reload
```

which rebuilds the robot adapter in place — no restart, no code change.

### Mock-mode knobs worth knowing

| Setting | Effect |
| --- | --- |
| `MOCK_FAILURE_RATE=0.3` | 3 in 10 utterances fail — exercises the error UI |
| `MOCK_FLAKY_CONNECTION=true` | the link drops periodically — exercises reconnect |
| `MOCK_LOCAL_AUDIO=true` | **on by default** — actually speaks through this PC's speakers |
| `MOCK_VOICE_ENGINE=gemini` | natural neural voice instead of the robotic built-in one |
| `GEMINI_VOICE=Puck` | which neural voice (Puck/Fenrir read youngest + male) |
| `MOCK_VOICE=Mark` | which *system* voice to use, when engine is `system` |
| `MOCK_SPEECH_PITCH=3` | raise the pitch — approximates the robot's young male voice |
| `MOCK_SPEECH_RATE=-2` | slow the simulator down for a noisy room (−10…10) |
| `SPEECH_ALLOW_INTERRUPT=false` | a second Speak while talking is rejected instead of interrupting |

<br>

## API

| Method | Path | Purpose |
| --- | --- | --- |
| `GET` | `/api/health` | web app liveness (always 200, even with the robot offline) |
| `GET` | `/api/robot/status` | connection state, latency, uptime, config problems |
| `POST` | `/api/robot/speak` | `{"text": "..."}` → speak it |
| `POST` | `/api/robot/stop` | interrupt the current utterance |
| `POST` | `/api/robot/reconnect` | retry the connection immediately |
| `GET` | `/api/robot/diagnostics` | adapter + transport detail, for troubleshooting |
| `GET` | `/api/speech/history` | recent utterances |
| `DELETE` | `/api/speech/history` | clear it |
| `GET` | `/api/audio` | saved clips + library stats |
| `GET` | `/api/audio/{id}/file` | the `.wav` itself (play or download) |
| `DELETE` | `/api/audio/{id}` | delete one clip |
| `DELETE` | `/api/audio` | delete all saved audio |
| `GET` | `/api/config` | non-secret configuration the UI needs |
| `POST` | `/api/config/reload` | re-read `.env` and rebuild the adapter |
| `WS` | `/ws` | live status, speech lifecycle, history updates |

Interactive docs: **http://localhost:8000/api/docs**

```bash
curl -X POST http://localhost:8000/api/robot/speak \
     -H "Content-Type: application/json" \
     -d "{\"text\": \"Hello, welcome to our showroom.\"}"
```

```json
{ "success": true, "status": "processing",
  "requestId": "3c7cfe41cf7d", "ackLatencyMs": 56 }
```

**`/api/robot/speak` returns when the robot has *accepted* the utterance, not when
it stops talking.** A 12-second sentence must not look like a 12-second-slow
button. The rest of the lifecycle — `speaking`, `completed`, failures — arrives on
the WebSocket.

<br>

## How the speed is achieved

| Decision | Why |
| --- | --- |
| Text is sent to the robot; audio is never generated on the PC | no synthesis, no file, no upload — one small HTTP request |
| One persistent connection, opened at startup and reused | no TCP/TLS handshake per utterance |
| HTTP response returns at acknowledgement | button feels instant; speaking progress streams separately |
| WebSocket push, zero polling | the UI changes the moment the robot does |
| Health probe is separate from the speech path | a slow health check can never delay a Speak |
| Short connect timeout (3 s), long request timeout (8 s) | an offline robot fails fast; a busy one is not cut off |

<br>

## Error handling

| Situation | What you see |
| --- | --- |
| Robot offline | `Robot is offline. Check the robot IP address and network connection.` |
| `ROBOT_IP` empty while `ROBOT_MODE=real` | a configuration banner naming the exact `.env` key |
| Invalid IP / hostname | configuration error before any connection is attempted |
| Connection timeout | fails after 3 s, UI stays responsive, retry continues in the background |
| TTS failure | `Speech request failed.` + the robot's own reason in the server log |
| Empty text | Speak is disabled; the API returns `400` |
| Text too long | rejected with the actual limit named |
| Disconnect mid-utterance | status flips immediately and the utterance is marked failed |
| Backend stopped | the page says so rather than showing a stale green light |

The robot being unreachable is treated as normal, not exceptional: the server
keeps serving the dashboard and retries with capped exponential backoff.

<br>

## Security

The browser never talks to the robot.

```
Browser  →  local backend (127.0.0.1)  →  robot
```

- The server binds to `127.0.0.1` by default — nothing is exposed to the LAN.
- Robot credentials stay in `.env` on the PC; `/api/config` returns a filtered
  view with no tokens, passwords or key paths.
- No robot port is proxied or forwarded to the page.

Set `HOST=0.0.0.0` only if you deliberately want to open the dashboard from a
tablet on the same network.

<br>

## Connecting the real robot

Full procedure: **[docs/AGIBOT_A3_INTEGRATION.md](docs/AGIBOT_A3_INTEGRATION.md)**
· Networking and troubleshooting: **[docs/NETWORK.md](docs/NETWORK.md)**

Short version, once you have the robot's IP:

```bash
ping 192.168.1.50                          # 1. is it reachable
python scripts/discover_robot.py 192.168.1.50   # 2. what does it expose
```

```bash
# 3. the decisive test - this makes the robot talk
python scripts/discover_robot.py 192.168.1.50 --speak "Hello, I am Expedition A3"
```

then in `.env` — **three lines, no code changes**:

```env
ROBOT_MODE=real
ROBOT_IP=192.168.1.50
ROBOT_PORT=59301
```

and restart (or `curl -X POST http://localhost:8000/api/config/reload`).

> Do **not** use `10.42.10.10`. It appears throughout AgiBot's examples but is
> the robot's *internal* address — your PC cannot reach it. Use the head unit's
> address on your own network.

<br>

## Requirements

- **Python 3.9+** (developed and tested on 3.11)
- Windows, macOS or Linux
- A browser from the last few years

Runtime dependencies: `fastapi`, `uvicorn`, `python-dotenv`, `pydantic`, `httpx`,
`websockets`. Optional: `pyttsx3` (mock audio), `paramiko` (SSH password auth),
`rclpy` (ROS 2 transport — comes from a ROS 2 install, not pip).