AGIBOT A3 · Voice Control

A local web dashboard for making an AGIBOT A3 humanoid speak typed text through its own built-in speaker.

PC Browser  →  Local Web App  →  Robot Service Layer  →  A3 Adapter  →  AGIBOT A3  →  Speaker

It runs today, with no robot attached, against a built-in simulator. When the robot arrives you change the IP in one file and switch one setting.

The A3's speech interface is already implemented. AgiBot publicly documents a native text-to-speech HTTP endpoint on the robot, and this project speaks it:

POST http://<robot>:59301/rpc/aimdk.protocol.TTSService/PlayTTS
Content-Type: application/json
{"text": "...", "priority_level": "INTERACTION_L6", "domain": "voice_control",
 "trace_id": "...", "is_interrupted": true}

Text goes in, the robot's own TTS comes out of its own speaker. No audio is generated or transferred by the PC, which is what makes it fast. Full detail, sources and open questions: docs/AGIBOT_A3_INTEGRATION.md.

Quick start

start.bat

…then open http://localhost:8000.

Or manually:

pip install -r requirements.txt
python backend/main.py

The app starts in Mock Mode: the dashboard, the API, the status pipeline, the history and every error path work exactly as they will with the real robot — the robot itself is simulated.

You will hear it. The simulator speaks each utterance through this PC's speakers, so you can rehearse a demo — wording, pacing, the Stop button — before the A3 is on the network. The real audio drives the UI: Completed appears when the sound actually stops, and Stop cuts the voice mid-word.

Two voice engines, set by MOCK_VOICE_ENGINE in .env:

	system	gemini
Sound	robotic, built-in OS voice	natural neural voice
New text	instant	~4 s (sentence) – 8 s (paragraph)
Repeated text	instant	instant — cached on disk
Needs	nothing	GEMINI_API_KEY + internet

python scripts/voices.py          # list both engines' voices
python scripts/voices.py --demo   # hear the system ones

Saved audio

Every line spoken with the neural voice is saved to audio_library/ as an ordinary .wav named after its text:

audio_library/
    welcome-to-our-showroom-78e24feb.wav
    please-follow-me-to-the-first-exhibit-653c14ff.wav
    index.json

That means a line spoken once replays instantly — no synthesis, no network, no wait — and the files are yours: play them in any media player, drop them into a video edit, or put them on a stand's playlist.

The dashboard's Saved audio panel lists them with play and download buttons, and history rows that already have audio get a ▶ for instant replay.

Before a live demo, warm your lines so nothing waits on synthesis:

python scripts/warm_voice.py --file demo_lines.txt
python scripts/warm_voice.py --stats

If Gemini is unreachable — no network, bad key, quota — the simulator silently falls back to the built-in voice, so a network problem never leaves you with silence.

Pick one with MOCK_VOICE= in .env; silence it entirely with MOCK_LOCAL_AUDIO=false.

This is the simulator's voice — the real A3 synthesises its own speech on-board. The robot's configured voice is "Yunxiao" (teenager, male, multi-language). Windows ships no teenage voice, so the default here is its lighter adult male voice (Mark) pitched up (MOCK_SPEECH_PITCH=3) to read younger. It is an approximation for rehearsal, not a match.

Verify the whole stack end to end at any time:

python scripts/selftest.py      # 31 checks: REST, WebSocket, speak, stop, errors, history

Testing the real robot path before the robot exists

Mock mode tests the application. To exercise the actual A3 wire protocol — the URL, the JSON body, the trace_id round trip, byte-limit chunking, Stop — run the local stand-in:

python scripts/fake_a3_server.py     # implements the documented A3 contract on :59301

then set ROBOT_MODE=real, ROBOT_IP=127.0.0.1 in .env. Every utterance prints in the stand-in's console. This project's aimdk transport passes all 31 checks against it.

Using the dashboard

Action	How
Speak	Type, then click Speak or press Ctrl + Enter
Stop	Stop button or Esc
Clear the box	Clear button
Repeat something	Click any entry in History — it goes back in the box
Reconnect	The ↻ button in the header

Enter inserts a newline and does not send — a half-typed sentence should never reach the robot's speaker mid-demo.

The header always states the truth about the connection: green Robot Connected, amber Connecting…, red Robot Disconnected, plus the measured round-trip latency to the robot.

Project layout

A3_text_to_speach/
│
├── frontend/                  no build step - plain ES modules, served by the backend
│   ├── index.html
│   ├── styles/main.css
│   └── js/
│       ├── app.js             wiring + speech lifecycle
│       ├── api.js             REST client
│       ├── socket.js          WebSocket client (auto-reconnect, RTT ping)
│       └── ui.js              all DOM rendering
│
├── backend/
│   ├── main.py                entry point: app factory, static serving, error mapping
│   ├── api/
│   │   ├── routes.py          REST endpoints
│   │   ├── websocket.py       live status/lifecycle channel
│   │   └── schemas.py
│   ├── services/
│   │   ├── speech_service.py  validation, single-flight, interrupt, latency
│   │   └── history.py
│   ├── robot/
│   │   ├── base.py            ← the only contract the app depends on
│   │   ├── factory.py         mode → adapter
│   │   ├── manager.py         connect / health-poll / backoff reconnect
│   │   ├── mock_robot.py      the simulator
│   │   ├── agibot_a3.py       ← THE INTEGRATION POINT
│   │   └── transports/
│   │       ├── aimdk_transport.py   ← AgiBot's documented A3 speech RPC (default)
│   │       └── http · ws · ros2 · ssh   fallbacks, fully config-driven
│   ├── core/                  event bus, text utils, logging
│   └── config/settings.py     every tunable, loaded from .env
│
├── docs/
│   ├── AGIBOT_A3_INTEGRATION.md   how A3 speech works + what to fill in
│   └── NETWORK.md                 PC ↔ robot networking and troubleshooting
│
├── scripts/
│   ├── voices.py              list / audition the simulator's voices
│   ├── warm_voice.py          pre-synthesise demo lines (instant playback)
│   ├── selftest.py            end-to-end test (31 checks)
│   ├── discover_robot.py      probe a robot IP for its speech interface
│   └── fake_a3_server.py      local stand-in for the A3's RPC, to test the real path
│
├── audio_library/             saved .wav files + index (instant replay)
├── pronunciation.json         how the simulator says tricky names
├── .env                       ← YOUR CONFIGURATION (robot IP goes here)
├── .env.example               documented template
├── requirements.txt
└── start.bat / start.sh

Why it is layered this way

Each arrow is a seam you can replace without touching the others:

Browser
   │  REST for commands, WebSocket for state
Backend API            (api/)
   │  domain objects only - no HTTP, no sockets
Speech Service         (services/)
   │  RobotAdapter interface
Robot Adapter          (robot/mock_robot.py | robot/agibot_a3.py)
   │  SpeechTransport interface
Transport              (robot/transports/*)
   │
AGIBOT A3

Nothing above robot/ knows a robot SDK exists. Adding an AgiBot X2, a second robot, or a different TTS engine means writing one adapter and adding one line to factory.py — the UI, the API and the service layer are untouched.

Configuration

Everything lives in .env (created from .env.example on first run). The source code contains no IP address, port or endpoint.

ROBOT_MODE=mock          # mock | real   ← the switch
ROBOT_IP=                # ← the robot's IP goes here
ROBOT_PORT=59301         # AgiBot's documented A3 TTS RPC port
A3_TRANSPORT=aimdk       # aimdk | http | ws | ros2 | ssh

After editing .env, either restart the server or call:

curl -X POST http://localhost:8000/api/config/reload

which rebuilds the robot adapter in place — no restart, no code change.

Mock-mode knobs worth knowing

Setting	Effect
MOCK_FAILURE_RATE=0.3	3 in 10 utterances fail — exercises the error UI
MOCK_FLAKY_CONNECTION=true	the link drops periodically — exercises reconnect
MOCK_LOCAL_AUDIO=true	on by default — actually speaks through this PC's speakers
MOCK_VOICE_ENGINE=gemini	natural neural voice instead of the robotic built-in one
GEMINI_VOICE=Puck	which neural voice (Puck/Fenrir read youngest + male)
MOCK_VOICE=Mark	which system voice to use, when engine is system
MOCK_SPEECH_PITCH=3	raise the pitch — approximates the robot's young male voice
MOCK_SPEECH_RATE=-2	slow the simulator down for a noisy room (−10…10)
SPEECH_ALLOW_INTERRUPT=false	a second Speak while talking is rejected instead of interrupting

API

Method	Path	Purpose
GET	/api/health	web app liveness (always 200, even with the robot offline)
GET	/api/robot/status	connection state, latency, uptime, config problems
POST	/api/robot/speak	{"text": "..."} → speak it
POST	/api/robot/stop	interrupt the current utterance
POST	/api/robot/reconnect	retry the connection immediately
GET	/api/robot/diagnostics	adapter + transport detail, for troubleshooting
GET	/api/speech/history	recent utterances
DELETE	/api/speech/history	clear it
GET	/api/audio	saved clips + library stats
GET	/api/audio/{id}/file	the .wav itself (play or download)
DELETE	/api/audio/{id}	delete one clip
DELETE	/api/audio	delete all saved audio
GET	/api/config	non-secret configuration the UI needs
POST	/api/config/reload	re-read .env and rebuild the adapter
WS	/ws	live status, speech lifecycle, history updates

Interactive docs: http://localhost:8000/api/docs

curl -X POST http://localhost:8000/api/robot/speak \
     -H "Content-Type: application/json" \
     -d "{\"text\": \"Hello, welcome to our showroom.\"}"

{ "success": true, "status": "processing",
  "requestId": "3c7cfe41cf7d", "ackLatencyMs": 56 }

/api/robot/speak returns when the robot has accepted the utterance, not when it stops talking. A 12-second sentence must not look like a 12-second-slow button. The rest of the lifecycle — speaking, completed, failures — arrives on the WebSocket.

How the speed is achieved

Decision	Why
Text is sent to the robot; audio is never generated on the PC	no synthesis, no file, no upload — one small HTTP request
One persistent connection, opened at startup and reused	no TCP/TLS handshake per utterance
HTTP response returns at acknowledgement	button feels instant; speaking progress streams separately
WebSocket push, zero polling	the UI changes the moment the robot does
Health probe is separate from the speech path	a slow health check can never delay a Speak
Short connect timeout (3 s), long request timeout (8 s)	an offline robot fails fast; a busy one is not cut off

Error handling

Situation	What you see
Robot offline	Robot is offline. Check the robot IP address and network connection.
ROBOT_IP empty while ROBOT_MODE=real	a configuration banner naming the exact .env key
Invalid IP / hostname	configuration error before any connection is attempted
Connection timeout	fails after 3 s, UI stays responsive, retry continues in the background
TTS failure	Speech request failed. + the robot's own reason in the server log
Empty text	Speak is disabled; the API returns 400
Text too long	rejected with the actual limit named
Disconnect mid-utterance	status flips immediately and the utterance is marked failed
Backend stopped	the page says so rather than showing a stale green light

The robot being unreachable is treated as normal, not exceptional: the server keeps serving the dashboard and retries with capped exponential backoff.

Security

The browser never talks to the robot.

Browser  →  local backend (127.0.0.1)  →  robot

• The server binds to 127.0.0.1 by default — nothing is exposed to the LAN.
• Robot credentials stay in .env on the PC; /api/config returns a filtered view with no tokens, passwords or key paths.
• No robot port is proxied or forwarded to the page.

Set HOST=0.0.0.0 only if you deliberately want to open the dashboard from a tablet on the same network.

Connecting the real robot

Full procedure: docs/AGIBOT_A3_INTEGRATION.md · Networking and troubleshooting: docs/NETWORK.md

Short version, once you have the robot's IP:

ping 192.168.1.50                          # 1. is it reachable
python scripts/discover_robot.py 192.168.1.50   # 2. what does it expose

# 3. the decisive test - this makes the robot talk
python scripts/discover_robot.py 192.168.1.50 --speak "Hello, I am Expedition A3"

then in .env — three lines, no code changes:

ROBOT_MODE=real
ROBOT_IP=192.168.1.50
ROBOT_PORT=59301

and restart (or curl -X POST http://localhost:8000/api/config/reload).

Do not use 10.42.10.10. It appears throughout AgiBot's examples but is the robot's internal address — your PC cannot reach it. Use the head unit's address on your own network.

Requirements

• Python 3.9+ (developed and tested on 3.11)
• Windows, macOS or Linux
• A browser from the last few years

Runtime dependencies: fastapi, uvicorn, python-dotenv, pydantic, httpx, websockets. Optional: pyttsx3 (mock audio), paramiko (SSH password auth), rclpy (ROS 2 transport — comes from a ROS 2 install, not pip).