A3_text_to_speach/docs/AGIBOT_A3_INTEGRATION.md

# AGIBOT A3 — how the speech integration works

**Short answer: the A3 has a documented, LAN-callable, native text-to-speech HTTP
endpoint, and this project talks to it directly.** No SDK, no login, no ROS, no
audio file transfer.

```
Browser  →  this backend  →  HTTP JSON-RPC  →  A3 head unit (HDU)  →  built-in TTS  →  speaker
```

Everything below is sourced. Claims that could not be verified are tagged
**[UNVERIFIED]** or **[UNKNOWN]** rather than smoothed over, and anything found on
a *different* AgiBot model is labelled as such — an A2/X2/A3-Ultra fact is not an
A3 fact.

> Research date: **2 September 2026**, against AgiBot doc tree **A3 v3.1 / v3.2**.
> If your unit's firmware differs, re-run the discovery procedure in §6.

---

## 1. The interface

AgiBot's A3 developer guide documents two developer transports on the robot:

| Transport | AgiBot's stated use | Notes |
| --- | --- | --- |
| **HTTP JSON-RPC** | "low-frequency, many-to-one calls" | `POST http://{IP}:{PORT}/rpc/{service}/{method}`, `Content-Type: application/json` **mandatory** |
| **ROS 2 topic** | "high-frequency, many-to-many calls" | ROS 2 Jazzy on Fast DDS; default QoS `keep_last`/depth 10/**`best_effort`** |

Source: [03-second_develop_interface_overview](https://open.agibot.com/docs/en/aimdk/a3/v3_2/dev_guide/03-second_develop_interface_overview)

The same page explicitly sanctions calling it from another machine:

> *"For remote calls, replace it with the reachable IP of the corresponding
> device in the current network."*
> *"…there are no restrictions [on language]. As long as you can send and receive
> HTTP requests, you can call all HTTP JSON RPC interfaces."*

**There is no documented gRPC or WebSocket interface on the robot.** (WebSocket
appears only in AgiBot's separate *cloud* LinkSoul agent SDK — see §3.)

### The TTS call

On the **HDU** (the head unit — it owns all audio), port **59301**:

```http
POST http://<ROBOT_IP>:59301/rpc/aimdk.protocol.TTSService/PlayTTS
Content-Type: application/json

{"text": "Hello, welcome to our showroom.",
 "priority_level": "INTERACTION_L6",
 "domain": "voice_control",
 "trace_id": "abc123",
 "is_interrupted": true}
```

```json
{"text": "...", "priority_level": "INTERACTION_L6", "priority_weight": 0,
 "domain": "voice_control", "trace_id": "abc123_18bZZLTk5VfJGSy8Cylsu4",
 "is_sucess": true, "error_message": "", "estimated_duration": 0}
```

Source: [07-02-audio_play](https://open.agibot.com/docs/en/aimdk/a3/v3_2/dev_guide/07-02-audio_play)
· runnable `curl` in AgiBot's own [quick start](https://open.agibot.com/docs/en/aimdk/a3/v3_1/dev_guide/06-second_develop_quick_start)

Companion methods on the same host and port:

| Method | Body | Purpose |
| --- | --- | --- |
| `StopTTSTraceId` | `{"trace_id": "..."}` | interrupt an utterance |
| `GetAudioStatus` | `{"trace_id": "..."}` | `TTSStatusType_Begin\|_Playing\|_End\|_Stop\|_Error\|_InQue\|_NOTInQue` |
| `PlayMediaFile` | `{"file_name": "..."}` | play a file from `/agibot/data/var/interaction/audio/` |

Volume lives on a **different port, 56666**, service `aimdk.protocol.HalAudioService`
(`GetAudioVolume` / `SetAudioVolume`, range 0–100).

> ⚠️ **AgiBot's own hardware warning: do not set volume above 70 — "exceeding
> this range may damage the speaker."** This project does not change volume, so
> it cannot trip that. If you add volume control later, clamp it at 70.

---

## 2. Four traps, and how this project handles each

These are the things that make a naive client fail. All four are handled in
[backend/robot/transports/aimdk_transport.py](../backend/robot/transports/aimdk_transport.py).

| # | The trap | What we do |
| --- | --- | --- |
| 1 | The success flag is spelled **`is_sucess`** — one `c` — in AgiBot's documentation. | Read `is_sucess` **and** `is_success`, so a corrected firmware still works. |
| 2 | The **returned `trace_id` is not the one you sent** — the robot appends a random suffix. Stop only works with the returned value. | The returned id is captured and is what `Stop` uses. |
| 3 | **1024 *bytes*** of UTF-8 is a hard limit (~200 characters). A 1000-character Chinese sentence is ~3000 bytes and is rejected outright. | Text is chunked on sentence boundaries by **byte** length; only the first chunk may interrupt, the rest queue behind it. |
| 4 | Status polling is capped at **≤ 0.2 Hz**; AgiBot warns high-frequency RPC can destabilise the robot. | The liveness probe is a **TCP connect, never an RPC**, and completion is estimated rather than polled. |

Two more behaviours worth knowing, from reading AimRT's own source
([http_rpc_backend.cc](https://raw.githubusercontent.com/AimRT/AimRT/main/src/plugins/net_plugin/http/http_rpc_backend.cc)):

- The route is `"/rpc" + <func name without prefix>` — the `pb:/` prefix seen in
  AimRT function names is **stripped**. Never put `pb:/` in a URL.
- **An RPC-level failure is HTTP 500, not a JSON error.** Unknown route is 404.
  **HTTP 200 means the transport worked, not that the robot spoke** —
  `is_sucess` only confirms the priority check passed.

---

## 3. Why this path, and not the others

| Path | Latency | Reliability | Complexity | Verdict |
| --- | --- | --- | --- | --- |
| **1. HTTP JSON-RPC `PlayTTS`** (this project) | 1 request RTT to enqueue (< 20 ms LAN); time-to-first-audio dominated by the robot's own synthesis | Highest — AgiBot's own primary documented path | Near zero — one HTTP POST | **✅ Chosen** |
| 2. Own TTS → WAV → SCP → `PlayFile` | 1–4 s cold; ~1 RTT for pre-cached phrases | Good; documented offline fallback | Moderate — SSH creds, file lifecycle | Fallback if #1 needs internet and you are air-gapped |
| 3. Own TTS → PCM → ROS 2 `/audiohal/audio/playback` | Best possible — tens of ms, true streaming | High ceiling, high risk (best-effort QoS, focus arbitration) | **High** — ROS 2 Jazzy on the PC, login-gated aarch64 packages, robot reboot into `only_voice` | Only if you need barge-in streaming |
| 4. LinkSoul cloud agent SDK (WebSocket) | Worst — PC → AgiBot cloud → robot | n/a | Commercial credentials, cloud dependency | Wrong tool: a conversational agent layer, not a "say this now" primitive |
| 5. SSH + direct ALSA write | — | — | — | **Do not ship.** No AgiBot document mentions ALSA. `hal_audio` owns the device behind an explicit focus/priority protocol; bypassing it also bypasses the volume damage limit. |

Path 1 wins on every axis the brief asked for: lowest latency, highest
reliability, simplest deployment, best-documented A3 compatibility, and it
scales because it is stateless HTTP.

**Critically, it sends *text*.** The robot synthesises. Nothing is rendered,
encoded or transferred by the PC — which is exactly the architecture the
"fast speech" requirement asks for.

---

## 4. What this project does with it

`A3_TRANSPORT=aimdk` (the default) selects
[aimdk_transport.py](../backend/robot/transports/aimdk_transport.py):

- one **keep-alive** `httpx` client opened at startup and reused for every
  utterance — no handshake per Speak;
- a `Timeout:` header (AimRT honours whole seconds) **and** a client-side
  timeout, so a wedged robot cannot hang the dashboard;
- ack latency measured and reported to the UI;
- `Stop` via `StopTTSTraceId` using the robot-returned `trace_id`;
- errors mapped to messages an operator can act on.

**Every documented value is a setting, not a literal**, because AimRT has no
service-discovery endpoint and AgiBot does not guarantee port or name stability
across firmware:

```env
A3_AIMDK_SERVICE=aimdk.protocol.TTSService
A3_AIMDK_PLAY_METHOD=PlayTTS
A3_AIMDK_STOP_METHOD=StopTTSTraceId
A3_AIMDK_PRIORITY=INTERACTION_L6
A3_AIMDK_MAX_BYTES=1024
```

If your firmware renames something, you edit `.env`. If it uses a wholly
different shape, `A3_TRANSPORT=http` gives you a fully templated HTTP client, and
`ws` / `ros2` / `ssh` are there as further fallbacks.

---

## 5. Connecting the robot — the exact steps

### 5.1 Where the IP goes

**One file: `.env`, in the project root.** Three lines:

```env
ROBOT_MODE=real
ROBOT_IP=192.168.1.50     ← the robot's IP on YOUR network
ROBOT_PORT=59301
```

> **Do not use `10.42.10.10`.** That address appears throughout AgiBot's
> examples, but it is the HDU's address on the robot's *internal* `eth_hdu`
> network. Your PC cannot reach it. Use the HDU's address on your WiFi/LAN.

Then either restart the server, or without restarting:

```bash
curl -X POST http://localhost:8000/api/config/reload
```

### 5.2 The order to do things in

```bash
# 1. reachable at all?
ping 192.168.1.50

# 2. what does it expose? (probes the documented ports and routes, read-only)
python scripts/discover_robot.py 192.168.1.50

# 3. the decisive test - does it actually speak?
python scripts/discover_robot.py 192.168.1.50 --speak "Hello, I am Expedition A3"

# 4. point the app at it, then
python scripts/selftest.py
```

### 5.3 What each failure means

| Result of step 3 | Meaning | Next step |
| --- | --- | --- |
| **The robot speaks** | Path 1 works. Done. | Set `.env`, restart, demo. |
| Connection refused / timeout | 59301 is not bound on the WLAN interface | §6 Step 2 — SSH in and check `ss -ltnp` |
| **HTTP 404** | Port is right, route is wrong | Check `A3_AIMDK_SERVICE` / `A3_AIMDK_PLAY_METHOD`; confirm firmware version |
| **HTTP 500** | Route exists, the RPC failed — **the port is reachable** | Read the response body; check `priority_level` |
| `is_sucess: false` | The robot rejected it | Read `error_message` — usually priority or text length |
| 200 + success but silence | Accepted but not played | Check robot volume, and whether TTS needs internet (§7) |

---

## 6. Discovery procedure for the live robot

`scripts/discover_robot.py` automates the PC-side checks. The rest needs an SSH
session on the robot.

```bash
# From the PC - which of the documented ports answer?
python scripts/discover_robot.py <ROBOT_IP>

# Equivalent by hand. A 404 proves an AimRT HTTP server is listening;
# connection-refused proves it is not.
for p in 59301 56666 51049; do
  curl -s -o /dev/null -w "$p -> %{http_code}\n" -m 3 \
    "http://<ROBOT_IP>:$p/rpc/does.not.Exist/Nope"
done
```

On the robot (username `agi` is **[UNVERIFIED]** — confirm with AgiBot):

```bash
# Is the RPC server bound to 0.0.0.0 or only to the internal 10.42.10.10?
ss -ltnp | grep -E '59301|56666|51049'
ip -br a                                    # the HDU's real WLAN address

# The authoritative port map, from firmware config
grep -rn 'listen_port' /agibot/software/v0/ 2>/dev/null | head -50

# ROS environment (resolves ROS_DOMAIN_ID, only needed for the ros2 transport)
source /agibot/software/v0/entry/env/env.sh || source /agibot/software/v0/entry/env.sh
echo "DOMAIN=$ROS_DOMAIN_ID RMW=$RMW_IMPLEMENTATION"

# The real audio surface — this substitutes for the missing discovery API
ros2 topic list   | grep -Ei 'audio|tts|interaction'
ros2 topic echo --once /interaction/tts_status   # while a PlayTTS is running
```

> **Do not put any of these in a loop.** AgiBot's
> [constraints page](https://open.agibot.com/docs/aimdk/a3/v3_2/dev_guide/04-second_develop_constraints_overview)
> caps status RPCs at ≤ 0.2 Hz, forbids deploying on the MDU ("risks the robot
> falling"), forbids `apt` install/uninstall, and states violations "may void
> warranty."

---

## 7. Open questions — send these to AgiBot

The five that actually affect this project, in priority order. (Chinese versions
follow, since AgiBot support answers faster in Chinese.)

1. **Is port 59301 on the HDU bound to all interfaces and reachable from a PC on
   the same WiFi/LAN using the HDU's WLAN IP — or only on the internal
   `eth_hdu` 10.42.10.x segment?**
   *HDU 的 59301 端口是否监听在所有网络接口上？能否从同一局域网的 PC 通过 HDU 的 WLAN IP 访问？还是仅绑定在内部 eth_hdu (10.42.10.x) 网段？*

2. **Does `TTSService/PlayTTS` require internet access?** Your English v3.2 page
   says it "requires network connection". Which engine performs synthesis, and is
   there a fully on-device/offline TTS mode? What happens with LAN but no internet?
   *`PlayTTS` 是否必须联网？语音合成由哪家引擎提供？是否有完全离线的端侧 TTS 方案？只有局域网、没有外网时会怎样？*

3. **Is there any authentication, token or IP allow-list on the `/rpc/`
   endpoints?** If not, what do you recommend on a shared network? (AimRT sets
   `Access-Control-Allow-Origin: *` unconditionally, so any page on the LAN can
   call these.)
   *`/rpc/` 接口是否有鉴权、token 或 IP 白名单？如果没有，在共享网络下推荐如何防护？*

4. **Can we select voice/timbre, language, speaking rate or emotion per
   `PlayTTS` call?** Is `SetTtsParameters` or an equivalent available on the A3,
   and at which endpoint? **Our unit's voice is currently set to "Yunxiao"
   (teenager, male, multi-language) — is that selectable per request, or only in
   the console?**
   *能否在每次 PlayTTS 调用时指定音色、语言、语速或情感？A3 是否提供 SetTtsParameters？接口地址是什么？
   我们的机器人当前音色为"Yunxiao"（少年、男声、多语言），该音色能否在每次调用时指定，还是只能在控制台配置？*

   > **Operator-reported, 2026-09-02:** the robot's configured voice is
   > **"Yunxiao"** — labelled *teenager | male*, *multi-language*, and marked as
   > the default (默). It was shown in a voice-library UI offering per-voice
   > "Speech Synthesis", which is consistent with AgiBot's LinkSoul timbre
   > library (30–40 timbres + voice cloning) — though that link is inference, not
   > something any published document states. This is **not** in AgiBot's docs;
   > it comes from the console for this specific unit. It suggests voice
   > selection happens on a platform/console rather than in the `PlayTTS` body,
   > which is exactly what question 4 needs to settle.

5. **Please confirm the complete `priority_level` enum** accepted by A3
   `PlayTTS`, and the semantics of `priority_weight` and `is_interrupted`. Your
   X2 documentation publishes seven levels — do the same apply to the A3?
   *请提供 A3 PlayTTS 支持的完整 priority_level 枚举，以及 priority_weight 和 is_interrupted 的语义。X2 文档公布了七个等级，A3 是否相同？*

Lower priority, worth asking in the same message: the response envelope
(is there a `header.code`/`header.msg` wrapper?); whether the AimDK SDK download
needs a verified-customer account; whether ports 59301/56666/51049 are stable
across firmware; and the `PlayMediaFile` sample rate — **your English v3.2 page
says 24 kHz and your Chinese v3.2 page says 16 kHz for the same call.**

---

## 8. Known contradictions in AgiBot's own documentation

Recorded so nobody wastes an afternoon on them.

| # | Contradiction | Impact here |
| --- | --- | --- |
| 1 | `PlayMediaFile` sample rate: **EN v3.2 says 24 kHz, ZH v3.2 says 16 kHz**, same page | None — we send text, not files |
| 2 | Microphone: v3.1 and v3.2 dev guides say the built-in mic is "not yet configured or enabled"; the User Manual and product page advertise an 8-mic array | None — output only |
| 3 | `env.sh` path: `/agibot/software/v0/entry/env.sh` vs `…/entry/env/env.sh` | Only matters for the `ros2` transport |
| 4 | ROS 2 distro: AgiBot says the A3 is **Jazzy**; AimRT's `ros2_plugin` doc says Humble | Only matters for the `ros2` transport |
| 5 | SDK wheel: the A3 v3.2 guide instructs `a2_aimdk-3.0.0…whl` in most places and `a3_aimdk-3.0.0…whl` in one — an **A2 artifact inside the A3 doc tree** | None — Path 1 needs no SDK |

**`ROS_DOMAIN_ID` for the A3 is [UNKNOWN]** — it is not printed in any A3
document. A2 uses 232 (**different model**), and one third-party A3-Ultra
deployment also uses 232 — suggestive, not proof. Read it from `env.sh` on the
robot.

---

## 9. Security note worth raising internally

No authentication scheme is documented for the `/rpc/` endpoints, and AimRT sets
`Access-Control-Allow-Origin: *` unconditionally with OPTIONS preflight handled.
**Taken together: any host — or any web page opened in any browser — on the same
LAN as the robot can make it speak.**

This project does not widen that exposure. The browser talks only to
`127.0.0.1:8000`; robot credentials and addresses stay server-side. But on a
shared or public network, treat the robot's RPC ports as unauthenticated and
segment the network accordingly.

---

## 10. Testing the real path before the robot arrives

`ROBOT_MODE=mock` tests the application. To test **this transport** — the URL
shape, the JSON body, the `trace_id` round trip, chunking, Stop, and the error
paths — run the local stand-in:

```bash
python scripts/fake_a3_server.py          # implements the documented contract on :59301
```

then in `.env`:

```env
ROBOT_MODE=real
ROBOT_IP=127.0.0.1
ROBOT_PORT=59301
```

and use the dashboard normally. Every utterance prints in the stand-in's console.

> It is a **test double written from public documentation**, not the robot.
> Passing against it proves the client is well-formed — not that the firmware
> behaves identically. Verify on the real unit.

---

## 11. Sources

**Official AgiBot, A3-specific**
- Audio interfaces (TTS, status, stop, volume, ROS 2 audio) — [EN](https://open.agibot.com/docs/en/aimdk/a3/v3_2/dev_guide/07-02-audio_play) · [ZH](https://open.agibot.com/docs/aimdk/a3/v3_2/dev_guide/07-02-audio_play)
- Interface overview (the two transports, URL shape, remote-call sanction) — [EN](https://open.agibot.com/docs/en/aimdk/a3/v3_2/dev_guide/03-second_develop_interface_overview)
- Quick start (the runnable `PlayTTS` curl) — [EN](https://open.agibot.com/docs/en/aimdk/a3/v3_1/dev_guide/06-second_develop_quick_start)
- Resource manager (`PlayFile`, WAV/PCM rules, resource dirs) — [EN](https://open.agibot.com/docs/en/aimdk/a3/v3_2/dev_guide/07-05-resource_manager)
- Interactive guide (`audio_msgs/msg/AudioPlayback`, focus services, `only_voice`) — [EN](https://open.agibot.com/docs/en/aimdk/a3/v3_2/dev_guide/08-interactive_guide)
- Development constraints (polling limits, MDU, apt, warranty) — [ZH](https://open.agibot.com/docs/aimdk/a3/v3_2/dev_guide/04-second_develop_constraints_overview)
- Hardware/software overview (HDU/MDU, internal IPs, Ubuntu 24.04) — [ZH](https://open.agibot.com/docs/aimdk/a3/v3_1/dev_guide/01-a3_overview)
- AimDK docs root — https://open.agibot.com/docs/aimdk (SDK downloads are login-gated)

**AimRT framework** (transport semantics, read from source)
- [net_plugin docs](https://docs.aimrt.org/tutorials/plugins/net_plugin.html) · [http_rpc_backend.cc](https://raw.githubusercontent.com/AimRT/AimRT/main/src/plugins/net_plugin/http/http_rpc_backend.cc)

**Different model — do not assume it applies to the A3**
- X2 voice interface, full 7-value priority enum — https://x2-aimdk.agibot.com/en/dev/Interface/interactor/voice.html

**Third-party corroboration** (an A3 **Ultra** deployment, not a specification)
- [tzj-hub/agibot_A3_sells_cars](https://github.com/tzj-hub/agibot_A3_sells_cars) — same base URL, methods and payload shape; reads `is_sucess` with an `is_success` fallback; its `a3_interfaces.py` is the only source anywhere for the `header.code`/`header.msg` error envelope, which is why this project reads that envelope defensively rather than depending on it.