Sedra/A3_text_to_speach
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
A3_text_to_speach/docs/AGIBOT_A3_INTEGRATION.md
Sidra 6e939ce6c3 Update 2026-09-03 00:10
2026-09-03 00:10:18 +04:00

19 KiB
Raw Blame History

AGIBOT A3 — how the speech integration works

Short answer: the A3 has a documented, LAN-callable, native text-to-speech HTTP endpoint, and this project talks to it directly. No SDK, no login, no ROS, no audio file transfer.

Browser  →  this backend  →  HTTP JSON-RPC  →  A3 head unit (HDU)  →  built-in TTS  →  speaker

Everything below is sourced. Claims that could not be verified are tagged [UNVERIFIED] or [UNKNOWN] rather than smoothed over, and anything found on a different AgiBot model is labelled as such — an A2/X2/A3-Ultra fact is not an A3 fact.

Research date: 2 September 2026, against AgiBot doc tree A3 v3.1 / v3.2. If your unit's firmware differs, re-run the discovery procedure in §6.

1. The interface

AgiBot's A3 developer guide documents two developer transports on the robot:

Transport AgiBot's stated use Notes
HTTP JSON-RPC "low-frequency, many-to-one calls" POST http://{IP}:{PORT}/rpc/{service}/{method}, Content-Type: application/json mandatory
ROS 2 topic "high-frequency, many-to-many calls" ROS 2 Jazzy on Fast DDS; default QoS keep_last/depth 10/best_effort

Source: 03-second_develop_interface_overview

The same page explicitly sanctions calling it from another machine:

"For remote calls, replace it with the reachable IP of the corresponding device in the current network." "…there are no restrictions [on language]. As long as you can send and receive HTTP requests, you can call all HTTP JSON RPC interfaces."

There is no documented gRPC or WebSocket interface on the robot. (WebSocket appears only in AgiBot's separate cloud LinkSoul agent SDK — see §3.)

The TTS call

On the HDU (the head unit — it owns all audio), port 59301:

POST http://<ROBOT_IP>:59301/rpc/aimdk.protocol.TTSService/PlayTTS
Content-Type: application/json

{"text": "Hello, welcome to our showroom.",
 "priority_level": "INTERACTION_L6",
 "domain": "voice_control",
 "trace_id": "abc123",
 "is_interrupted": true}

{"text": "...", "priority_level": "INTERACTION_L6", "priority_weight": 0,
 "domain": "voice_control", "trace_id": "abc123_18bZZLTk5VfJGSy8Cylsu4",
 "is_sucess": true, "error_message": "", "estimated_duration": 0}

Source: 07-02-audio_play · runnable curl in AgiBot's own quick start

Companion methods on the same host and port:

Method Body Purpose
StopTTSTraceId {"trace_id": "..."} interrupt an utterance
GetAudioStatus {"trace_id": "..."} TTSStatusType_Begin|_Playing|_End|_Stop|_Error|_InQue|_NOTInQue
PlayMediaFile {"file_name": "..."} play a file from /agibot/data/var/interaction/audio/

Volume lives on a different port, 56666, service aimdk.protocol.HalAudioService (GetAudioVolume / SetAudioVolume, range 0–100).

⚠️ AgiBot's own hardware warning: do not set volume above 70 — "exceeding this range may damage the speaker." This project does not change volume, so it cannot trip that. If you add volume control later, clamp it at 70.

2. Four traps, and how this project handles each

These are the things that make a naive client fail. All four are handled in backend/robot/transports/aimdk_transport.py.

# The trap What we do
1 The success flag is spelled is_sucess — one c — in AgiBot's documentation. Read is_sucess and is_success, so a corrected firmware still works.
2 The returned trace_id is not the one you sent — the robot appends a random suffix. Stop only works with the returned value. The returned id is captured and is what Stop uses.
3 1024 bytes of UTF-8 is a hard limit (~200 characters). A 1000-character Chinese sentence is ~3000 bytes and is rejected outright. Text is chunked on sentence boundaries by byte length; only the first chunk may interrupt, the rest queue behind it.
4 Status polling is capped at ≤ 0.2 Hz; AgiBot warns high-frequency RPC can destabilise the robot. The liveness probe is a TCP connect, never an RPC, and completion is estimated rather than polled.

Two more behaviours worth knowing, from reading AimRT's own source (http_rpc_backend.cc):

  • The route is "/rpc" + <func name without prefix> — the pb:/ prefix seen in AimRT function names is stripped. Never put pb:/ in a URL.
  • An RPC-level failure is HTTP 500, not a JSON error. Unknown route is 404. HTTP 200 means the transport worked, not that the robot spoke — is_sucess only confirms the priority check passed.

3. Why this path, and not the others

Path Latency Reliability Complexity Verdict
1. HTTP JSON-RPC PlayTTS (this project) 1 request RTT to enqueue (< 20 ms LAN); time-to-first-audio dominated by the robot's own synthesis Highest — AgiBot's own primary documented path Near zero — one HTTP POST ✅ Chosen
2. Own TTS → WAV → SCP → PlayFile 1–4 s cold; ~1 RTT for pre-cached phrases Good; documented offline fallback Moderate — SSH creds, file lifecycle Fallback if #1 needs internet and you are air-gapped
3. Own TTS → PCM → ROS 2 /audiohal/audio/playback Best possible — tens of ms, true streaming High ceiling, high risk (best-effort QoS, focus arbitration) High — ROS 2 Jazzy on the PC, login-gated aarch64 packages, robot reboot into only_voice Only if you need barge-in streaming
4. LinkSoul cloud agent SDK (WebSocket) Worst — PC → AgiBot cloud → robot n/a Commercial credentials, cloud dependency Wrong tool: a conversational agent layer, not a "say this now" primitive
5. SSH + direct ALSA write — — — Do not ship. No AgiBot document mentions ALSA. hal_audio owns the device behind an explicit focus/priority protocol; bypassing it also bypasses the volume damage limit.

Path 1 wins on every axis the brief asked for: lowest latency, highest reliability, simplest deployment, best-documented A3 compatibility, and it scales because it is stateless HTTP.

Critically, it sends text. The robot synthesises. Nothing is rendered, encoded or transferred by the PC — which is exactly the architecture the "fast speech" requirement asks for.

4. What this project does with it

A3_TRANSPORT=aimdk (the default) selects aimdk_transport.py:

  • one keep-alive httpx client opened at startup and reused for every utterance — no handshake per Speak;
  • a Timeout: header (AimRT honours whole seconds) and a client-side timeout, so a wedged robot cannot hang the dashboard;
  • ack latency measured and reported to the UI;
  • Stop via StopTTSTraceId using the robot-returned trace_id;
  • errors mapped to messages an operator can act on.

Every documented value is a setting, not a literal, because AimRT has no service-discovery endpoint and AgiBot does not guarantee port or name stability across firmware:

A3_AIMDK_SERVICE=aimdk.protocol.TTSService
A3_AIMDK_PLAY_METHOD=PlayTTS
A3_AIMDK_STOP_METHOD=StopTTSTraceId
A3_AIMDK_PRIORITY=INTERACTION_L6
A3_AIMDK_MAX_BYTES=1024

If your firmware renames something, you edit .env. If it uses a wholly different shape, A3_TRANSPORT=http gives you a fully templated HTTP client, and ws / ros2 / ssh are there as further fallbacks.

5. Connecting the robot — the exact steps

5.1 Where the IP goes

One file: .env, in the project root. Three lines:

ROBOT_MODE=real
ROBOT_IP=192.168.1.50     ← the robot's IP on YOUR network
ROBOT_PORT=59301

Do not use 10.42.10.10. That address appears throughout AgiBot's examples, but it is the HDU's address on the robot's internal eth_hdu network. Your PC cannot reach it. Use the HDU's address on your WiFi/LAN.

Then either restart the server, or without restarting:

curl -X POST http://localhost:8000/api/config/reload

5.2 The order to do things in

# 1. reachable at all?
ping 192.168.1.50

# 2. what does it expose? (probes the documented ports and routes, read-only)
python scripts/discover_robot.py 192.168.1.50

# 3. the decisive test - does it actually speak?
python scripts/discover_robot.py 192.168.1.50 --speak "Hello, I am Expedition A3"

# 4. point the app at it, then
python scripts/selftest.py

5.3 What each failure means

Result of step 3 Meaning Next step
The robot speaks Path 1 works. Done. Set .env, restart, demo.
Connection refused / timeout 59301 is not bound on the WLAN interface §6 Step 2 — SSH in and check ss -ltnp
HTTP 404 Port is right, route is wrong Check A3_AIMDK_SERVICE / A3_AIMDK_PLAY_METHOD; confirm firmware version
HTTP 500 Route exists, the RPC failed — the port is reachable Read the response body; check priority_level
is_sucess: false The robot rejected it Read error_message — usually priority or text length
200 + success but silence Accepted but not played Check robot volume, and whether TTS needs internet (§7)

6. Discovery procedure for the live robot

scripts/discover_robot.py automates the PC-side checks. The rest needs an SSH session on the robot.

# From the PC - which of the documented ports answer?
python scripts/discover_robot.py <ROBOT_IP>

# Equivalent by hand. A 404 proves an AimRT HTTP server is listening;
# connection-refused proves it is not.
for p in 59301 56666 51049; do
  curl -s -o /dev/null -w "$p -> %{http_code}\n" -m 3 \
    "http://<ROBOT_IP>:$p/rpc/does.not.Exist/Nope"
done

On the robot (username agi is [UNVERIFIED] — confirm with AgiBot):

# Is the RPC server bound to 0.0.0.0 or only to the internal 10.42.10.10?
ss -ltnp | grep -E '59301|56666|51049'
ip -br a                                    # the HDU's real WLAN address

# The authoritative port map, from firmware config
grep -rn 'listen_port' /agibot/software/v0/ 2>/dev/null | head -50

# ROS environment (resolves ROS_DOMAIN_ID, only needed for the ros2 transport)
source /agibot/software/v0/entry/env/env.sh || source /agibot/software/v0/entry/env.sh
echo "DOMAIN=$ROS_DOMAIN_ID RMW=$RMW_IMPLEMENTATION"

# The real audio surface — this substitutes for the missing discovery API
ros2 topic list   | grep -Ei 'audio|tts|interaction'
ros2 topic echo --once /interaction/tts_status   # while a PlayTTS is running

Do not put any of these in a loop. AgiBot's constraints page caps status RPCs at ≤ 0.2 Hz, forbids deploying on the MDU ("risks the robot falling"), forbids apt install/uninstall, and states violations "may void warranty."

7. Open questions — send these to AgiBot

The five that actually affect this project, in priority order. (Chinese versions follow, since AgiBot support answers faster in Chinese.)

  1. Is port 59301 on the HDU bound to all interfaces and reachable from a PC on the same WiFi/LAN using the HDU's WLAN IP — or only on the internal eth_hdu 10.42.10.x segment? HDU 的 59301 端口是否监听在所有网络接口上？能否从同一局域网的 PC 通过 HDU 的 WLAN IP 访问？还是仅绑定在内部 eth_hdu (10.42.10.x) 网段？

  2. Does TTSService/PlayTTS require internet access? Your English v3.2 page says it "requires network connection". Which engine performs synthesis, and is there a fully on-device/offline TTS mode? What happens with LAN but no internet? PlayTTS 是否必须联网？语音合成由哪家引擎提供？是否有完全离线的端侧 TTS 方案？只有局域网、没有外网时会怎样？

  3. Is there any authentication, token or IP allow-list on the /rpc/ endpoints? If not, what do you recommend on a shared network? (AimRT sets Access-Control-Allow-Origin: * unconditionally, so any page on the LAN can call these.) /rpc/ 接口是否有鉴权、token 或 IP 白名单？如果没有，在共享网络下推荐如何防护？

  4. Can we select voice/timbre, language, speaking rate or emotion per PlayTTS call? Is SetTtsParameters or an equivalent available on the A3, and at which endpoint? Our unit's voice is currently set to "Yunxiao" (teenager, male, multi-language) — is that selectable per request, or only in the console? 能否在每次 PlayTTS 调用时指定音色、语言、语速或情感？A3 是否提供 SetTtsParameters？接口地址是什么？ 我们的机器人当前音色为"Yunxiao"（少年、男声、多语言），该音色能否在每次调用时指定，还是只能在控制台配置？

    Operator-reported, 2026-09-02: the robot's configured voice is "Yunxiao" — labelled teenager | male, multi-language, and marked as the default (默). It was shown in a voice-library UI offering per-voice "Speech Synthesis", which is consistent with AgiBot's LinkSoul timbre library (30–40 timbres + voice cloning) — though that link is inference, not something any published document states. This is not in AgiBot's docs; it comes from the console for this specific unit. It suggests voice selection happens on a platform/console rather than in the PlayTTS body, which is exactly what question 4 needs to settle.

  5. Please confirm the complete priority_level enum accepted by A3 PlayTTS, and the semantics of priority_weight and is_interrupted. Your X2 documentation publishes seven levels — do the same apply to the A3? 请提供 A3 PlayTTS 支持的完整 priority_level 枚举，以及 priority_weight 和 is_interrupted 的语义。X2 文档公布了七个等级，A3 是否相同？

Lower priority, worth asking in the same message: the response envelope (is there a header.code/header.msg wrapper?); whether the AimDK SDK download needs a verified-customer account; whether ports 59301/56666/51049 are stable across firmware; and the PlayMediaFile sample rate — your English v3.2 page says 24 kHz and your Chinese v3.2 page says 16 kHz for the same call.

8. Known contradictions in AgiBot's own documentation

Recorded so nobody wastes an afternoon on them.

# Contradiction Impact here
1 PlayMediaFile sample rate: EN v3.2 says 24 kHz, ZH v3.2 says 16 kHz, same page None — we send text, not files
2 Microphone: v3.1 and v3.2 dev guides say the built-in mic is "not yet configured or enabled"; the User Manual and product page advertise an 8-mic array None — output only
3 env.sh path: /agibot/software/v0/entry/env.sh vs …/entry/env/env.sh Only matters for the ros2 transport
4 ROS 2 distro: AgiBot says the A3 is Jazzy; AimRT's ros2_plugin doc says Humble Only matters for the ros2 transport
5 SDK wheel: the A3 v3.2 guide instructs a2_aimdk-3.0.0…whl in most places and a3_aimdk-3.0.0…whl in one — an A2 artifact inside the A3 doc tree None — Path 1 needs no SDK

ROS_DOMAIN_ID for the A3 is [UNKNOWN] — it is not printed in any A3 document. A2 uses 232 (different model), and one third-party A3-Ultra deployment also uses 232 — suggestive, not proof. Read it from env.sh on the robot.

9. Security note worth raising internally

No authentication scheme is documented for the /rpc/ endpoints, and AimRT sets Access-Control-Allow-Origin: * unconditionally with OPTIONS preflight handled. Taken together: any host — or any web page opened in any browser — on the same LAN as the robot can make it speak.

This project does not widen that exposure. The browser talks only to 127.0.0.1:8000; robot credentials and addresses stay server-side. But on a shared or public network, treat the robot's RPC ports as unauthenticated and segment the network accordingly.

10. Testing the real path before the robot arrives

ROBOT_MODE=mock tests the application. To test this transport — the URL shape, the JSON body, the trace_id round trip, chunking, Stop, and the error paths — run the local stand-in:

python scripts/fake_a3_server.py          # implements the documented contract on :59301

then in .env:

ROBOT_MODE=real
ROBOT_IP=127.0.0.1
ROBOT_PORT=59301

and use the dashboard normally. Every utterance prints in the stand-in's console.

It is a test double written from public documentation, not the robot. Passing against it proves the client is well-formed — not that the firmware behaves identically. Verify on the real unit.

11. Sources

Official AgiBot, A3-specific

  • Audio interfaces (TTS, status, stop, volume, ROS 2 audio) — EN · ZH
  • Interface overview (the two transports, URL shape, remote-call sanction) — EN
  • Quick start (the runnable PlayTTS curl) — EN
  • Resource manager (PlayFile, WAV/PCM rules, resource dirs) — EN
  • Interactive guide (audio_msgs/msg/AudioPlayback, focus services, only_voice) — EN
  • Development constraints (polling limits, MDU, apt, warranty) — ZH
  • Hardware/software overview (HDU/MDU, internal IPs, Ubuntu 24.04) — ZH
  • AimDK docs root — https://open.agibot.com/docs/aimdk (SDK downloads are login-gated)

AimRT framework (transport semantics, read from source)

Different model — do not assume it applies to the A3

Third-party corroboration (an A3 Ultra deployment, not a specification)

  • tzj-hub/agibot_A3_sells_cars — same base URL, methods and payload shape; reads is_sucess with an is_success fallback; its a3_interfaces.py is the only source anywhere for the header.code/header.msg error envelope, which is why this project reads that envelope defensively rather than depending on it.