376 lines
21 KiB
Markdown
376 lines
21 KiB
Markdown
# AGIBOT X2 Dashboard
|
||
|
||
A web control for the AGIBOT X2 humanoid. Open `http://<robot-ip>:8770` from a laptop, phone or tablet on the same network and you get live telemetry, motion control, camera and LiDAR feeds, a posed 3D digital twin, and speech/expression control — with no app to install and no ROS on the client.
|
||
|
||
The browser never speaks ROS 2. A small agent on the robot bridges ROS 2 DDS to plain TCP, and the dashboard serves an ordinary web page over HTTP.
|
||
|
||
---
|
||
|
||
## Contents
|
||
|
||
- [Architecture](#architecture)
|
||
- [Quick start](#quick-start)
|
||
- [The interface — 13 tabs](#the-interface--13-tabs)
|
||
- [Safety model](#safety-model)
|
||
- [Backend modules](#backend-modules)
|
||
- [The on-robot agent](#the-on-robot-agent)
|
||
- [HTTP API](#http-api)
|
||
- [Configuration](#configuration)
|
||
- [Extensions](#extensions)
|
||
- [Deployment](#deployment)
|
||
- [Repository layout](#repository-layout)
|
||
- [Design notes](#design-notes)
|
||
|
||
---
|
||
|
||
## Architecture
|
||
|
||
Two processes, one socket between them:
|
||
|
||
```
|
||
Browser ──HTTP + WebSocket──▶ Dashboard server ──JSON lines over TCP──▶ Agent ──▶ ROS 2 DDS
|
||
(any device) :8770 (FastAPI, Python) :8781 (rclpy, on robot)
|
||
```
|
||
|
||
**The dashboard server never imports `rclpy`.** That single decision drives most of the design:
|
||
|
||
- It runs anywhere — including Windows — because it needs no ROS installation.
|
||
- The web UI stays up when the robot is **off**, so it can show an offline gate that says "power the robot on" instead of dying with a connection error.
|
||
- Reconnection is automatic and continuous. Switch the robot off and the UI shows the gate; switch it on and the dashboard reattaches by itself.
|
||
|
||
Every bridge implements one interface (`bridge_base.py`), so the server and the entire frontend are identical whether a real robot is attached or not.
|
||
|
||
| Bridge mode | Behaviour |
|
||
|---|---|
|
||
| `auto` | Talk to the robot agent if a host is configured, otherwise simulate |
|
||
| `agent` | Require the real robot — never silently fall back to simulation |
|
||
| `mock` | Always simulate, for UI work away from the robot |
|
||
|
||
The mock is deliberately more than a random-number generator: battery drains at a rate that depends on what the robot is doing, odometry integrates the velocity you command, joints ease toward their targets, and mode transitions enforce the same preconditions the real robot does. UI logic exercised against the mock behaves the same way once a real bridge is attached.
|
||
|
||
---
|
||
|
||
## Quick start
|
||
|
||
**On the robot** (needs ROS 2 Humble and the AimDK workspace):
|
||
|
||
```bash
|
||
cd ~/x2_dashboard_agent
|
||
./run_agent.sh # sources ROS 2 + AimDK, starts the agent on :8781
|
||
```
|
||
|
||
**The dashboard** (on the robot, or on any machine that can reach it):
|
||
|
||
```bash
|
||
cd x2_dashboard
|
||
python3 -m backend # binds 0.0.0.0:8770
|
||
```
|
||
|
||
It prints every address it is reachable on, so you can open it from a phone without knowing the host's IP in advance.
|
||
|
||
To run both together with a crash-restart watchdog, use `run_dashboard.sh` — it is idempotent and safe to call every minute (see [Deployment](#deployment)).
|
||
|
||
> **DDS profile required.** Camera and LiDAR topics will connect but deliver *nothing* without the vendor Fast DDS profile. `run_agent.sh` exports `FASTRTPS_DEFAULT_PROFILES_FILE` for this reason — see [Design notes](#design-notes).
|
||
|
||
---
|
||
|
||
## The interface — 13 tabs
|
||
|
||
### Overview
|
||
The at-a-glance page: one hero figure, supporting stat tiles, and trend sparklines. Battery, mode, uptime, attitude and compass.
|
||
|
||
### Control
|
||
Mode switching, input-source arbitration, driving, and preset motions.
|
||
|
||
- **18 motion modes** in 6 groups — safety/basic, joint control, standing, locomotion, posture, external. Each carries a description and a danger flag.
|
||
- **Driving** with `W`/`A`/`S`/`D` or arrow keys; `Space` is an emergency stop that zeroes velocity immediately.
|
||
- **34 preset motions** in 5 groups: greeting (wave, handshake, salute, bow), expressive (heart, thumbs up, hug, cheer, sad), gesture (clap, fist, cross arms), performance (two bass dances, photo poses) and head (point, shake).
|
||
- Velocity is only accepted in driveable modes; joint commands only in `JOINT_DEFAULT` / `JOINT_FREEZE`.
|
||
|
||
### Motion
|
||
Joint-level control and end effectors. Four joint groups — **head, waist, arms, legs** — with per-joint sliders, live position/effort bars, and **4 hand presets**.
|
||
|
||
### Sensors
|
||
IMU (chest and torso), the **8-zone head touch sensor**, LiDAR status, and a topic-liveness table so you can see which topics are actually publishing.
|
||
|
||
### Vision
|
||
**Nine selectable feeds** — six live cameras plus three perception debug views — each switched on by hand:
|
||
|
||
| Feed | Notes |
|
||
|---|---|
|
||
| Head front (imx900c) | 2688×1944 JPEG, ~10 Hz |
|
||
| Head rear | 2064×1552 JPEG, ~10 Hz, mounted upright |
|
||
| Orbbec Gemini 335 colour | Module is mounted **upside down** — rotated 180° by default |
|
||
| Orbbec depth | 16-bit millimetre map, ~14 Hz, tops out ~4.2 m |
|
||
| Stereo pair (left / right) | Both eyes, JPEG ~10 Hz |
|
||
| Perception debug ×3 | Input image, segmentation mask, line colour map |
|
||
|
||
Nothing is subscribed on the robot until you switch a feed on, and switching it off **destroys the subscription on the robot** rather than merely hiding the `<img>`. This is not a nicety: a frame is 170–430 KB and the RGB feeds together publish at ~60 Hz, which pushed roughly **15 MB/s through DDS for pictures nobody was looking at** — over the same Wi-Fi the robot uses to walk.
|
||
|
||
Frames are polled as single images rather than streamed, so a quiet topic degrades into "nothing is publishing" instead of a hung connection.
|
||
|
||
### LiDAR
|
||
Live 3D point cloud from the chest sensor. A scan is ~25 500 points (816 KB) at 2 Hz, decimated on the robot to 4 000 points before crossing the network. Off until switched on, like the cameras.
|
||
|
||
### Model — the digital twin
|
||
The real X2 URDF, posed and shaded by live telemetry. The robot reports all **31 revolute joint positions and efforts at 100 Hz**; this tab draws them across a rigid-body tree of **41 links**. An arm folded in the picture but straight in the room means the encoder, the model, or your idea of which robot you are connected to is wrong.
|
||
|
||
Shaded by **effort, not temperature** — this robot publishes no joint temperature at all. `aimdk_msgs/JointState` carries name, position, velocity, effort and error code and nothing else, and across all 53 `aimdk_msgs` types the only temperature fields belong to the PMU and the battery. Effort is honest and nearly as useful: motor heating is I²R and current tracks torque, so the joint pulling hardest is the joint getting hottest.
|
||
|
||
Load is shown as a **percentage of each joint's rated torque**, taken from the URDF's `<limit effort="...">` and baked into `model.json`. That matters — rated torque here runs from 0.6 Nm to 120 Nm, so an absolute Nm ramp would paint every wrist permanently cold and every leg permanently hot regardless of what the robot was doing.
|
||
|
||
Geometry is baked offline: **112 MB of vendor STL down to 1.4 MB**. Nothing parses a URDF at runtime.
|
||
|
||
### Interaction
|
||
Speech (text-to-speech with priority and interrupt), volume and mute, screen expressions/emoji, and the LED strip. Also hosts the voice-session switch that starts and stops the conversation loop.
|
||
|
||
### Power
|
||
Battery detail — percentage, voltage, current, temperature, cycle count, capacity in mAh, output power — plus **8 power rails** (48 V bus, 48 V output, 12 V output, head power, Orin NX, RK3588, fan rail, 48 V PMOS), fan RPM and thermals, with history charts.
|
||
|
||
### Navigation
|
||
Odometry, heading and travelled path. **No mapping controls are offered on purpose**: this firmware does not expose the documented SLAM interface (`/integrated_command`, `/relocalization_pose`, `GetStoredMapByName` are all absent from the live graph), and a button that silently does nothing is worse than no button.
|
||
|
||
### Extensions
|
||
Renders whatever plugins the backend found. Nothing in this tab knows about any specific plugin — see [Extensions](#extensions).
|
||
|
||
### Console
|
||
Event log, raw ROS access (publish to any topic, call any service) and the discovered ROS graph.
|
||
|
||
### Settings
|
||
Robot discovery, network, transport and safety limits. This is where the "works on any IP" promise lives: the dashboard never stores a fixed robot address — it discovers one on whatever network you are currently attached to, and reports every address it is itself reachable on.
|
||
|
||
Includes a **QR code** of the numeric dashboard URL. Typing an IP into a phone is exactly the friction that makes people want a hostname; a QR sidesteps the naming question entirely, since it needs no mDNS, no multicast and no DNS — it works even on Wi-Fi that isolates clients from each other.
|
||
|
||
---
|
||
|
||
## Safety model
|
||
|
||
| Guard | Detail |
|
||
|---|---|
|
||
| **Deadman** | Velocity commands expire after `locomotion_deadman_s` (default 0.5 s). The UI republishes at 20 Hz while a key is held; release it and the robot stops. |
|
||
| **Zero-torque confirm** | Entering `PASSIVE_DEFAULT` requires an explicit confirmation dialog (`require_confirm_zero_torque`). These modes collapse a free-standing robot. |
|
||
| **Danger flags** | Passive, zero-torque, soft e-stop, run and both stair gaits are flagged in the spec, and the UI renders them as destructive actions. |
|
||
| **Mode gating** | Velocity is rejected outside driveable modes; joint commands are rejected outside joint-control modes. |
|
||
| **Velocity clamps** | Forward −0.6…0.8 m/s, lateral ±0.7 m/s, angular ±0.8 rad/s — configurable downward in Settings. |
|
||
| **Space bar** | Global emergency stop, zeroes velocity immediately. |
|
||
|
||
---
|
||
|
||
## Backend modules
|
||
|
||
| Module | Lines | Purpose |
|
||
|---|---:|---|
|
||
| `server.py` | 1062 | FastAPI app — all HTTP routes and the WebSocket |
|
||
| `x2_spec.py` | 773 | The interface specification (see below) |
|
||
| `bridge_agent.py` | 610 | TCP bridge to the on-robot agent, with auto-reconnect |
|
||
| `nic.py` | 592 | Picks the LAN address this machine is *actually* reachable on |
|
||
| `bridge_mock.py` | 520 | Physics-lite simulated X2 |
|
||
| `announce.py` | 348 | mDNS — advertises the dashboard as `<name>.local` |
|
||
| `netinfo.py` | 346 | Address enumeration, subnet derivation, robot discovery sweep |
|
||
| `voice_session.py` | 306 | On/off switch for the conversation loop |
|
||
| `bridge_base.py` | 241 | The contract every bridge implements |
|
||
| `plugin_api.py` | 229 | The extension API |
|
||
| `settings.py` | 227 | Runtime config, persisted to `config.json` |
|
||
| `registry.py` | 203 | Plugin discovery, loading and hot reload |
|
||
| `recovery.py` | 118 | Find the robot and start its agent over SSH |
|
||
| `hub.py` | 107 | Fan-out from one bridge to N browsers |
|
||
| `__main__.py` | 98 | Entry point — `python -m backend` |
|
||
|
||
### `x2_spec.py` — the ground truth
|
||
|
||
Every constant in this file was **read back from the robot itself** by introspecting the installed `aimdk_msgs` package and the live ROS 2 graph — not copied from the published documentation. Where the two disagree, the robot wins. Differences found:
|
||
|
||
- `McAction` has **18 modes, not 5**. `PASSIVE_DEFAULT` (1) and `ZERO_TORQUE_DEFAULT` (4) are distinct; the docs conflated them.
|
||
- `McControlArea` is a **bitmask** (`LEFT_HAND=1`, `RIGHT_HAND=2`, `HEAD=4`, `WAIST=8`) — which is why the docs' "area 3" (both arms) and "area 11" (whole body) work out: 3 = 1|2, 11 = 1|2|8.
|
||
- Several documented preset motion IDs (1007, 1010, 1011, 3017, 3024, 3025, 3031) **do not exist** in the firmware enum.
|
||
- `PmuState` spells its fan field `fan_pecentage` (sic), and `battery_remaining_capacity` is **mAh, not percent** — the percentage lives in `battery_remaining_capacity_percentage`.
|
||
|
||
### `hub.py` — why each browser gets its own queue
|
||
|
||
One bridge produces state; N browsers consume it. Each client has a bounded queue, so a slow tab (a backgrounded phone, say) can never stall the telemetry loop for everyone else — it just drops frames.
|
||
|
||
### `nic.py` — why address selection is not a connectivity test
|
||
|
||
The dashboard publishes `http://<ip>:8770`, so picking the wrong address hands someone a link that cannot work. This is genuinely hard: a *disconnected* Ethernet adapter still holding an address will **answer a local HTTP fetch successfully**, because Windows routes traffic to any of its own addresses through loopback, skipping the adapter entirely — and it answers *faster* than the real one. No connect, bind or fetch test can distinguish a reachable address from a dead one; the decision has to come from adapter metadata. The rule used here was chosen by running candidates against 23 synthetic adapter tables (VPNs of three shapes, Hyper-V bridges, mobile hotspot, docked Ethernet, duplicate-IP detection, campus public addressing, renamed adapters).
|
||
|
||
### `recovery.py` — why recovery lives on the dashboard host
|
||
|
||
The X2 cannot reliably bring its own agent up after a power cycle:
|
||
|
||
- `systemd --user` units only run while the user has a login session, and the `agi` account cannot enable lingering (`loginctl enable-linger` is denied, and sudo forbids running as root).
|
||
- The cron fallback does not fire either — the robot's clock jumps backwards by several hours shortly after boot (RTC vs NTP), and cron stalls on a backward jump.
|
||
|
||
So the dashboard finds the robot wherever DHCP put it and starts the agent over SSH.
|
||
|
||
---
|
||
|
||
## The on-robot agent
|
||
|
||
`x2_dashboard_agent/x2_agent.py` — ~1 480 lines, runs on PC2 (Jetson Orin). Its only dependency beyond `rclpy` is the standard library.
|
||
|
||
**Protocol:** newline-delimited JSON.
|
||
|
||
```
|
||
-> {"type":"hello","data":{...}} (agent, on connect)
|
||
-> {"type":"state","data":{...}} (agent, ~10 Hz)
|
||
<- {"type":"cmd","id":7,"name":"set_mode","args":{}} (client)
|
||
-> {"type":"result","id":7,"ok":true,"message":"..."} (agent)
|
||
```
|
||
|
||
**Commands:** `set_mode`, `get_mode`, `set_velocity`, `stop`, `preset`, `register_source`, `set_joints`, `set_hand`, `speak`, `set_volume`, `set_mute`, `emoji`, `led`, `camera_frame`, `graph`, `publish_raw`, `ping`, `stream_set`, `stream_list`, `lidar_points`.
|
||
|
||
**Flags:** `--host` (default `0.0.0.0`), `--port` (default `8781`), `--hz` (state broadcast rate, default 10).
|
||
|
||
Cameras and the LiDAR are **on-demand**: no subscription exists until `stream_set` turns one on.
|
||
|
||
---
|
||
|
||
## HTTP API
|
||
|
||
All under `/api`. The WebSocket at `/ws` carries the live telemetry stream.
|
||
|
||
| Area | Endpoints |
|
||
|---|---|
|
||
| **State** | `GET /bootstrap` · `/state` · `/series` · `/series/keys` · `/events` · `/topics` · `/health` |
|
||
| **Settings** | `GET`/`POST /settings` |
|
||
| **Network** | `GET /network` · `POST /network/probe` · `/network/scan` · `/bridge/restart` |
|
||
| **Motion** | `GET`/`POST /mode` · `POST /velocity` · `/stop` · `/preset` · `/joints` · `/hand` |
|
||
| **Interaction** | `POST /speak` · `/volume` · `/mute` · `/emoji` · `/led` · `GET`/`POST /voice/session` |
|
||
| **Arbitration** | `POST /input-source` |
|
||
| **Raw ROS** | `POST /raw/publish` · `/raw/service` |
|
||
| **Streams** | `GET /streams` · `POST /streams/{key}` · `GET /lidar/points` · `/camera/{key}/frame` |
|
||
| **Recovery** | `POST /robot/find` · `/robot/wake` · `GET /robot/status` · `GET /qr` |
|
||
| **Plugins** | `GET /plugins` · `POST /plugins/reload` · `POST /plugins/{id}/{control}` |
|
||
|
||
Interactive API docs are served at `/api/docs`.
|
||
|
||
---
|
||
|
||
## Configuration
|
||
|
||
`config.json` sits beside the project root. Every field is editable from the Settings tab while the server is running.
|
||
|
||
| Key | Default | Meaning |
|
||
|---|---|---|
|
||
| `robot_host` | `127.0.0.1` | Agent address |
|
||
| `agent_port` | `8781` | Agent TCP port |
|
||
| `robot_label` | `AGIBOT X2` | Display name |
|
||
| `dashboard_name` | `agibot` | mDNS name → `agibot.local` |
|
||
| `advertise_name` | `true` | Publish that name over mDNS |
|
||
| `auto_discover` | `true` | Sweep the subnet when the saved address stops answering |
|
||
| `auto_start_agent` | `false` | SSH in and start the agent if it is not running |
|
||
| `robot_ssh_user` / `_password` / `_port` | `agi` / *(empty)* / `22` | SSH credentials for recovery |
|
||
| `agent_start_command` | `ensure_agent.sh` | What to run over SSH |
|
||
| `ros_domain_id` | `0` | Shown for reference; used by the agent |
|
||
| `rmw_implementation` | `rmw_fastrtps_cpp` | Shown for reference |
|
||
| `bridge_mode` | `agent` | `auto` · `agent` · `mock` |
|
||
| `host` / `port` | `0.0.0.0` / `8770` | HTTP bind |
|
||
| `telemetry_hz` | `10` | State broadcast rate |
|
||
| `history_seconds` | `120` | Chart history window |
|
||
| `require_confirm_zero_torque` | `true` | Confirm before passive/zero-torque |
|
||
| `locomotion_deadman_s` | `0.5` | Velocity command expiry |
|
||
| `max_forward_velocity` | `0.8` | m/s |
|
||
| `max_lateral_velocity` | `0.7` | m/s |
|
||
| `max_angular_velocity` | `0.8` | rad/s |
|
||
| `theme` / `accent` | `dark` / `blue` | UI |
|
||
|
||
`host` is the only key the browser may not change.
|
||
|
||
> **Note:** `robot_ssh_password` ships empty. Fill it in through the Settings tab at runtime rather than committing a value.
|
||
|
||
---
|
||
|
||
## Extensions
|
||
|
||
Drop a `.py` file into `backend/plugins/` that builds a `Plugin`, and the dashboard grows a new panel for it — no frontend work, no server edits. Each control you declare is rendered by the browser from the manifest, and clicking it calls your handler.
|
||
|
||
```python
|
||
from backend.plugin_api import Plugin
|
||
|
||
plugin = Plugin(id="hello", name="Hello", icon="👋")
|
||
|
||
@plugin.action("wave", label="Wave hello")
|
||
async def wave(ctx):
|
||
await ctx.bridge.play_preset(motion=1002, area=2)
|
||
return "Waved"
|
||
```
|
||
|
||
Press **Reload** in the Extensions tab to pick up edits without restarting the server. A plugin that fails to import does not take the dashboard down — the error is recorded and shown in the UI next to the plugin that caused it.
|
||
|
||
Two worked examples ship in `backend/plugins/`:
|
||
|
||
- **`example_greeter.py`** — sequencing several subsystems: one button drives mode, motion, speech, screen and lights together.
|
||
- **`example_battery_guard.py`** — a background monitor: watches the PMU battery level, warns once per threshold crossing, and can drop the robot into a safe mode before it browns out mid-stride. Demonstrates `on_tick`, `ctx.push` readouts and charted series.
|
||
|
||
`_template.py` is a blank starting point.
|
||
|
||
---
|
||
|
||
## Deployment
|
||
|
||
Three systemd **user** units, in `systemd_user/`:
|
||
|
||
| Unit | Role |
|
||
|---|---|
|
||
| `x2-dashboard.service` | `Type=oneshot` launcher — runs `run_dashboard.sh`, then returns |
|
||
| `x2-dashboard.timer` | Watchdog — fires 30 s after boot, then every wall-clock minute |
|
||
| `x2-dashboard-agent.service` | The ROS bridge alone |
|
||
|
||
```bash
|
||
cp systemd_user/*.service systemd_user/*.timer ~/.config/systemd/user/
|
||
systemctl --user daemon-reload
|
||
systemctl --user enable --now x2-dashboard.timer
|
||
```
|
||
|
||
Two non-obvious details are load-bearing:
|
||
|
||
- **`Type=oneshot` *without* `RemainAfterExit`.** The unit must return to `inactive` after each run, or the timer never re-triggers.
|
||
- **`OnCalendar=*:*:00`, not `OnUnitActiveSec`.** The launcher `setsid`s its children into their own sessions so they outlive the one-shot; an activity-relative timer would compute `NextElapse=infinity`. `KillMode=none` stops systemd reaping those children when the unit deactivates.
|
||
|
||
`run_dashboard.sh` is idempotent by design — it starts only what is not already running, so it is safe from the timer, from `@reboot`, and by hand. It guards on `/proc` rather than trusting `pgrep -f` alone, because `pgrep -f` also matches any shell whose command line merely mentions the pattern; that bug launched a second backend on top of the first during the ~10 s before the port binds. It also trims its own logs at 2 MB.
|
||
|
||
---
|
||
|
||
## Repository layout
|
||
|
||
```
|
||
X2_dashboard/
|
||
├── x2_dashboard/ the dashboard server + web UI
|
||
│ ├── backend/
|
||
│ │ ├── server.py FastAPI app, all routes, WebSocket
|
||
│ │ ├── x2_spec.py interface spec, read off the real robot
|
||
│ │ ├── bridge_base.py the bridge contract
|
||
│ │ ├── bridge_agent.py TCP bridge to the on-robot agent
|
||
│ │ ├── bridge_mock.py simulated X2
|
||
│ │ ├── hub.py fan-out to N browsers
|
||
│ │ ├── registry.py plugin discovery + hot reload
|
||
│ │ ├── plugin_api.py the extension API
|
||
│ │ ├── settings.py config load/save
|
||
│ │ ├── announce.py mDNS
|
||
│ │ ├── netinfo.py discovery + addressing
|
||
│ │ ├── nic.py reachable-address selection
|
||
│ │ ├── recovery.py find + wake the robot over SSH
|
||
│ │ ├── voice_session.py conversation-loop control
|
||
│ │ └── plugins/ drop-in extensions
|
||
│ ├── web/
|
||
│ │ ├── index.html
|
||
│ │ ├── css/app.css design tokens, layout, components
|
||
│ │ ├── js/
|
||
│ │ │ ├── core.js store, transport, notifications
|
||
│ │ │ ├── main.js shell, rail, routing, shortcuts
|
||
│ │ │ ├── ui.js DOM builders
|
||
│ │ │ ├── charts.js SVG charts
|
||
│ │ │ ├── model3d.js WebGL URDF renderer
|
||
│ │ │ └── tabs/ the 13 pages
|
||
│ │ └── model/ quantised URDF geometry (1.4 MB)
|
||
│ └── config.json
|
||
├── x2_dashboard_agent/ runs ON the robot
|
||
│ ├── x2_agent.py ROS 2 ⇄ TCP bridge
|
||
│ ├── run_agent.sh sources ROS 2 + AimDK + DDS profile
|
||
│ ├── run_dashboard.sh idempotent launcher for both
|
||
│ ├── ensure_agent.sh called over SSH by recovery
|
||
│ └── x2-dashboard-agent.service
|
||
└── systemd_user/ unit files
|
||
```
|
||
|