kassam 0bb7d3ed83 Update 2026-07-10 12:16:04

2026-07-10 12:16:05 +04:00

16 KiB

Raw Blame History

Sanad Fleet Agents

On-robot agents that report each robot's state to the YS Lootah fleet server. Every robot type ships as its own self-contained Docker image with a user-level systemd auto-start service, deployed, managed, and removed over SSH by a single script. No docker-compose on the robot, no sudo required.

For the end-to-end data flow (topics → agent → HTTP → server), see PIPELINE.md.

What this is
Directory layout
The three agents
Prerequisites
Quick start
Installer reference
The auto-start service (systemd)
Configuration reference
Per-robot specifics
The fleet test server
How a deploy works internally
Fleet inventory
Troubleshooting
Security notes

1. What this is

The fleet server (built by YS Lootah — "their side") needs each robot to push its status outbound over HTTPS. This repo is the robot side: a small agent per robot type that reads local state and POSTs it to documented endpoints, plus the tooling to install/manage it across the fleet.

G1 → uploads the navigation map.
R1 / Go2 → stream telemetry (battery, charging, status, position, faults).

Each is a drop-in Docker image; install it on any new robot of that type and it just works.

2. Directory layout

Project/fleet/
├── README.md                 ← this file
├── PIPELINE.md               ← end-to-end data-flow reference
├── fleet_install.sh          ← deploy / manage / remove over SSH (interactive + scriptable)
├── fleet_test_server.py      ← workstation stand-in for the fleet server (for tests)
└── agents/
    ├── g1/                   ← MAP uploader
    │   ├── sanad_api_g1.py
    │   ├── Dockerfile        (lean: python + requests)
    │   ├── requirements.txt
    │   ├── docker-compose.yml (manual local use only)
    │   └── .env.example
    ├── g1t/                  ← TELEMETRY (G1, unitree_hg)
    │   ├── sanad_api_g1t.py
    │   ├── Dockerfile        (DDS) + vendor/
    │   └── .env.example
    ├── r1/                   ← TELEMETRY (unitree_hg)
    │   ├── sanad_api_r1.py
    │   ├── Dockerfile        (DDS: CycloneDDS + unitree_sdk2py)
    │   ├── vendor/           (unitree_sdk2py wheel + crc libs)
    │   └── .env.example
    └── go2/                  ← TELEMETRY (unitree_go)  [unverified on hardware]
        ├── sanad_api_go2.py
        ├── Dockerfile
        ├── vendor/
        └── .env.example

agents/ is the single source of truth. The installer rsyncs the selected agent to the robot; you never edit files on the robot.

3. The three agents

agent (type)	robot	endpoint	what it sends	cadence
`sanad_api_g1` (`g1`)	Unitree G1	`POST /api/v1/fleet/ingest/{sn}/map`	RTAB-Map `.db` + places (points)	on change (~30 s scan)
`sanad_api_g1t` (`g1t`)	Unitree G1	`POST /api/v1/fleet/ingest/telemetry`	battery, charging, status, position, faults, mac	every ~2 s
`sanad_api_r1` (`r1`)	Unitree R1 EDU	`POST /api/v1/fleet/ingest/telemetry`	same telemetry	every ~2 s
`sanad_api_go2` (`go2`)	Unitree Go2	`POST /api/v1/fleet/ingest/telemetry`	same telemetry	every ~2 s

The G1 runs both g1 (map) and g1t (telemetry) — two independent containers/services, both keyed by the same sn (same physical robot). g1t uses the same unitree_hg DDS as R1, and reads position from the firmware odom topic (rt/lf/odommodestate) over DDS.

All requests carry Authorization: Bearer <device_token>. Full payload schemas and the data pipeline are in PIPELINE.md.

Design principles

No ROS in the agents. G1 reads map files directly; R1/Go2 read DDS via unitree_sdk2py. This keeps images small and portable.
Read-only. Telemetry agents never command motion (only passive DDS reads +, optionally, the read-only GET_FSM_ID RPC).
Never crash the loop. Every tick is wrapped; transient errors are logged and retried. Telemetry sends a heartbeat when it can't read state so the robot stays "online".
Change-detected uploads. The map is only re-sent when its content hash changes.

4. Prerequisites

Workstation (the machine you deploy from):

bash, ssh, rsync, python3.
SSH key access to each robot (ssh unitree@<ip> must work without a password — the installer uses BatchMode=yes).
On the same network as the robots (they POST back to the workstation during test).

Robot:

Docker (Engine ≥ 20). The unitree user must be in the docker group.
Internet at build time (pulls the base image; R1/Go2 also apt/pip the DDS stack).
systemd with a user bus (standard on Ubuntu 20.04+). No sudo needed.
Architecture: arm64 (Jetson / backpack). Images build natively on the robot.

5. Quick start

cd Project/fleet

# Interactive — asks robot type, IP, and (if new) robot name + server:
./fleet_install.sh

# …or scripted:
./fleet_install.sh install r1 10.255.254.82 --sn r1_82 \
    --server-ip 10.255.254.83 --port 8799 --token <device-token>

# See what it's sending, tail logs, check the service:
./fleet_install.sh data   r1 10.255.254.82
./fleet_install.sh logs   r1 10.255.254.82
./fleet_install.sh status r1 10.255.254.82

# End-to-end test against your workstation acting as the server:
./fleet_install.sh test   r1 10.255.254.82

# Remove everything:
./fleet_install.sh uninstall r1 10.255.254.82

6. Installer reference

Interactive mode (no arguments)

./fleet_install.sh

Flow:

Which robot? 1) g1 2) r1 3) go2
Robot IP and SSH user (default unitree; it verifies SSH works).
It detects whether the agent is already installed (systemd unit file or container present) and branches:
- Installed → menu: 1) show data 2) status 3) logs 4) reinstall 5) UNINSTALL 6) quit.
- Not installed → prompts Robot name (SN), Fleet server IP (auto-detected default), port, device token, then installs.

Scriptable commands

./fleet_install.sh <command> <g1|r1|go2> <ip> [options]

command	action
`install`	rsync agent → robot, build image, create container, install + enable systemd service
`uninstall`	disable/remove service, remove container, image, and `~/sanad_api_<type>`
`status`	systemd service state + container state
`data`	recent telemetry/map log lines (what it's currently sending)
`logs`	`docker logs -f` (live tail)
`test`	start the workstation server, push a real post from the robot, verify receipt (PASS/FAIL)

Options

option	default	meaning
`--sn NAME`	`<type>_<last-octet>`	robot's fleet id (the `sn` field)
`--server-ip IP`	auto (route toward robot)	fleet server the robot posts to
`--port N`	`8799`	fleet server port
`--token TOK`	`test-token`	device bearer token
`--user USER`	`unitree`	SSH user on the robot
`--keep-server`	off	(test) leave the workstation test server running

7. The auto-start service (systemd)

Because the robots have no passwordless sudo, the agent runs as a user-level systemd service (no root needed):

Unit file: ~/.config/systemd/user/sanad-api-<type>.service
Boot auto-start: enabled via loginctl enable-linger <user> (allowed without sudo) so the user manager starts at boot before login.

The service owns the container lifecycle:

[Service]
Type=simple
Restart=always
RestartSec=5
ExecStart=/usr/bin/docker start -a sanad-api-<type>
ExecStop=/usr/bin/docker stop -t 10 sanad-api-<type>

The container is created with docker create (no docker restart policy) so systemd is the single owner — Restart=always also covers the boot race where the user manager starts before dockerd is ready (it retries every 5 s).

Managing it on the robot:

systemctl --user status  sanad-api-r1
systemctl --user restart sanad-api-r1
systemctl --user stop    sanad-api-r1
journalctl --user -u sanad-api-r1 -f      # or: docker logs -f sanad-api-r1

8. Configuration reference

Config is env-only. The installer writes ~/sanad_api_<type>/.env on the robot; .env.example in each agent dir documents every key.

Common (all agents)

var	required	default	meaning
`SERVER_URL`	✅	—	fleet server base URL
`DEVICE_TOKEN`	✅	—	bearer token (per robot)
`SN`	—	`<type>_0000`	robot fleet id
`VERIFY_TLS`	—	`1`	verify server TLS cert (`0` for self-signed dev)
`HTTP_TIMEOUT`	—	10–30	per-request timeout (s)
`POLL_INTERVAL`	—	2 (tel) / 30 (map)	loop cadence (s)

G1 map uploader

var	default	meaning
`ROBOT`	`sanad`	web_nav3 robot name → maps subdir + `X-Robot-Name`
`MAPS_DIR`	`/data/maps`	mounted web_nav3 `maps/` (`<robot>/*.db`)
`DATA_DIR`	—	mounted web_nav3 `web/data/` (per-map places)
`LEGACY_PLACES`	—	optional legacy `places.json`
`WEB_NAV3_URL`	—	optional `http://127.0.0.1:8765` (to learn the active map)
`MAP_SELECT`	`all`	`all` · `active` · `newest`
`MAP_UPLOAD_MODE`	`multipart`	`multipart` · `base64json`
`MAP_ENDPOINT`	`/api/v1/fleet/ingest/{sn}/map`	path template
`STATE_DIR`	`/data/state`	upload-fingerprint state

R1 / Go2 telemetry

var	default	meaning
`DDS_INTERFACE`	R1 `eth10` / Go2 `eth0`	NIC that sees robot DDS
`DDS_DOMAIN`	`0`	DDS domain id
`MAC_INTERFACE`	= `DDS_INTERFACE`	NIC whose MAC is reported
`R1_READ_FSM`	`0`	(R1) read loco FSM for status — read-only GET RPC
`R1_POSITION_SOURCE` / `GO2_POSITION_SOURCE`	`none`	`none` · `rosbridge` (· `sportmode` Go2)
`ROSBRIDGE_URL`	`ws://127.0.0.1:9090`	position source when rosbridge
`LOW_SOC`	`15`	%→ `LOW_BATTERY` fault
`MOTOR_TEMP_MAX`	`85`	°C→ `MOTOR_OVERTEMP` fault
`TELEMETRY_ENDPOINT`	`/api/v1/fleet/ingest/telemetry`	path

9. Per-robot specifics

G1 (map) — real maps live inside the nav container

The G1's saved maps are stored inside the p4_Foxy_sanad (Package_4 nav) container at /home/unitree/marcus_nav2_test/maps, which Package_4 does not bind-mount to the host. So the uploader (which mounts a host dir) sees 0 maps until you do one of:

Add a host bind-mount for maps/ (and web/data/) to the Package_4 nav service, then set the uploader's MAPS_HOST_DIR/DATA_HOST_DIR to those paths.
Put both containers on a shared named volume for the maps dir.

The test command seeds a throwaway fixture map so the upload path is verified regardless.

R1 (telemetry)

DDS link is eth10 (= 192.168.123.164); wlan0 is the fleet LAN.
Battery: rt/lf/bmsstate (BmsState_), soc 0–100, charging from current sign. Coexists fine with the running sanadr1 app (DDS allows many readers).

Go2 (telemetry) — ⚠️ unverified on hardware

Uses unitree_go DDS. Battery is nested in rt/lowstate.bms_state (Go2 has no separate BMS topic).
Written from the SDK layout; image builds and unitree_go imports, simulate is correct, but not yet run on a real Go2 — confirm the bms_state current sign (charging polarity) and sportmodestate fields on the robot.

10. The fleet test server

fleet_test_server.py stands in for the real fleet server so you can verify a deploy end-to-end from your workstation.

PORT=8799 REQLOG=/tmp/fleet.jsonl python3 fleet_test_server.py

Binds 0.0.0.0:8799.
GET /ping → 200 (reachability check the installer uses).
Accepts POST …/map and POST …/telemetry; logs each request to REQLOG and prints a live summary.

./fleet_install.sh test … starts it automatically, triggers a post from the robot, asserts the payload arrived, and prints PASS/FAIL.

11. How a deploy works internally

install:
  rsync agents/<type>/  →  unitree@<ip>:~/sanad_api_<type>/   (--delete, minus .env/state)
  write ~/sanad_api_<type>/.env                               (SERVER_URL, TOKEN, SN, iface…)
  ssh: docker build -t sanad-api-<type>:latest .              (native arm64)
  ssh: docker create --name sanad-api-<type> <run-args> …     (no docker restart policy)
  write ~/.config/systemd/user/sanad-api-<type>.service
  ssh: loginctl enable-linger ; systemctl --user enable --now …

run-args:  --network host --env-file …/.env
           (g1 also: -v maps:ro -v web_data:ro -v state)

--network host is required: G1 to reach 127.0.0.1:8765 (web_nav3) if used, and R1/Go2 so the robot's DDS traffic is visible (DDS multicast doesn't cross a NAT bridge).

12. Fleet inventory

robot	agent(s)	IP	SSH	arch	DDS iface	SN
G1	map (`g1`) + telemetry (`g1t`)	`10.255.254.58`	`unitree` (key)	arm64	`eth0`	`g1_7892`
R1	telemetry (`r1`)	`10.255.254.82`	`unitree` (key)	arm64	`eth10`	`r1_82`
Go2	telemetry (`go2`)	(TBD)	`unitree`	arm64	`eth0`	—

Workstation (deploy host + test fleet server): 10.255.254.83 (wlp4s0).

13. Troubleshooting

symptom	cause / fix
`robot cannot reach the workstation server (http 000)` during `test`	robot→workstation blocked. Check both are on the same subnet; open the port if a firewall is on (`sudo ufw allow 8799`).
G1 uploads nothing / "0 maps"	no saved map, or maps aren't mounted — see §9 G1.
R1/Go2 `telemetry POST failed … Connection refused`	the target server (e.g. the test server) is down. Point at the real server: `install … --server-ip <fleet> --token <real>`.
R1/Go2 `battery=null, status=offline` forever	DDS not seen. Wrong `DDS_INTERFACE` (R1 = `eth10`) or robot firmware down. Confirm with `ip -o addr` on the robot.
build is very slow (R1/Go2)	first build compiles CycloneDDS (minutes). Run detached and poll: `ssh … 'setsid bash -c "cd ~/sanad_api_r1 && docker build -t sanad-api-r1:latest . > build.log 2>&1" </dev/null &'` then watch `build.log`. Docker layer cache survives reboots.
service didn't come up after reboot	linger must be on: `loginctl show-user
`unbound variable` / weird prompt behavior	ensure you're on the current `fleet_install.sh` (uses `ssh -n` for command calls so SSH doesn't eat stdin).

14. Security notes

Outbound only. Agents open no inbound ports on the robot; they POST out over HTTPS with a per-robot bearer token.
Least privilege. Runs as the unitree user (docker group), user-level systemd, no root. Read-only w.r.t. the robot (no motion commands).
Token handling. .env holds the device token and is never rsynced back or committed (.gitignored). The installer writes it directly to the robot.
TLS. VERIFY_TLS=1 in production; 0 only for the local test server.

16 KiB Raw Blame History Unescape Escape