kassam/fleet
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
fleet/README.md

17 KiB
Raw Blame History

Sanad Fleet Agents

On-robot agents that report each robot's state to the YS Lootah fleet server. Every robot type ships as its own self-contained Docker image with a user-level systemd auto-start service, deployed, managed, and removed over SSH by a single script. No docker-compose on the robot, no sudo required.

Sanad Fleet — architecture and data pipeline

For the end-to-end data flow (topics → agent → HTTP → server), see PIPELINE.md.

Table of contents

  1. What this is
  2. Directory layout
  3. The three agents
  4. Prerequisites
  5. Quick start
  6. Installer reference
  7. The auto-start service (systemd)
  8. Configuration reference
  9. Per-robot specifics
  10. The fleet test server
  11. How a deploy works internally
  12. Fleet inventory
  13. Troubleshooting
  14. Security notes

1. What this is

The fleet server (built by YS Lootah — "their side") needs each robot to push its status outbound over HTTPS. This repo is the robot side: a small agent per robot type that reads local state and POSTs it to documented endpoints, plus the tooling to install/manage it across the fleet.

  • G1 → uploads the navigation map.
  • R1 / Go2 → stream telemetry (battery, charging, status, position, faults).

Each is a drop-in Docker image; install it on any new robot of that type and it just works.

2. Directory layout

Project/fleet/
├── README.md                 ← this file
├── PIPELINE.md               ← end-to-end data-flow reference
├── fleet_install.sh          ← deploy / manage / remove over SSH (interactive + scriptable)
├── fleet_test_server.py      ← workstation stand-in for the fleet server (for tests)
└── agents/
    ├── g1/                   ← MAP uploader
    │   ├── sanad_api_g1.py
    │   ├── Dockerfile        (lean: python + requests)
    │   ├── requirements.txt
    │   ├── docker-compose.yml (manual local use only)
    │   └── .env.example
    ├── r1/                   ← TELEMETRY (unitree_hg)
    │   ├── sanad_api_r1.py
    │   ├── Dockerfile        (DDS: CycloneDDS + unitree_sdk2py)
    │   ├── vendor/           (unitree_sdk2py wheel + crc libs)
    │   └── .env.example
    └── go2/                  ← TELEMETRY (unitree_go)  [unverified on hardware]
        ├── sanad_api_go2.py
        ├── Dockerfile
        ├── vendor/
        └── .env.example

agents/ is the single source of truth. The installer rsyncs the selected agent to the robot; you never edit files on the robot.

3. The three agents

agent (type) robot DDS family notes
sanad_api_g1 (g1) Unitree G1 unitree_hg canonical source (see below)
sanad_api_r1 (r1) Unitree R1 EDU unitree_hg generated from g1; R1 FSM ids, eth10
sanad_api_go2 (go2) Unitree Go2 unitree_go generated from g1; battery nested in LowState.bms_state — ⚠ unverified on hardware

One agent = one full-feature service. Each robot runs a single container that does everything, across five endpoints:

  • telemetry (…/telemetry, ~2 s) — 27 fields: identity, software, firmware, battery(+detail), motor temps, storage, status, position, control (loco mode), faults, and status mirrors for map/logs/alerts/remote + full timing.
  • map (…/{sn}/map, on change) — the Sanad/SLAM saved map, uploaded ONCE per content. Formats: RTAB-Map .db (skipped over the ~8 MB server cap) and slam_toolbox pgm/yaml → image JSON (PNG + resolution + origin). Status shown in telemetry map.
  • alerts (…/{sn}/alert) — each NEW fault, immediately (strings — the ingest 500s on fault objects).
  • logs (…/{sn}/logs, 60 s) — the agent's own lines + the robot's Sanad app logs ([sanadr1-logs], auto-discovered, backfilled, noise-filtered).
  • remote (…/{sn}/remote, 60 s) — registers the Sanad dashboard URL (web) and ssh unitree@<ip> (ssh) — no changes to the Sanad app.

All three agents are generated from g1 by tools/gen_agents.py — run it after any change to agents/g1/ to keep r1/go2 in lockstep (no manual editing of r1/go2).

All requests carry Authorization: Bearer <device_token>. Full payload schema and pipeline in PIPELINE.md.

Design principles

  • No ROS. Maps read from files; state read from DDS via unitree_sdk2py.
  • Read-only toward the robot. Never commands motion — the control panel is status-only; mode switching is deliberately not built (it can drop the robot).
  • Never crash the loop. Every tick is wrapped; a heartbeat keeps the robot "online" when state is unreadable.
  • Change-detected + one-time uploads. Maps re-send only on content change.
  • Resilient shipping. Failed log ships are requeued (last ~400 lines) until the server accepts them.

4. Prerequisites

Workstation (the machine you deploy from):

  • bash, ssh, rsync, python3.
  • SSH key access to each robot (ssh unitree@<ip> must work without a password — the installer uses BatchMode=yes).
  • On the same network as the robots (they POST back to the workstation during test).

Robot:

  • Docker (Engine ≥ 20). The unitree user must be in the docker group.
  • Internet at build time (pulls the base image; R1/Go2 also apt/pip the DDS stack).
  • systemd with a user bus (standard on Ubuntu 20.04+). No sudo needed.
  • Architecture: arm64 (Jetson / backpack). Images build natively on the robot.

5. Quick start

cd Project/fleet

# Interactive — asks robot type, IP, and (if new) robot name + server:
./fleet_install.sh

# …or scripted:
./fleet_install.sh install r1 10.255.254.82 --sn E39N4000Q6D7E70F --name r1_82 \
    --server-ip 10.255.254.83 --port 8799 --token <device-token>

# See what it's sending, tail logs, check the service:
./fleet_install.sh data   r1 10.255.254.82
./fleet_install.sh logs   r1 10.255.254.82
./fleet_install.sh status r1 10.255.254.82

# End-to-end test against your workstation acting as the server:
./fleet_install.sh test   r1 10.255.254.82

# Remove everything:
./fleet_install.sh uninstall r1 10.255.254.82

6. Installer reference

Interactive mode (no arguments)

./fleet_install.sh

Flow:

  1. Which robot? 1) g1 2) r1 3) go2
  2. Robot IP and SSH user (default unitree; it verifies SSH works).
  3. It detects whether the agent is already installed (systemd unit file or container present) and branches:
    • Installed → menu: 1) show data 2) status 3) logs 4) reinstall 5) UNINSTALL 6) quit.
    • Not installed → prompts Robot name (SN), Fleet server IP (auto-detected default), port, device token, then installs.

Scriptable commands

./fleet_install.sh <command> <g1|r1|go2> <ip> [options]
command action
install rsync agent → robot, build image, create container, install + enable systemd service
uninstall disable/remove service, remove container, image, and ~/sanad_api_<type>
status systemd service state + container state
data recent telemetry/map log lines (what it's currently sending)
logs docker logs -f (live tail)
test start the workstation server, push a real post from the robot, verify receipt (PASS/FAIL)

Options

install requires the essentials to be entered explicitly — the robot's real serial, the token, and the server:

./fleet_install.sh install <type> <ip> --sn <robot-serial> --token <device-token> \
    --server-url https://eco.yslootahrobotics.com [--post <ingest-path>] [--name <display>]
option default meaning
--sn SERIAL required for install robot's REAL serial (keys the robot on the server), e.g. E39N4000Q6D7E70F
--name NAME <model>_<last-octet> friendly display name (e.g. r1_82, g1_58)
--token TOK test-token device bearer token
--server-url URL — full fleet server URL (https://…) → VERIFY_TLS=1
--post PATH agent default ingest POST path (telemetry endpoint; map endpoint for g1)
--server-ip IP auto (route toward robot) alternative: local test server by IP (VERIFY_TLS=0)
--port N 8799 test-server port (with --server-ip)
--user USER unitree SSH user on the robot
--keep-server off (test) leave the workstation test server running

The interactive flow prompts for the same set: robot type → IP → SN (required) → display name → SERVER_URL-or-IP → token → POST endpoint.

7. The auto-start service (systemd)

Because the robots have no passwordless sudo, the agent runs as a user-level systemd service (no root needed):

  • Unit file: ~/.config/systemd/user/sanad-api-<type>.service
  • Boot auto-start: enabled via loginctl enable-linger <user> (allowed without sudo) so the user manager starts at boot before login.
  • The service owns the container lifecycle: 
    [Service]
Type=simple
Restart=always
RestartSec=5
ExecStart=/usr/bin/docker start -a sanad-api-<type>
ExecStop=/usr/bin/docker stop -t 10 sanad-api-<type>
  • The container is created with docker create (no docker restart policy) so systemd is the single owner — Restart=always also covers the boot race where the user manager starts before dockerd is ready (it retries every 5 s).

Managing it on the robot:

systemctl --user status  sanad-api-r1
systemctl --user restart sanad-api-r1
systemctl --user stop    sanad-api-r1
journalctl --user -u sanad-api-r1 -f      # or: docker logs -f sanad-api-r1

8. Configuration reference

Config is env-only. The installer writes ~/sanad_api_<type>/.env on the robot; .env.example in each agent dir documents every key.

Common (all agents)

var required default meaning
SERVER_URL ✅ — fleet server base URL
DEVICE_TOKEN ✅ — bearer token (per robot)
SN — <type>_0000 robot fleet id
VERIFY_TLS — 1 verify server TLS cert (0 for self-signed dev)
HTTP_TIMEOUT — 10–30 per-request timeout (s)
POLL_INTERVAL — 2 (tel) / 30 (map) loop cadence (s)

G1 map uploader

var default meaning
ROBOT sanad web_nav3 robot name → maps subdir + X-Robot-Name
MAPS_DIR /data/maps mounted web_nav3 maps/ (<robot>/*.db)
DATA_DIR — mounted web_nav3 web/data/ (per-map places)
LEGACY_PLACES — optional legacy places.json
WEB_NAV3_URL — optional http://127.0.0.1:8765 (to learn the active map)
MAP_SELECT all all · active · newest
MAP_UPLOAD_MODE multipart multipart · base64json
MAP_ENDPOINT /api/v1/fleet/ingest/{sn}/map path template
STATE_DIR /data/state upload-fingerprint state

R1 / Go2 telemetry

var default meaning
DDS_INTERFACE R1 eth10 / Go2 eth0 NIC that sees robot DDS
DDS_DOMAIN 0 DDS domain id
MAC_INTERFACE = DDS_INTERFACE NIC whose MAC is reported
R1_READ_FSM 0 (R1) read loco FSM for status — read-only GET RPC
R1_POSITION_SOURCE / GO2_POSITION_SOURCE none none · rosbridge (· sportmode Go2)
ROSBRIDGE_URL ws://127.0.0.1:9090 position source when rosbridge
LOW_SOC 15 %→ LOW_BATTERY fault
MOTOR_TEMP_MAX 85 °C→ MOTOR_OVERTEMP fault
TELEMETRY_ENDPOINT /api/v1/fleet/ingest/telemetry path

9. Per-robot specifics

G1 (map) — real maps live inside the nav container

The G1's saved maps are stored inside the p4_Foxy_sanad (Package_4 nav) container at /home/unitree/marcus_nav2_test/maps, which Package_4 does not bind-mount to the host. So the uploader (which mounts a host dir) sees 0 maps until you do one of:

  1. Add a host bind-mount for maps/ (and web/data/) to the Package_4 nav service, then set the uploader's MAPS_HOST_DIR/DATA_HOST_DIR to those paths.
  2. Put both containers on a shared named volume for the maps dir.

The test command seeds a throwaway fixture map so the upload path is verified regardless.

R1 (telemetry)

  • DDS link is eth10 (= 192.168.123.164); wlan0 is the fleet LAN.
  • Battery: rt/lf/bmsstate (BmsState_), soc 0–100, charging from current sign. Coexists fine with the running sanadr1 app (DDS allows many readers).

Go2 (telemetry) — ⚠️ unverified on hardware

  • Uses unitree_go DDS. Battery is nested in rt/lowstate.bms_state (Go2 has no separate BMS topic).
  • Written from the SDK layout; image builds and unitree_go imports, simulate is correct, but not yet run on a real Go2 — confirm the bms_state current sign (charging polarity) and sportmodestate fields on the robot.

10. The fleet test server

fleet_test_server.py stands in for the real fleet server so you can verify a deploy end-to-end from your workstation.

PORT=8799 REQLOG=/tmp/fleet.jsonl python3 fleet_test_server.py
  • Binds 0.0.0.0:8799.
  • GET /ping → 200 (reachability check the installer uses).
  • Accepts POST …/map and POST …/telemetry; logs each request to REQLOG and prints a live summary.

./fleet_install.sh test … starts it automatically, triggers a post from the robot, asserts the payload arrived, and prints PASS/FAIL.

11. How a deploy works internally

install:
  rsync agents/<type>/  →  unitree@<ip>:~/sanad_api_<type>/   (--delete, minus .env/state)
  write ~/sanad_api_<type>/.env                               (SERVER_URL, TOKEN, SN, iface…)
  ssh: docker build -t sanad-api-<type>:latest .              (native arm64)
  ssh: docker create --name sanad-api-<type> <run-args> …     (no docker restart policy)
  write ~/.config/systemd/user/sanad-api-<type>.service
  ssh: loginctl enable-linger ; systemctl --user enable --now …

run-args:  --network host --env-file …/.env
           (g1 also: -v maps:ro -v web_data:ro -v state)

--network host is required: G1 to reach 127.0.0.1:8765 (web_nav3) if used, and R1/Go2 so the robot's DDS traffic is visible (DDS multicast doesn't cross a NAT bridge).

12. Fleet inventory

robot agent(s) IP SSH arch DDS iface SN
G1 telemetry + map (g1) 10.255.254.58 unitree (key) arm64 eth0 E21D6000PB89GF88 (name g1_58)
R1 telemetry (r1) 10.255.254.82 unitree (key) arm64 eth10 E39N4000Q6D7E70F (name r1_82)
Go2 telemetry (go2) (TBD) unitree arm64 eth0 —

Workstation (deploy host + test fleet server): 10.255.254.83 (wlp4s0).

13. Troubleshooting

symptom cause / fix
robot cannot reach the workstation server (http 000) during test robot→workstation blocked. Check both are on the same subnet; open the port if a firewall is on (sudo ufw allow 8799).
G1 uploads nothing / "0 maps" no saved map, or maps aren't mounted — see §9 G1.
R1/Go2 telemetry POST failed … Connection refused the target server (e.g. the test server) is down. Point at the real server: install … --server-ip <fleet> --token <real>.
R1/Go2 battery=null, status=offline forever DDS not seen. Wrong DDS_INTERFACE (R1 = eth10) or robot firmware down. Confirm with ip -o addr on the robot.
build is very slow (R1/Go2) first build compiles CycloneDDS (minutes). Run detached and poll: ssh … 'setsid bash -c "cd ~/sanad_api_r1 && docker build -t sanad-api-r1:latest . > build.log 2>&1" </dev/null &' then watch build.log. Docker layer cache survives reboots.
service didn't come up after reboot linger must be on: `loginctl show-user
unbound variable / weird prompt behavior ensure you're on the current fleet_install.sh (uses ssh -n for command calls so SSH doesn't eat stdin).

14. Security notes

  • Outbound only. Agents open no inbound ports on the robot; they POST out over HTTPS with a per-robot bearer token.
  • Least privilege. Runs as the unitree user (docker group), user-level systemd, no root. Read-only w.r.t. the robot (no motion commands).
  • Token handling. .env holds the device token and is never rsynced back or committed (.gitignored). The installer writes it directly to the robot.
  • TLS. VERIFY_TLS=1 in production; 0 only for the local test server.