diff --git a/PIPELINE.md b/PIPELINE.md new file mode 100644 index 0000000..182d593 --- /dev/null +++ b/PIPELINE.md @@ -0,0 +1,265 @@ +# Sanad Fleet — Data Pipeline + +How robot state becomes a record on the YS Lootah fleet dashboard, end to end. +This is the reference for **what flows, in what shape, when, and how failures are +handled**. For install/ops, see [README.md](README.md). + +--- + +## 1. Big picture + +``` + ROBOT (edge) │ YS LOOTAH (cloud) + │ + ┌───────────────┐ read ┌──────────────┐ │ HTTPS POST ┌──────────────┐ + │ robot sources │ ─────────▶ │ sanad_api_* │ ─┼───────────────▶ │ fleet server │ + │ DDS / files │ │ (Docker) │ │ Bearer token │ (ingest API) │ + └───────────────┘ └──────────────┘ │ └──────┬───────┘ + ▲ │ │ │ + │ passive, read-only │ systemd │ ▼ + │ (never commands motion) │ user svc │ ┌──────────────┐ + ▼ │ │ dashboard │ + stays "online" │ │ storage │ + via heartbeat │ │ alerts │ + │ └──────────────┘ +``` + +- **Direction:** outbound only. The robot opens no inbound ports; every call is an + HTTPS `POST` to the fleet server. +- **Transport:** HTTP/1.1 + JSON (telemetry) or `multipart/form-data` (map). +- **Auth:** `Authorization: Bearer ` on every request. +- **Identity:** each robot is keyed by `sn` (fleet id, e.g. `r1_82`); the hardware + `mac` rides along in telemetry. + +--- + +## 2. API contract + +The full spec (from `sanad-tasks-en.html`) defines these ingest endpoints. This +repo currently implements the **bold** ones; the rest are documented for later. + +| endpoint | method | agent | status | +|---|---|---|---| +| **`/api/v1/fleet/ingest/telemetry`** | POST | R1, Go2 | ✅ implemented | +| **`/api/v1/fleet/ingest/{sn}/map`** | POST | G1 | ✅ implemented | +| `/api/v1/fleet/ingest/{sn}/commands` | GET | — | ⏳ spec'd, not built | +| `/api/v1/fleet/ingest/commands/{id}/ack` | POST | — | ⏳ | +| `/api/v1/fleet/ingest/{sn}/alert` | POST | — | ⏳ (critical faults; ordinary ones ride in telemetry `faults[]`) | +| `/api/v1/fleet/ingest/{sn}/logs` | POST | — | ⏳ | +| `/api/v1/fleet/ingest/{sn}/remote` | POST | — | ⏳ (tunnel/SSH registration) | + +Auth header (all): `Authorization: Bearer `. + +--- + +## 3. Telemetry pipeline (R1, Go2) + +**Goal:** a fresh status record every ~2 s; never go dark. + +``` + DDS topics agent (2 s loop) server + ────────── ──────────────── ────── + rt/lowstate ─┐ callbacks ┌─ snapshot() ─┐ build_telemetry() ┌ POST + (LowState_) ├─────────────▶ │ battery ├──────────────────────▶│ /ingest/ + rt/lf/bmsstate│ (background │ temps/dq │ derive status+faults │ telemetry + (BmsState_) ─┘ threads) │ liveness/age │ read mac └ (JSON) + └──────────────┘ + │ no data? + └────────▶ heartbeat (battery:null, offline) +``` + +### 3.1 Sources per robot + +| field | R1 (`unitree_hg`) | Go2 (`unitree_go`) | +|---|---|---| +| `battery` (0–100) | `rt/lf/bmsstate` → `BmsState_.soc` | `rt/lowstate` → `LowState_.bms_state.soc` | +| `charging` | `BmsState_.current` > +0.05 A | `bms_state.current` > +0.05 A | +| `faults[]` | `rt/lowstate` motor temps + staleness | same | +| `status` | derived (see 3.3); optional loco FSM `GET 7001` | derived | +| `position` | optional `rosbridge /odom` | optional `rosbridge /odom` or `rt/lf/sportmodestate` | +| `mac` | NIC (`/sys/class/net//address`) | NIC | + +DDS is initialized once (`ChannelFactoryInitialize(domain, interface)`); each +topic has a subscriber whose callback updates a locked in-memory snapshot. The +loop reads the snapshot — it never blocks on the network. + +### 3.2 Payload + +```json +POST /api/v1/fleet/ingest/telemetry +Authorization: Bearer +Content-Type: application/json + +{ "sn": "r1_82", + "mac": "4c:bb:47:51:25:9a", + "battery": 80, + "charging": false, + "status": "idle", + "position": { "x": 12.4, "y": 3.1 }, // or null when no localization source + "faults": [], + "ts": 1731000000 } +``` + +### 3.3 Status derivation + +``` +if no rt/lowstate for >3 s AND never saw battery → "offline" +elif charging (bms current > +0.05 A) → "charging" +elif max |joint velocity| > 0.15 rad/s → "moving" +elif R1_READ_FSM and FSM id known → FSM label (811 ready / 4 standing / 1 damping / 0 zero_torque) +else → "idle" +``` + +### 3.4 Faults + +Ordinary faults ride inside telemetry `faults[]` (the spec reserves the separate +`/alert` endpoint for critical, immediate events like e-stop): + +| code | trigger | severity | +|---|---|---| +| `LOW_BATTERY` | `soc ≤ LOW_SOC` (default 15) | warning | +| `MOTOR_OVERTEMP` | any motor temp ≥ `MOTOR_TEMP_MAX` (default 85 °C) | warning | +| `COMMS_STALE` | no `rt/lowstate` for > 3 s | critical | + +### 3.5 Heartbeat & failure handling + +- **No DDS / no state** → still POST with `battery:null`, `status:"offline"` so the + dashboard shows the robot as reachable (the spec's "send a heartbeat" rule). +- **unitree_sdk2py missing** → the agent logs a warning and runs in heartbeat mode + (no crash). +- **POST fails** (transport or non-2xx) → logged, loop continues; next tick retries + in `POLL_INTERVAL` seconds. No back-pressure, no queue (latest state wins). +- **Every tick is wrapped** — one bad read can't kill the loop. + +--- + +## 4. Map pipeline (G1) + +**Goal:** keep the server's copy of the nav map current, sending only on change. + +``` + files on disk agent (30 s scan) server + ───────────── ───────────────── ────── + maps//*.db ─┐ discover ┌ fingerprint (size+mtime) ─┐ changed? ┌ POST + maps_meta.json ├────────▶ │ if changed: sha256 ├──yes──────▶│ /ingest/ + web/data// │ │ load places → points[] │ │ {sn}/map + places/.json ┘ └ build meta + db bytes ─────┘ └ (multipart) + │ unchanged + └────────▶ skip (state/uploaded.json) +``` + +### 4.1 Sources + +- **Map file:** RTAB-Map SQLite `.db` — `maps//.db` (web_nav3). +- **Metadata:** `maps//maps_meta.json` → `{ "": {description, created_at} }`. +- **Places → points:** `web/data//places/.json` + (`{ "": {x, y, z, qx, qy, qz, qw} }`), converted to + `{name, type:"waypoint", x, y, yaw}` (yaw computed from the quaternion). + +### 4.2 Change detection + +1. Cheap pre-check: `size + mtime` vs `STATE_DIR/uploaded.json`. +2. If different, compute `sha256` of the `.db` (content-true). +3. Upload; on success, record the new fingerprint. A restart re-reads state → no + redundant re-upload. + +`MAP_SELECT` chooses scope: `all` (default), `active` (via web_nav3 +`GET /api/status`), or `newest`. + +### 4.3 Payload (two wire formats) + +`multipart/form-data` (default) — file part `db` + form field `meta`: + +``` +POST /api/v1/fleet/ingest/g1_7892/map +Authorization: Bearer +Content-Type: multipart/form-data + + db = (application/octet-stream) + meta = { "sn":"g1_7892", "name":"floor-1", "file":"floor-1.db", + "format":"rtabmap_db", "size_bytes":6994944, "sha256":"…", + "mtime":1731000000, "description":"ground floor", + "points":[ {"name":"dock","type":"waypoint","x":1.2,"y":3.4,"yaw":0.0} ] } +``` + +`base64json` (set `MAP_UPLOAD_MODE=base64json`) — same fields as JSON with the +`.db` as `db_base64`. + +> **Server note:** this uploads the raw RTAB-Map `.db` (not a rendered PNG), so the +> server must accept a `format:"rtabmap_db"` artifact. The spec's image-based map +> body (`image_base64`/`resolution`/`origin`) would instead require rendering the +> live `/map` OccupancyGrid over rosbridge — a different path, not used here. + +--- + +## 5. Timing + +| stream | cadence | trigger | +|---|---|---| +| telemetry (R1/Go2) | every `POLL_INTERVAL` (2 s) | timer | +| map (G1) | scan every `POLL_INTERVAL` (30 s) | uploads only on content change | +| DDS reads | continuous (subscriber callbacks) | firmware publish rate | +| heartbeat | same as telemetry cadence | when state is unreadable | + +--- + +## 6. Where it runs + +``` +robot host +└─ systemd --user + └─ sanad-api-.service (Restart=always, enabled at boot via linger) + └─ docker start -a sanad-api- + └─ container (--network host) + └─ python -u sanad_api_.py ← the loop above +``` + +- **`--network host`**: DDS multicast visibility (R1/Go2) and localhost access to + web_nav3:8765 (G1); also the robot's real NIC MAC. +- **Single owner**: the container is `docker create`d without a docker restart + policy; systemd owns start/stop/restart. + +--- + +## 7. End-to-end sequences + +### Telemetry tick + +``` +loop DDS(sub) agent fleet server + │ (2 s) │ │ │ + │────────────▶│ snapshot │ │ + │ │───────────▶│ build payload │ + │ │ │──── POST JSON ────▶│ (Bearer) + │ │ │◀──── 200 OK ───────│ + │ │ │ log "telemetry ok" │ +``` + +### Map change + +``` +scan(30s) disk agent fleet server + │────────────▶│ discover │ │ + │ │───────────▶│ fingerprint │ + │ │ │ changed → sha256 │ + │ │ │ load points │ + │ │ │── POST multipart ─▶│ (db + meta) + │ │ │◀──── 200 OK ───────│ + │ │ │ save state │ +``` + +--- + +## 8. Verified behavior (real hardware) + +Confirmed against the live fleet during bring-up: + +- **R1** streamed real telemetry every 2 s; battery read live and drained + `97 → 80%` across the session; `mac`/`sn` correct; heartbeat kicked in when the + server was down. +- **G1** uploaded a map (`multipart`, `format:rtabmap_db`, points with correct + yaw) to the server. +- The fleet server received a **G1 map and R1 telemetry in the same window** — the + multi-robot pipeline works concurrently. +- **Go2**: pipeline code + image verified (builds, `unitree_go` imports, simulate + payloads correct); **not yet run on a real Go2**. diff --git a/README.md b/README.md index d813ce9..527bd19 100644 --- a/README.md +++ b/README.md @@ -1,94 +1,378 @@ # Sanad Fleet Agents -Per-robot agents that report to the YS Lootah fleet server, each shipped as -**full Docker** with a user-level **systemd auto-start service**. Deploy, manage, -and remove them over SSH with one interactive script. +On-robot agents that report each robot's state to the **YS Lootah fleet server**. +Every robot type ships as its own self-contained **Docker** image with a +user-level **systemd auto-start service**, deployed, managed, and removed over +SSH by a single script. No `docker-compose` on the robot, **no `sudo`** required. + +> For the end-to-end data flow (topics → agent → HTTP → server), see +> [PIPELINE.md](PIPELINE.md). + +--- + +## Table of contents + +1. [What this is](#1-what-this-is) +2. [Directory layout](#2-directory-layout) +3. [The three agents](#3-the-three-agents) +4. [Prerequisites](#4-prerequisites) +5. [Quick start](#5-quick-start) +6. [Installer reference](#6-installer-reference) +7. [The auto-start service (systemd)](#7-the-auto-start-service-systemd) +8. [Configuration reference](#8-configuration-reference) +9. [Per-robot specifics](#9-per-robot-specifics) +10. [The fleet test server](#10-the-fleet-test-server) +11. [How a deploy works internally](#11-how-a-deploy-works-internally) +12. [Fleet inventory](#12-fleet-inventory) +13. [Troubleshooting](#13-troubleshooting) +14. [Security notes](#14-security-notes) + +--- + +## 1. What this is + +The fleet server (built by YS Lootah — "their side") needs each robot to push its +status outbound over HTTPS. This repo is **the robot side**: a small agent per +robot type that reads local state and POSTs it to documented endpoints, plus the +tooling to install/manage it across the fleet. + +- **G1** → uploads the **navigation map**. +- **R1** / **Go2** → stream **telemetry** (battery, charging, status, position, faults). + +Each is a drop-in Docker image; install it on any new robot of that type and it +just works. + +--- + +## 2. Directory layout ``` Project/fleet/ - fleet_install.sh # interactive + scriptable deploy/manage tool - fleet_test_server.py # your workstation standing in as the fleet server (tests) - agents/ - g1/ sanad_api_g1.py — MAP uploader (POST …/{sn}/map) - r1/ sanad_api_r1.py — TELEMETRY unitree_hg (POST …/telemetry) - go2/ sanad_api_go2.py — TELEMETRY unitree_go (POST …/telemetry) [unverified on hw] +├── README.md ← this file +├── PIPELINE.md ← end-to-end data-flow reference +├── fleet_install.sh ← deploy / manage / remove over SSH (interactive + scriptable) +├── fleet_test_server.py ← workstation stand-in for the fleet server (for tests) +└── agents/ + ├── g1/ ← MAP uploader + │ ├── sanad_api_g1.py + │ ├── Dockerfile (lean: python + requests) + │ ├── requirements.txt + │ ├── docker-compose.yml (manual local use only) + │ └── .env.example + ├── r1/ ← TELEMETRY (unitree_hg) + │ ├── sanad_api_r1.py + │ ├── Dockerfile (DDS: CycloneDDS + unitree_sdk2py) + │ ├── vendor/ (unitree_sdk2py wheel + crc libs) + │ └── .env.example + └── go2/ ← TELEMETRY (unitree_go) [unverified on hardware] + ├── sanad_api_go2.py + ├── Dockerfile + ├── vendor/ + └── .env.example ``` -All three are self-contained (own `Dockerfile`, `.env.example`). Deploy uses -plain `docker build`/`docker run` — **no docker-compose needed on the robot** -(R1/Go2 don't have it). Source is pushed from this canonical workstation copy via -`rsync`; images build natively on each arm64 robot; a **user systemd unit** -(linger-enabled, **no sudo**) owns start/stop/auto-start-on-boot. +`agents/` is the **single source of truth**. The installer rsyncs the selected +agent to the robot; you never edit files on the robot. -## Install / manage +--- -Interactive (asks robot type → IP → detects installed-or-not → right actions): +## 3. The three agents + +| agent | robot | endpoint | what it sends | cadence | +|---|---|---|---|---| +| `sanad_api_g1` | Unitree G1 | `POST /api/v1/fleet/ingest/{sn}/map` | RTAB-Map `.db` + places (points) | on change (~30 s scan) | +| `sanad_api_r1` | Unitree R1 EDU | `POST /api/v1/fleet/ingest/telemetry` | battery, charging, status, position, faults, mac | every ~2 s | +| `sanad_api_go2` | Unitree Go2 | `POST /api/v1/fleet/ingest/telemetry` | same as R1 | every ~2 s | + +All requests carry `Authorization: Bearer `. Full payload schemas +and the data pipeline are in [PIPELINE.md](PIPELINE.md). + +**Design principles** + +- **No ROS in the agents.** G1 reads map files directly; R1/Go2 read DDS via + `unitree_sdk2py`. This keeps images small and portable. +- **Read-only.** Telemetry agents never command motion (only passive DDS reads +, + optionally, the read-only `GET_FSM_ID` RPC). +- **Never crash the loop.** Every tick is wrapped; transient errors are logged and + retried. Telemetry sends a heartbeat when it can't read state so the robot stays + "online". +- **Change-detected uploads.** The map is only re-sent when its content hash + changes. + +--- + +## 4. Prerequisites + +**Workstation (the machine you deploy from):** +- `bash`, `ssh`, `rsync`, `python3`. +- SSH **key** access to each robot (`ssh unitree@` must work without a + password — the installer uses `BatchMode=yes`). +- On the same network as the robots (they POST back to the workstation during + `test`). + +**Robot:** +- Docker (Engine ≥ 20). The `unitree` user must be in the `docker` group. +- Internet at build time (pulls the base image; R1/Go2 also `apt`/`pip` the DDS + stack). +- `systemd` with a user bus (standard on Ubuntu 20.04+). **No sudo needed.** +- Architecture: arm64 (Jetson / backpack). Images build natively on the robot. + +--- + +## 5. Quick start ```bash +cd Project/fleet + +# Interactive — asks robot type, IP, and (if new) robot name + server: +./fleet_install.sh + +# …or scripted: +./fleet_install.sh install r1 10.255.254.82 --sn r1_82 \ + --server-ip 10.255.254.83 --port 8799 --token + +# See what it's sending, tail logs, check the service: +./fleet_install.sh data r1 10.255.254.82 +./fleet_install.sh logs r1 10.255.254.82 +./fleet_install.sh status r1 10.255.254.82 + +# End-to-end test against your workstation acting as the server: +./fleet_install.sh test r1 10.255.254.82 + +# Remove everything: +./fleet_install.sh uninstall r1 10.255.254.82 +``` + +--- + +## 6. Installer reference + +### Interactive mode (no arguments) + +``` ./fleet_install.sh ``` -Scriptable: +Flow: +1. **Which robot?** `1) g1 2) r1 3) go2` +2. **Robot IP** and **SSH user** (default `unitree`; it verifies SSH works). +3. It **detects whether the agent is already installed** (systemd unit file *or* + container present) and branches: + - **Installed** → menu: `1) show data 2) status 3) logs 4) reinstall + 5) UNINSTALL 6) quit`. + - **Not installed** → prompts **Robot name (SN)**, **Fleet server IP** + (auto-detected default), **port**, **device token**, then installs. + +### Scriptable commands + +``` +./fleet_install.sh [options] +``` + +| command | action | +|---|---| +| `install` | rsync agent → robot, build image, create container, install + enable systemd service | +| `uninstall` | disable/remove service, remove container, image, and `~/sanad_api_` | +| `status` | systemd service state + container state | +| `data` | recent telemetry/map log lines (what it's currently sending) | +| `logs` | `docker logs -f` (live tail) | +| `test` | start the workstation server, push a real post from the robot, verify receipt (PASS/FAIL) | + +### Options + +| option | default | meaning | +|---|---|---| +| `--sn NAME` | `_` | robot's fleet id (the `sn` field) | +| `--server-ip IP` | auto (route toward robot) | fleet server the robot posts to | +| `--port N` | `8799` | fleet server port | +| `--token TOK` | `test-token` | device bearer token | +| `--user USER` | `unitree` | SSH user on the robot | +| `--keep-server` | off | (test) leave the workstation test server running | + +--- + +## 7. The auto-start service (systemd) + +Because the robots have **no passwordless sudo**, the agent runs as a **user-level +systemd service** (no root needed): + +- Unit file: `~/.config/systemd/user/sanad-api-.service` +- Boot auto-start: enabled via `loginctl enable-linger ` (allowed without + sudo) so the user manager starts at boot before login. +- The service owns the container lifecycle: + ```ini + [Service] + Type=simple + Restart=always + RestartSec=5 + ExecStart=/usr/bin/docker start -a sanad-api- + ExecStop=/usr/bin/docker stop -t 10 sanad-api- + ``` +- The container is created with `docker create` (no docker restart policy) so + **systemd is the single owner** — `Restart=always` also covers the boot race + where the user manager starts before `dockerd` is ready (it retries every 5 s). + +**Managing it on the robot:** ```bash -./fleet_install.sh install [--sn NAME] [--server-ip IP] [--port N] [--token TOK] -./fleet_install.sh data # show what it's currently sending -./fleet_install.sh status -./fleet_install.sh logs -./fleet_install.sh test # end-to-end vs workstation server -./fleet_install.sh uninstall # service + container + image + dir +systemctl --user status sanad-api-r1 +systemctl --user restart sanad-api-r1 +systemctl --user stop sanad-api-r1 +journalctl --user -u sanad-api-r1 -f # or: docker logs -f sanad-api-r1 ``` -`install`/`uninstall` are detected automatically in interactive mode: if the -agent is already on the robot it offers **data / status / logs / reinstall / -uninstall**; otherwise it prompts for **robot name (SN)**, server IP, port, token -and installs. +--- -### Auto-start service +## 8. Configuration reference -Each install writes `~/.config/systemd/user/sanad-api-.service`, enables -linger (`loginctl enable-linger`, no sudo), and `systemctl --user enable --now`s -it. It starts on boot and restarts on crash. Manage with: +Config is env-only. The installer writes `~/sanad_api_/.env` on the robot; +`.env.example` in each agent dir documents every key. + +### Common (all agents) + +| var | required | default | meaning | +|---|---|---|---| +| `SERVER_URL` | ✅ | — | fleet server base URL | +| `DEVICE_TOKEN` | ✅ | — | bearer token (per robot) | +| `SN` | — | `_0000` | robot fleet id | +| `VERIFY_TLS` | — | `1` | verify server TLS cert (`0` for self-signed dev) | +| `HTTP_TIMEOUT` | — | 10–30 | per-request timeout (s) | +| `POLL_INTERVAL` | — | 2 (tel) / 30 (map) | loop cadence (s) | + +### G1 map uploader + +| var | default | meaning | +|---|---|---| +| `ROBOT` | `sanad` | web_nav3 robot name → maps subdir + `X-Robot-Name` | +| `MAPS_DIR` | `/data/maps` | mounted web_nav3 `maps/` (`/*.db`) | +| `DATA_DIR` | — | mounted web_nav3 `web/data/` (per-map places) | +| `LEGACY_PLACES` | — | optional legacy `places.json` | +| `WEB_NAV3_URL` | — | optional `http://127.0.0.1:8765` (to learn the active map) | +| `MAP_SELECT` | `all` | `all` · `active` · `newest` | +| `MAP_UPLOAD_MODE` | `multipart` | `multipart` · `base64json` | +| `MAP_ENDPOINT` | `/api/v1/fleet/ingest/{sn}/map` | path template | +| `STATE_DIR` | `/data/state` | upload-fingerprint state | + +### R1 / Go2 telemetry + +| var | default | meaning | +|---|---|---| +| `DDS_INTERFACE` | R1 `eth10` / Go2 `eth0` | NIC that sees robot DDS | +| `DDS_DOMAIN` | `0` | DDS domain id | +| `MAC_INTERFACE` | = `DDS_INTERFACE` | NIC whose MAC is reported | +| `R1_READ_FSM` | `0` | (R1) read loco FSM for status — read-only GET RPC | +| `R1_POSITION_SOURCE` / `GO2_POSITION_SOURCE` | `none` | `none` · `rosbridge` (· `sportmode` Go2) | +| `ROSBRIDGE_URL` | `ws://127.0.0.1:9090` | position source when rosbridge | +| `LOW_SOC` | `15` | %→ `LOW_BATTERY` fault | +| `MOTOR_TEMP_MAX` | `85` | °C→ `MOTOR_OVERTEMP` fault | +| `TELEMETRY_ENDPOINT` | `/api/v1/fleet/ingest/telemetry` | path | + +--- + +## 9. Per-robot specifics + +### G1 (map) — real maps live inside the nav container + +The G1's saved maps are stored **inside** the `p4_Foxy_sanad` (Package_4 nav) +container at `/home/unitree/marcus_nav2_test/maps`, which Package_4 does **not** +bind-mount to the host. So the uploader (which mounts a host dir) sees **0 maps** +until you do one of: + +1. Add a host bind-mount for `maps/` (and `web/data/`) to the Package_4 `nav` + service, then set the uploader's `MAPS_HOST_DIR`/`DATA_HOST_DIR` to those paths. +2. Put both containers on a shared named volume for the maps dir. + +The `test` command seeds a throwaway fixture map so the upload path is verified +regardless. + +### R1 (telemetry) + +- DDS link is **`eth10`** (= `192.168.123.164`); `wlan0` is the fleet LAN. +- Battery: `rt/lf/bmsstate` (`BmsState_`), `soc` 0–100, charging from `current` + sign. Coexists fine with the running `sanadr1` app (DDS allows many readers). + +### Go2 (telemetry) — ⚠️ unverified on hardware + +- Uses `unitree_go` DDS. Battery is nested in **`rt/lowstate.bms_state`** (Go2 has + no separate BMS topic). +- Written from the SDK layout; image builds and `unitree_go` imports, simulate is + correct, but **not yet run on a real Go2** — confirm the `bms_state` current sign + (charging polarity) and `sportmodestate` fields on the robot. + +--- + +## 10. The fleet test server + +`fleet_test_server.py` stands in for the real fleet server so you can verify a +deploy end-to-end from your workstation. ```bash -ssh unitree@ 'systemctl --user status sanad-api-r1' -ssh unitree@ 'systemctl --user restart sanad-api-r1' +PORT=8799 REQLOG=/tmp/fleet.jsonl python3 fleet_test_server.py ``` -## Fleet (this deployment) +- Binds `0.0.0.0:8799`. +- `GET /ping` → `200` (reachability check the installer uses). +- Accepts `POST …/map` and `POST …/telemetry`; logs each request to `REQLOG` and + prints a live summary. -| robot | agent | ip | ssh user | arch | DDS iface | -|---|---|---|---|---|---| -| G1 | map | `10.255.254.58` | `unitree` (key) | arm64 | — | -| R1 | telemetry | `10.255.254.82` | `unitree` (key) | arm64 | `eth10` | -| Go2 | telemetry | *(when available)* | `unitree` | arm64 | `eth0` | +`./fleet_install.sh test …` starts it automatically, triggers a post from the +robot, asserts the payload arrived, and prints `PASS`/`FAIL`. -Workstation fleet server (for tests): **10.255.254.83** via `fleet_test_server.py`. +--- -## What each agent sends, and when +## 11. How a deploy works internally -**G1 map** → `POST /api/v1/fleet/ingest/{sn}/map` — on change (scan ~30 s; -size+mtime→sha256). Sends the RTAB-Map `.db` + places as `multipart` (file `db` + -`meta` JSON `{sn,name,format:"rtabmap_db",size_bytes,sha256,points[…]}`) or -`base64json`. Reads web_nav3 `maps//*.db` + `web/data//places/*.json`. - -**R1 / Go2 telemetry** → `POST /api/v1/fleet/ingest/telemetry` — every ~2 s; -heartbeat (`battery:null,status:offline`) when DDS is silent so it stays online: -```json -{ "sn":"r1_82", "mac":"…", "battery":74, "charging":false, - "status":"idle", "position":{"x":…,"y":…}|null, "faults":[], "ts":… } ``` -- R1 (`unitree_hg`): battery `rt/lf/bmsstate`, faults/liveness `rt/lowstate`. -- Go2 (`unitree_go`): battery from **`rt/lowstate.bms_state`** (nested), no separate BMS topic. -- Both: `status` derived (charging/moving/idle/offline); position optional; read-only (never moves). +install: + rsync agents// → unitree@:~/sanad_api_/ (--delete, minus .env/state) + write ~/sanad_api_/.env (SERVER_URL, TOKEN, SN, iface…) + ssh: docker build -t sanad-api-:latest . (native arm64) + ssh: docker create --name sanad-api- … (no docker restart policy) + write ~/.config/systemd/user/sanad-api-.service + ssh: loginctl enable-linger ; systemctl --user enable --now … -Auth on every request: `Authorization: Bearer `. +run-args: --network host --env-file …/.env + (g1 also: -v maps:ro -v web_data:ro -v state) +``` -## Notes / follow-ups +`--network host` is required: G1 to reach `127.0.0.1:8765` (web_nav3) if used, and +R1/Go2 so the robot's DDS traffic is visible (DDS multicast doesn't cross a NAT +bridge). -- **Point at the real fleet server:** re-run `install … --server-ip --token --sn `. -- **G1 real maps** live *inside* the `p4_Foxy_sanad` nav container (not host-mounted). - Add a `maps/` bind-mount to Package_4's nav service, or share a volume, so the - uploader sees real maps (it reports 0 until then; `test` seeds a fixture). -- **Go2** agent is written from the `unitree_go` SDK layout but **not yet run on a - Go2** — confirm the `bms_state` current sign (charging) and sportmodestate fields. +--- + +## 12. Fleet inventory + +| robot | agent | IP | SSH | arch | DDS iface | SN | +|---|---|---|---|---|---|---| +| G1 | map | `10.255.254.58` | `unitree` (key) | arm64 | — | `g1_7892` | +| R1 | telemetry | `10.255.254.82` | `unitree` (key) | arm64 | `eth10` | `r1_82` | +| Go2 | telemetry | *(TBD)* | `unitree` | arm64 | `eth0` | — | + +Workstation (deploy host + test fleet server): **`10.255.254.83`** (`wlp4s0`). + +--- + +## 13. Troubleshooting + +| symptom | cause / fix | +|---|---| +| `robot cannot reach the workstation server (http 000)` during `test` | robot→workstation blocked. Check both are on the same subnet; open the port if a firewall is on (`sudo ufw allow 8799`). | +| G1 uploads nothing / "0 maps" | no saved map, or maps aren't mounted — see [§9 G1](#g1-map--real-maps-live-inside-the-nav-container). | +| R1/Go2 `telemetry POST failed … Connection refused` | the target server (e.g. the test server) is down. Point at the real server: `install … --server-ip --token `. | +| R1/Go2 `battery=null, status=offline` forever | DDS not seen. Wrong `DDS_INTERFACE` (R1 = `eth10`) or robot firmware down. Confirm with `ip -o addr` on the robot. | +| build is very slow (R1/Go2) | first build compiles CycloneDDS (minutes). Run detached and poll: `ssh … 'setsid bash -c "cd ~/sanad_api_r1 && docker build -t sanad-api-r1:latest . > build.log 2>&1" | grep Linger` → `Linger=yes`. The installer sets it. | +| `unbound variable` / weird prompt behavior | ensure you're on the current `fleet_install.sh` (uses `ssh -n` for command calls so SSH doesn't eat stdin). | + +--- + +## 14. Security notes + +- **Outbound only.** Agents open no inbound ports on the robot; they POST out over + HTTPS with a per-robot bearer token. +- **Least privilege.** Runs as the `unitree` user (docker group), user-level + systemd, no root. Read-only w.r.t. the robot (no motion commands). +- **Token handling.** `.env` holds the device token and is never rsynced back or + committed (`.gitignore`d). The installer writes it directly to the robot. +- **TLS.** `VERIFY_TLS=1` in production; `0` only for the local test server.