diff --git a/PIPELINE.md b/PIPELINE.md index 8ebc188..688e440 100644 --- a/PIPELINE.md +++ b/PIPELINE.md @@ -43,7 +43,7 @@ repo currently implements the **bold** ones; the rest are documented for later. | endpoint | method | agent | status | |---|---|---|---| | **`/api/v1/fleet/ingest/telemetry`** | POST | G1, R1, Go2 | ✅ ~2 s; 27 fields incl. software/firmware/control + all status mirrors | -| **`/api/v1/fleet/ingest/{sn}/map`** | POST | G1, R1, Go2 | ✅ once per content; rtabmap `.db` (≤~7 MB) + slam_toolbox `pgm/yaml`→PNG | +| **`/api/v1/fleet/ingest/{sn}/map`** | POST | G1, R1, Go2 | ✅ once per content; rtabmap `.db` (≤~7 MB) + slam_toolbox/Nav2/Pudu `pgm+yaml`→PNG | | **`/api/v1/fleet/ingest/{sn}/alert`** | POST | G1, R1, Go2 | ✅ each NEW fault, rising edge, string body | | **`/api/v1/fleet/ingest/{sn}/logs`** | POST | G1, R1, Go2 | ✅ agent + `[sanadr1-logs]` project lines every `LOGS_INTERVAL` (60 s) | | **`/api/v1/fleet/ingest/{sn}/remote`** | POST | G1, R1, Go2 | ✅ registers `web` (Sanad dashboard URL) + `ssh` (`ssh unitree@`) every 60 s | @@ -54,7 +54,7 @@ Auth header (all): `Authorization: Bearer `. --- -## 3. Telemetry pipeline (R1, Go2) +## 3. Telemetry pipeline (all agents) **Goal:** a fresh status record every ~2 s; never go dark. @@ -72,13 +72,14 @@ Auth header (all): `Authorization: Bearer `. ### 3.1 Sources per robot -| field | R1 (`unitree_hg`) | Go2 (`unitree_go`) | +| field | G1 / R1 (`unitree_hg`) | Go2 (`unitree_go`) | |---|---|---| | `battery` (0–100) | `rt/lf/bmsstate` → `BmsState_.soc` | `rt/lowstate` → `LowState_.bms_state.soc` | | `charging` | `BmsState_.current` > +0.05 A | `bms_state.current` > +0.05 A | | `faults[]` | `rt/lowstate` motor temps + staleness | same | | `status` | derived (see 3.3); optional loco FSM `GET 7001` | derived | -| `position` | optional `rosbridge /odom` | optional `rosbridge /odom` or `rt/lf/sportmodestate` | +| `position` | G1: `rt/lf/odommodestate` · R1: optional `rosbridge /odom` | optional `rosbridge /odom` or `rt/lf/sportmodestate` | +| `control` | Sanad `GET /api/controller/status` — **read-only** loco mode (G1 `200/4/2/702`, R1 `0/1/4/811`) | same path, Go2 labels | | `mac` | NIC (`/sys/class/net//address`) | NIC | DDS is initialized once (`ChannelFactoryInitialize(domain, interface)`); each @@ -155,14 +156,15 @@ else → "idle" ### 3.4 Faults -Ordinary faults ride inside telemetry `faults[]` (the spec reserves the separate -`/alert` endpoint for critical, immediate events like e-stop): +Faults ride inside telemetry `faults[]` as **strings** (the ingest 500s on fault +objects) — and each **new** fault also fires the `/alert` endpoint (rising edge): | code | trigger | severity | |---|---|---| -| `LOW_BATTERY` | `soc ≤ LOW_SOC` (default 15) | warning | +| `LOW_BATTERY` | `soc ≤ LOW_SOC` (default **50**) | warning | | `MOTOR_OVERTEMP` | any motor temp ≥ `MOTOR_TEMP_MAX` (default 85 °C) | warning | | `COMMS_STALE` | no `rt/lowstate` for > 3 s | critical | +| `GEMINI_BILLING` / `ROBOT_ERROR` | project-log scan (see §5) | critical | ### 3.5 Heartbeat & failure handling @@ -176,42 +178,73 @@ Ordinary faults ride inside telemetry `faults[]` (the spec reserves the separate --- -## 4. Map pipeline (G1) +## 4. Map pipeline (all agents) -**Goal:** keep the server's copy of the nav map current, sending only on change. +**Goal:** keep the server's copy of every saved nav map current, sending each map +**once per content change**. ``` - files on disk agent (30 s scan) server - ───────────── ───────────────── ────── - maps//*.db ─┐ discover ┌ fingerprint (size+mtime) ─┐ changed? ┌ POST - maps_meta.json ├────────▶ │ if changed: sha256 ├──yes──────▶│ /ingest/ - web/data// │ │ load places → points[] │ │ {sn}/map - places/.json ┘ └ build meta + db bytes ─────┘ └ (multipart) - │ unchanged - └────────▶ skip (state/uploaded.json) + files on disk agent (30 s scan) server + ───────────── ───────────────── ────── + MAPS_DIR//*.db ─┐ discover ┌ fingerprint fmt:size:mtime ┐ changed? ┌ POST + MAPS_DIR/**.yaml + .pgm ├─────────▶ │ .db → sha256+multipart├──yes────▶│ /ingest/ + EXTRA_MAP_DIRS (Nav2/Pudu) ─┘ │ pgm+yaml → PNG image JSON │ │ {sn}/map + └────────────────────────────┘ └ + │ unchanged + └────────▶ skip (state/uploaded.json) ``` -### 4.1 Sources +### 4.1 Formats & discovery -- **Map file:** RTAB-Map SQLite `.db` — `maps//.db` (web_nav3). -- **Metadata:** `maps//maps_meta.json` → `{ "": {description, created_at} }`. -- **Places → points:** `web/data//places/.json` - (`{ "": {x, y, z, qx, qy, qz, qw} }`), converted to - `{name, type:"waypoint", x, y, yaw}` (yaw computed from the quaternion). +Two map formats are understood: + +- **RTAB-Map `.db`** (VSLAM) — sent as-is (multipart), with `maps_meta.json` + description and places → `points[]`. **Skipped when larger than + `MAP_MAX_UPLOAD_MB`** (7 — the server 413s at ~8 MB) with a `map.error` note. + A 120 s **stability guard** skips a `.db` still being written (active mapping → + `map.state: "pending"`). +- **`slam_toolbox` / Nav2 / Pudu sets** — `.yaml` + `.pgm` + (+ optional `.posegraph`/`.data`) → uploaded as the spec's **image JSON** + (pure-stdlib PGM→PNG + `resolution` + `origin` + size). Small; always uploads. + +**Search roots** (each existing dir is scanned): +`MAPS_DIR`, `MAPS_DIR/`, `MAPS_DIR/maps_slam`, plus every dir in +`EXTRA_MAP_DIRS` (colon-separated — the installer detects the robot's +`*_nav2_docker/maps` dir and mounts it at `/data/nav2_maps`). + +**Pudu twin dedup:** the Pudu converter emits a plain map *and* a keepout-**baked** +twin; when both `.yaml` and `_keepout_baked.yaml` exist, only the baked +one uploads — the server gets **one** canonical map. + +**Places → points:** `web/data//places/.json` +(`{ "": {x, y, z, qx, qy, qz, qw} }`) → `{name, type:"waypoint", x, y, yaw}` +(yaw from the quaternion). ### 4.2 Change detection -1. Cheap pre-check: `size + mtime` vs `STATE_DIR/uploaded.json`. -2. If different, compute `sha256` of the `.db` (content-true). +1. Cheap pre-check: `fmt : size : mtime` vs `STATE_DIR/uploaded.json`. +2. If different, compute `sha256` (of the `.db`, or of the rendered image JSON). 3. Upload; on success, record the new fingerprint. A restart re-reads state → no redundant re-upload. `MAP_SELECT` chooses scope: `all` (default), `active` (via web_nav3 `GET /api/status`), or `newest`. -### 4.3 Payload (two wire formats) +### 4.3 Wire formats -`multipart/form-data` (default) — file part `db` + form field `meta`: +**Image JSON** (slam_toolbox / Nav2 / Pudu sets — the spec's map body): + +```json +POST /api/v1/fleet/ingest/E39N4000Q6D7E70F/map +{ "sn": "E39N4000Q6D7E70F", "name": "map_keepout_baked", "format": "image", + "resolution": 0.05, "origin": [-5.85, -11.8, 0.0], + "width": 178, "height": 304, + "image_base64": "", + "points": [ {"name":"dock","type":"waypoint","x":1.2,"y":3.4,"yaw":0.0} ] } +``` + +**Multipart** (RTAB-Map `.db`, default `MAP_UPLOAD_MODE=multipart`) — file part +`db` + form field `meta`: ``` POST /api/v1/fleet/ingest/g1_7892/map @@ -228,25 +261,72 @@ Content-Type: multipart/form-data `base64json` (set `MAP_UPLOAD_MODE=base64json`) — same fields as JSON with the `.db` as `db_base64`. -> **Server note:** this uploads the raw RTAB-Map `.db` (not a rendered PNG), so the -> server must accept a `format:"rtabmap_db"` artifact. The spec's image-based map -> body (`image_base64`/`resolution`/`origin`) would instead require rendering the -> live `/map` OccupancyGrid over rosbridge — a different path, not used here. +> **Server note:** raster (`pgm+yaml`) maps use the spec's image body and display +> on the dashboard. The raw `.db` path additionally requires the server to accept a +> `format:"rtabmap_db"` artifact — and is size-capped (~8 MB), so a big VSLAM `.db` +> is skipped; export a raster with `tests/map_export_once.py` instead. + +### 4.4 One-shot upload (`--map-only`) + +`--map-only` runs one discovery + upload pass with **no DDS and no telemetry** — +push a new map without disturbing a live feed: + +```bash +docker run --rm --network host --env-file .env \ + -e EXTRA_MAP_DIRS=/data/nav2_maps -v :/data/nav2_maps:ro \ + sanad-api-:latest --map-only --force +``` --- -## 5. Timing +## 5. Logs, alerts & remote registration + +``` + sanad app json-log ──tail──▶ ProjectLogTail ──┐ + agent's own lines ──ring───────────────────── ├─▶ every 60 s ─▶ POST /{sn}/logs + ┘ (failed ships requeue, last ~400 lines) + + project log ──scan 10 s──▶ LogAlertScanner ──new signature──▶ POST /{sn}/alert + faults[] ──rising edge──────────────────────────────────▶ POST /{sn}/alert + + localhost ports 8001,8014,8011-8013,8000,8080 ──probe──▶ dashboard URL + └─▶ every 60 s ─▶ POST /{sn}/remote (kind "web" + kind "ssh") +``` + +- **Logs** — the agent tails the robot's Sanad container json-log via the read-only + `/host` mount (no docker socket): auto-discovers a running `sanad*` container, + unwraps the json-log, labels lines `[sanadr1-logs]`, filters uvicorn access-log + noise, backfills the last `PROJECT_LOG_BACKFILL` (100) relevant lines on start. +- **Alerts** — `ALERT_LOG_PATTERNS` (`CODE=regex` split by `;;`, **case-sensitive**; + `(?i)` inline where needed) scan the project log every 10 s; each **new** + signature (code + digit-stripped line) alerts once per `ALERT_LOG_COOLDOWN` + (300 s). On startup the last `ALERT_BACKFILL_BYTES` (8 MiB) are scanned so an + already-active error still alerts. Defaults catch **Gemini billing** and any + `ERROR`/`CRITICAL`/`Traceback`. +- **Remote** — probes `REMOTE_PORTS` on localhost for the Sanad dashboard page and + registers `{kind:"web", url:"http://:"}` plus + `{kind:"ssh", command:"ssh unitree@"}` — no change to the Sanad app. + ⚠ LAN-only URL unless `REMOTE_URL` pins a public tunnel. + +--- + +## 6. Timing | stream | cadence | trigger | |---|---|---| -| telemetry (R1/Go2) | every `POLL_INTERVAL` (2 s) | timer | -| map (G1) | scan every `POLL_INTERVAL` (30 s) | uploads only on content change | +| telemetry | every `POLL_INTERVAL` (2 s) | timer | +| map | scan every `MAP_POLL_INTERVAL` (30 s) | uploads only on content change | +| alert scan | every `ALERT_SCAN_INTERVAL` (10 s) | new log signature / rising-edge fault | +| log ship | every `LOGS_INTERVAL` (60 s) | ring buffer + project tail | +| remote register | every `REMOTE_INTERVAL` (60 s) | re-asserts the dashboard/ssh entry | | DDS reads | continuous (subscriber callbacks) | firmware publish rate | | heartbeat | same as telemetry cadence | when state is unreadable | --- -## 6. Where it runs +## 7. Where it runs + +![Deploy — workstation to robot over SSH](deploy.svg) ``` robot host @@ -264,7 +344,7 @@ robot host --- -## 7. End-to-end sequences +## 8. End-to-end sequences ### Telemetry tick @@ -293,16 +373,25 @@ scan(30s) disk agent fleet server --- -## 8. Verified behavior (real hardware) +## 9. Verified behavior (real hardware) Confirmed against the live fleet during bring-up: -- **R1** streamed real telemetry every 2 s; battery read live and drained - `97 → 80%` across the session; `mac`/`sn` correct; heartbeat kicked in when the - server was down. -- **G1** uploaded a map (`multipart`, `format:rtabmap_db`, points with correct - yaw) to the server. -- The fleet server received a **G1 map and R1 telemetry in the same window** — the - multi-robot pipeline works concurrently. +- **R1 is LIVE on production** (`https://eco.yslootahrobotics.com`, sn + `E39N4000Q6D7E70F`): telemetry every 2 s (HTTP 200), real battery draining live, + `control.mode` tracks the actual loco FSM (`lock`/`zero_torque` observed). +- **Maps:** R1's VSLAM raster export uploaded and displayed **on the dashboard**; + the converted **Pudu office map** (`map_keepout_baked`, 178×304 @ 0.05 m) + uploaded via `--map-only` → HTTP 200. Oversized `.db` correctly skipped (413 cap). +- **Alerts verified live:** `LOW_BATTERY` fired at 50/49/48 %…, `GEMINI_BILLING` + and `ROBOT_ERROR` fired from the real sanadr1 log — no false positives after the + case-sensitivity fix. +- **Remote:** R1 registered `http://10.255.254.82:8001` (web) + `ssh unitree@…` + → HTTP 200. +- **Logs:** shipped labeled `[sanadr1-logs]` lines; requeue held them through the + server-side `/logs` 500 until the handler was fixed (then `stored:105`). +- **G1** uploaded a map (`multipart`, `format:rtabmap_db`, points with correct yaw); + G1 map + R1 telemetry landed in the same window — multi-robot works concurrently. + ⚠ G1 currently offline (battery) + deploy hold. - **Go2**: pipeline code + image verified (builds, `unitree_go` imports, simulate payloads correct); **not yet run on a real Go2**. diff --git a/README.md b/README.md index 7ae58f5..9b71e75 100644 --- a/README.md +++ b/README.md @@ -1,190 +1,246 @@ # Sanad Fleet Agents -On-robot agents that report each robot's state to the **YS Lootah fleet server**. -Every robot type ships as its own self-contained **Docker** image with a -user-level **systemd auto-start service**, deployed, managed, and removed over -SSH by a single script. No `docker-compose` on the robot, **no `sudo`** required. +On-robot agents that report each Unitree robot's state to the **YS Lootah fleet +server**. One agent per robot type (**G1 · R1 · Go2**), each a self-contained +**Docker** image with a user-level **systemd auto-start service**, deployed and +managed over SSH by a single script — **no `docker-compose`, no `sudo`** on the robot. -![Sanad Fleet — architecture and data pipeline](diagram.svg) +![Sanad Fleet — one agent, five endpoints](diagram.svg) -> For the end-to-end data flow (topics → agent → HTTP → server), see -> [PIPELINE.md](PIPELINE.md). +> Full topic-by-topic data flow and payload schema: **[PIPELINE.md](PIPELINE.md)**. + +--- + +## At a glance + +- **One agent = one service = everything.** Each robot runs a single container that + streams telemetry **and** uploads maps, ships logs, raises alerts, and registers + its dashboard — across **five endpoints**. +- **Read-only toward the robot.** It reads DDS + files; it **never commands motion**. +- **Outbound HTTPS only.** No inbound port is opened on the robot; every request + carries `Authorization: Bearer ` and is keyed by the robot's serial (`sn`). +- **One source of truth.** `agents/g1/` is canonical; `r1` and `go2` are **generated** + from it (`tools/gen_agents.py`). You never hand-edit r1/go2, and never edit on the robot. + +| endpoint | when | what | +|---|---|---| +| `POST …/ingest/telemetry` | every ~2 s | 27 fields — identity, software/firmware, battery(+detail), motor temps, storage, status, position, **control mode**, faults, + status of map/logs/alerts/remote | +| `POST …/ingest/{sn}/map` | on change | saved nav map → PNG + resolution + origin (also RTAB-Map `.db`) | +| `POST …/ingest/{sn}/alert` | on new fault | battery ≤ 50 %, Gemini-billing, any error (as strings) | +| `POST …/ingest/{sn}/logs` | every 60 s | the agent's own lines **+ the robot's Sanad app logs** | +| `POST …/ingest/{sn}/remote` | every 60 s | dashboard URL (`web`) + `ssh unitree@` (`ssh`) | + +*(A `GET …/ingest/{sn}/commands` channel also exists; remote mode-**switching** is +deliberately **not** built — status only. See [§2](#2-the-agent).)* --- ## Table of contents -1. [What this is](#1-what-this-is) -2. [Directory layout](#2-directory-layout) -3. [The three agents](#3-the-three-agents) -4. [Prerequisites](#4-prerequisites) -5. [Quick start](#5-quick-start) -6. [Installer reference](#6-installer-reference) -7. [The auto-start service (systemd)](#7-the-auto-start-service-systemd) -8. [Configuration reference](#8-configuration-reference) -9. [Per-robot specifics](#9-per-robot-specifics) -10. [The fleet test server](#10-the-fleet-test-server) -11. [How a deploy works internally](#11-how-a-deploy-works-internally) -12. [Fleet inventory](#12-fleet-inventory) -13. [Troubleshooting](#13-troubleshooting) -14. [Security notes](#14-security-notes) +1. [Directory layout](#1-directory-layout) +2. [The agent](#2-the-agent) +3. [What it reports (the 27 fields)](#3-what-it-reports-the-27-fields) +4. [Maps: discovery & upload](#4-maps-discovery--upload) +5. [Prerequisites](#5-prerequisites) +6. [Quick start](#6-quick-start) +7. [How a deploy works](#7-how-a-deploy-works) +8. [Installer reference](#8-installer-reference) +9. [The auto-start service (systemd)](#9-the-auto-start-service-systemd) +10. [Configuration reference](#10-configuration-reference) +11. [Per-robot specifics](#11-per-robot-specifics) +12. [The fleet test server](#12-the-fleet-test-server) +13. [Fleet inventory](#13-fleet-inventory) +14. [Troubleshooting](#14-troubleshooting) +15. [Security notes](#15-security-notes) --- -## 1. What this is - -The fleet server (built by YS Lootah — "their side") needs each robot to push its -status outbound over HTTPS. This repo is **the robot side**: a small agent per -robot type that reads local state and POSTs it to documented endpoints, plus the -tooling to install/manage it across the fleet. - -- **G1** → uploads the **navigation map**. -- **R1** / **Go2** → stream **telemetry** (battery, charging, status, position, faults). - -Each is a drop-in Docker image; install it on any new robot of that type and it -just works. - ---- - -## 2. Directory layout +## 1. Directory layout ``` Project/fleet/ ├── README.md ← this file -├── PIPELINE.md ← end-to-end data-flow reference +├── PIPELINE.md ← end-to-end data-flow + payload schema +├── diagram.svg deploy.svg ← the two diagrams in this doc ├── fleet_install.sh ← deploy / manage / remove over SSH (interactive + scriptable) ├── fleet_test_server.py ← workstation stand-in for the fleet server (for tests) +├── tools/ +│ └── gen_agents.py ← regenerates r1 + go2 from the canonical g1 +├── tests/ ← mock capture + one-shot map export helpers └── agents/ - ├── g1/ ← MAP uploader + ├── g1/ ← CANONICAL source (edit here) │ ├── sanad_api_g1.py │ ├── Dockerfile (lean: python + requests) - │ ├── requirements.txt - │ ├── docker-compose.yml (manual local use only) - │ └── .env.example - ├── r1/ ← TELEMETRY (unitree_hg) - │ ├── sanad_api_r1.py - │ ├── Dockerfile (DDS: CycloneDDS + unitree_sdk2py) - │ ├── vendor/ (unitree_sdk2py wheel + crc libs) - │ └── .env.example - └── go2/ ← TELEMETRY (unitree_go) [unverified on hardware] - ├── sanad_api_go2.py - ├── Dockerfile - ├── vendor/ - └── .env.example + │ ├── requirements.txt · docker-compose.yml (local use only) · .env.example + ├── r1/ ← generated · unitree_hg · R1 FSM ids · eth10 + │ ├── sanad_api_r1.py · Dockerfile (DDS: CycloneDDS + unitree_sdk2py) · vendor/ · .env.example + └── go2/ ← generated · unitree_go · ⚠ unverified on hardware + └── sanad_api_go2.py · Dockerfile · vendor/ · .env.example ``` -`agents/` is the **single source of truth**. The installer rsyncs the selected -agent to the robot; you never edit files on the robot. +> **Golden rule:** edit `agents/g1/`, then run `python3 tools/gen_agents.py` to +> regenerate `r1` and `go2`. The installer rsyncs the selected agent to the robot; +> files on the robot are never edited by hand. --- -## 3. The three agents +## 2. The agent | agent (type) | robot | DDS family | notes | |---|---|---|---| -| `sanad_api_g1` (`g1`) | Unitree G1 | `unitree_hg` | canonical source (see below) | -| `sanad_api_r1` (`r1`) | Unitree R1 EDU | `unitree_hg` | generated from g1; R1 FSM ids, `eth10` | -| `sanad_api_go2` (`go2`) | Unitree Go2 | `unitree_go` | generated from g1; battery nested in `LowState.bms_state` — ⚠ unverified on hardware | +| `sanad_api_g1` (`g1`) | Unitree G1 | `unitree_hg` | **canonical source** | +| `sanad_api_r1` (`r1`) | Unitree R1 EDU | `unitree_hg` | generated from g1 · R1 FSM ids `0/1/4/811` · `eth10` | +| `sanad_api_go2` (`go2`) | Unitree Go2 | `unitree_go` | generated from g1 · battery nested in `LowState.bms_state` · ⚠ unverified on hardware | -**One agent = one full-feature service.** Each robot runs a single container that -does everything, across five endpoints: - -- **telemetry** (`…/telemetry`, ~2 s) — 27 fields: identity, `software`, `firmware`, - battery(+detail), motor temps, storage, status, position, `control` (loco mode), - faults, and status mirrors for map/logs/alerts/remote + full timing. -- **map** (`…/{sn}/map`, on change) — the Sanad/SLAM saved map, uploaded ONCE per - content. Formats: **RTAB-Map `.db`** (skipped over the ~8 MB server cap) and - **slam_toolbox** `pgm/yaml` → image JSON (PNG + resolution + origin). Status shown - in telemetry `map`. -- **alerts** (`…/{sn}/alert`) — each NEW fault, immediately (strings — the ingest - 500s on fault objects). -- **logs** (`…/{sn}/logs`, 60 s) — the agent's own lines **+ the robot's Sanad app - logs** (`[sanadr1-logs]`, auto-discovered, backfilled, noise-filtered). -- **remote** (`…/{sn}/remote`, 60 s) — registers the Sanad **dashboard URL** (`web`) - and **`ssh unitree@`** (`ssh`) — no changes to the Sanad app. - -**All three agents are generated from `g1`** by `tools/gen_agents.py` — run it after -any change to `agents/g1/` to keep r1/go2 in lockstep (no manual editing of r1/go2). - -All requests carry `Authorization: Bearer `. Full payload schema and -pipeline in [PIPELINE.md](PIPELINE.md). +All three run the **same five loops** (telemetry 2 s · map 30 s · alert scan 10 s · +log ship 60 s · remote register 60 s). They differ only in the DDS layer and a few +robot-specific ids — which is exactly what the generator handles. **Design principles** -- **No ROS.** Maps read from files; state read from DDS via `unitree_sdk2py`. -- **Read-only toward the robot.** Never commands motion — the control panel is - status-only; mode switching is deliberately not built (it can drop the robot). +- **No ROS.** Maps are read from files; robot state from DDS via `unitree_sdk2py`. +- **Read-only.** The control panel reports the loco mode but **cannot switch it** — + mode switching can drop the robot, so it is intentionally not built (`CONTROL_ENABLE=0`). - **Never crash the loop.** Every tick is wrapped; a heartbeat keeps the robot - "online" when state is unreadable. -- **Change-detected + one-time uploads.** Maps re-send only on content change. + "online" when state is momentarily unreadable. +- **Idempotent uploads.** Maps re-send only when their content changes. - **Resilient shipping.** Failed log ships are requeued (last ~400 lines) until the server accepts them. --- -## 4. Prerequisites +## 3. What it reports (the 27 fields) -**Workstation (the machine you deploy from):** +Every telemetry POST is one JSON object. Grouped for readability: + +| group | fields | +|---|---| +| **identity** | `sn` · `name` · `mac` · `brand` · `type` · `model` | +| **system** | `software{ros,os,os_version,kernel,arch,python,agent}` · `firmware{board,l4t,kernel,robot,bms}` | +| **power** | `battery` · `charging` · `battery_detail{voltage_v,current_a,temp_c,soh,cycles}` | +| **health** | `motor_temp{max,avg,min}` *(null = not receiving)* · `storage{total_gb,free_gb,used_percent}` · `faults[]` *(strings)* | +| **state** | `status` *(idle/moving/charging/offline)* · `position{x,y}` · `control{fsm_id,mode,armed,walk_ready,teleop_active,…}` | +| **sub-status** | `map{…}` · `logs{…}` · `project_logs` · `remote{…}` · `alerts{…}` | +| **timing** | `time` · `started_at` · `last_start` · `uptime_s` · `ts` | + +`control.mode` is a friendly label of the loco FSM (R1: `zero_torque`/`damp`/`lock`/ +`running`; G1: `running`/`lock`/`squat`/…), read from the Sanad dashboard status API +— no DDS command, no motion. Full schema in **[PIPELINE.md](PIPELINE.md)**. + +--- + +## 4. Maps: discovery & upload + +The agent finds saved maps on the robot and uploads each **once per content change**. +It understands two formats: + +- **`slam_toolbox` / Nav2 / Pudu sets** — `map.pgm` + `map.yaml` → rendered to an + image JSON (pure-stdlib PGM→PNG + resolution + origin). Small; always uploads. +- **RTAB-Map `.db`** — sent as-is, but **skipped if larger than the server's ~8 MB + cap** (`MAP_MAX_UPLOAD_MB`, default 7) with a note in `map.error`. + +**Where it looks** (each existing dir is scanned for `*.pgm` + `*.yaml` and `*.db`): + +- `MAPS_DIR` (`/data/maps`) — the robot's primary SLAM/nav maps dir (installer probes it). +- `EXTRA_MAP_DIRS` (`/data/nav2_maps`) — a **separate Nav2/Pudu deploy-maps dir** + (e.g. `~/r1_nav2_docker/maps`) mounted alongside. The installer auto-detects and + mounts it, so converted **Pudu office maps** are discovered and uploaded too. + +Two conveniences: + +- The Pudu converter emits a plain map **and** a keepout-**baked** twin; when both + exist the agent keeps only the baked one, so the server gets **one** canonical map. +- `--map-only` uploads discovered maps once with **no DDS and no telemetry** — handy + to push a new map without disturbing a live feed: + ```bash + docker run --rm --network host --env-file .env \ + -e EXTRA_MAP_DIRS=/data/nav2_maps -v :/data/nav2_maps:ro \ + sanad-api-:latest --map-only --force + ``` + +Map status is mirrored into every telemetry post as +`map:{uploaded,state,maps_found,last_map,error,checked_ts}`. + +--- + +## 5. Prerequisites + +**Workstation (deploy host)** - `bash`, `ssh`, `rsync`, `python3`. -- SSH **key** access to each robot (`ssh unitree@` must work without a - password — the installer uses `BatchMode=yes`). -- On the same network as the robots (they POST back to the workstation during - `test`). +- SSH **key** access to each robot — `ssh unitree@` must work without a password + (the installer uses `BatchMode=yes`). +- On the same network as the robots (they POST back to the workstation during `test`). -**Robot:** -- Docker (Engine ≥ 20). The `unitree` user must be in the `docker` group. -- Internet at build time (pulls the base image; R1/Go2 also `apt`/`pip` the DDS - stack). -- `systemd` with a user bus (standard on Ubuntu 20.04+). **No sudo needed.** +**Robot** +- Docker (Engine ≥ 20); the `unitree` user in the `docker` group. +- Internet at build time (base image; r1/go2 also build the DDS stack). +- `systemd` **user** bus (standard on Ubuntu 20.04+). **No sudo needed.** - Architecture: arm64 (Jetson / backpack). Images build natively on the robot. --- -## 5. Quick start +## 6. Quick start ```bash cd Project/fleet -# Interactive — asks robot type, IP, and (if new) robot name + server: +# Interactive — asks robot type, IP, and (if new) name + token + server: ./fleet_install.sh -# …or scripted: -./fleet_install.sh install r1 10.255.254.82 --sn E39N4000Q6D7E70F --name r1_82 \ - --server-ip 10.255.254.83 --port 8799 --token +# …or scripted against the real fleet server: +./fleet_install.sh install r1 10.255.254.82 \ + --sn E39N4000Q6D7E70F --name r1_82 \ + --token --server-url https://eco.yslootahrobotics.com -# See what it's sending, tail logs, check the service: -./fleet_install.sh data r1 10.255.254.82 -./fleet_install.sh logs r1 10.255.254.82 -./fleet_install.sh status r1 10.255.254.82 - -# End-to-end test against your workstation acting as the server: -./fleet_install.sh test r1 10.255.254.82 - -# Remove everything: -./fleet_install.sh uninstall r1 10.255.254.82 +# Inspect / manage: +./fleet_install.sh data r1 10.255.254.82 # what it's currently sending +./fleet_install.sh logs r1 10.255.254.82 # live docker logs +./fleet_install.sh status r1 10.255.254.82 # systemd + container state +./fleet_install.sh test r1 10.255.254.82 # e2e against the workstation server +./fleet_install.sh uninstall r1 10.255.254.82 # remove everything ``` --- -## 6. Installer reference +## 7. How a deploy works -### Interactive mode (no arguments) +![Deploy — workstation to robot over SSH](deploy.svg) + +``` +rsync agents// → unitree@:~/sanad_api_/ (--delete; keeps .env + state) +write ~/sanad_api_/.env (SERVER_URL, TOKEN, SN, iface, …) +ssh: docker build -t sanad-api-:latest . (native arm64) +ssh: docker create --name sanad-api- (no docker restart policy) +write ~/.config/systemd/user/sanad-api-.service +ssh: loginctl enable-linger ; systemctl --user enable --now sanad-api- +``` + +**run-args** (all agents): `--network host --env-file …/.env` plus read-only mounts: +`-v /:/host:ro` (real disk/firmware + Sanad json-log), `-v :/data/maps:ro`, +`-v :/data/web_data:ro`, `-v :/data/nav2_maps:ro` *(when present)*, +and `-v …/state:/data/state` (rw, upload fingerprints). + +`--network host` is required so the robot's DDS multicast is visible (it doesn't cross +a NAT bridge) and so the agent can reach the Sanad dashboard on `127.0.0.1`. + +--- + +## 8. Installer reference + +### Interactive (no arguments) ``` ./fleet_install.sh ``` -Flow: 1. **Which robot?** `1) g1 2) r1 3) go2` -2. **Robot IP** and **SSH user** (default `unitree`; it verifies SSH works). -3. It **detects whether the agent is already installed** (systemd unit file *or* - container present) and branches: - - **Installed** → menu: `1) show data 2) status 3) logs 4) reinstall - 5) UNINSTALL 6) quit`. - - **Not installed** → prompts **Robot name (SN)**, **Fleet server IP** - (auto-detected default), **port**, **device token**, then installs. +2. **Robot IP** + **SSH user** (default `unitree`; verifies SSH works). +3. **Detects whether it's already installed** (unit file *or* container) and branches: + - **Installed** → menu: show data · status · logs · reinstall · **uninstall** · quit. + - **Not installed** → prompts **SN (required)**, **server** (URL or auto-detected IP), + **token**, then installs. -### Scriptable commands +### Scriptable ``` ./fleet_install.sh [options] @@ -192,49 +248,42 @@ Flow: | command | action | |---|---| -| `install` | rsync agent → robot, build image, create container, install + enable systemd service | -| `uninstall` | disable/remove service, remove container, image, and `~/sanad_api_` | +| `install` | rsync → build → create → install + enable the systemd service | +| `uninstall` | disable/remove service, container, image, and `~/sanad_api_` | | `status` | systemd service state + container state | -| `data` | recent telemetry/map log lines (what it's currently sending) | +| `data` | recent telemetry/map log lines (what it's sending) | | `logs` | `docker logs -f` (live tail) | -| `test` | start the workstation server, push a real post from the robot, verify receipt (PASS/FAIL) | +| `test` | start the workstation server, push a real post from the robot, verify (PASS/FAIL) | -### Options - -**`install` requires the essentials to be entered explicitly** — the robot's real -serial, the token, and the server: +`install` requires the essentials to be explicit: ```bash -./fleet_install.sh install --sn --token \ - --server-url https://eco.yslootahrobotics.com [--post ] [--name ] +./fleet_install.sh install --sn --token \ + --server-url https://eco.yslootahrobotics.com [--post ] [--name ] ``` | option | default | meaning | |---|---|---| -| `--sn SERIAL` | **required for install** | robot's REAL serial (keys the robot on the server), e.g. `E39N4000Q6D7E70F` | -| `--name NAME` | `_` | friendly display name (e.g. `r1_82`, `g1_58`) | +| `--sn SERIAL` | **required** | robot's REAL serial — keys it on the server (e.g. `E39N4000Q6D7E70F`) | +| `--name NAME` | `_` | friendly display name (`r1_82`, `g1_58`) | | `--token TOK` | `test-token` | device bearer token | -| `--server-url URL` | — | full fleet server URL (`https://…`) → `VERIFY_TLS=1` | -| `--post PATH` | agent default | ingest POST path (telemetry endpoint; map endpoint for `g1`) | -| `--server-ip IP` | auto (route toward robot) | alternative: local test server by IP (`VERIFY_TLS=0`) | +| `--server-url URL` | — | full HTTPS fleet server → `VERIFY_TLS=1` | +| `--post PATH` | agent default | override the telemetry ingest path | +| `--server-ip IP` | auto (route to robot) | use the local test server instead → `VERIFY_TLS=0` | | `--port N` | `8799` | test-server port (with `--server-ip`) | | `--user USER` | `unitree` | SSH user on the robot | | `--keep-server` | off | (test) leave the workstation test server running | -The interactive flow prompts for the same set: robot type → IP → **SN (required)** -→ display name → SERVER_URL-or-IP → token → POST endpoint. - --- -## 7. The auto-start service (systemd) +## 9. The auto-start service (systemd) -Because the robots have **no passwordless sudo**, the agent runs as a **user-level -systemd service** (no root needed): +Because the robots have **no passwordless sudo**, the agent runs as a **user-level** +systemd service (no root): -- Unit file: `~/.config/systemd/user/sanad-api-.service` -- Boot auto-start: enabled via `loginctl enable-linger ` (allowed without - sudo) so the user manager starts at boot before login. -- The service owns the container lifecycle: +- Unit: `~/.config/systemd/user/sanad-api-.service` +- Boot auto-start via `loginctl enable-linger ` (allowed without sudo). +- systemd owns the container lifecycle: ```ini [Service] Type=simple @@ -243,171 +292,161 @@ systemd service** (no root needed): ExecStart=/usr/bin/docker start -a sanad-api- ExecStop=/usr/bin/docker stop -t 10 sanad-api- ``` -- The container is created with `docker create` (no docker restart policy) so - **systemd is the single owner** — `Restart=always` also covers the boot race - where the user manager starts before `dockerd` is ready (it retries every 5 s). - -**Managing it on the robot:** +- Created with `docker create` (no docker restart policy) so **systemd is the single + owner**; `Restart=always` also covers the boot race where the user manager starts + before `dockerd` is ready. +**On the robot:** ```bash systemctl --user status sanad-api-r1 systemctl --user restart sanad-api-r1 -systemctl --user stop sanad-api-r1 journalctl --user -u sanad-api-r1 -f # or: docker logs -f sanad-api-r1 ``` --- -## 8. Configuration reference +## 10. Configuration reference -Config is env-only. The installer writes `~/sanad_api_/.env` on the robot; -`.env.example` in each agent dir documents every key. +Config is env-only; the installer writes `~/sanad_api_/.env`. Every key is +documented in each agent's `.env.example`. -### Common (all agents) - -| var | required | default | meaning | -|---|---|---|---| -| `SERVER_URL` | ✅ | — | fleet server base URL | -| `DEVICE_TOKEN` | ✅ | — | bearer token (per robot) | -| `SN` | — | `_0000` | robot fleet id | -| `VERIFY_TLS` | — | `1` | verify server TLS cert (`0` for self-signed dev) | -| `HTTP_TIMEOUT` | — | 10–30 | per-request timeout (s) | -| `POLL_INTERVAL` | — | 2 (tel) / 30 (map) | loop cadence (s) | - -### G1 map uploader +### Core (all agents) | var | default | meaning | |---|---|---| -| `ROBOT` | `sanad` | web_nav3 robot name → maps subdir + `X-Robot-Name` | -| `MAPS_DIR` | `/data/maps` | mounted web_nav3 `maps/` (`/*.db`) | -| `DATA_DIR` | — | mounted web_nav3 `web/data/` (per-map places) | -| `LEGACY_PLACES` | — | optional legacy `places.json` | -| `WEB_NAV3_URL` | — | optional `http://127.0.0.1:8765` (to learn the active map) | -| `MAP_SELECT` | `all` | `all` · `active` · `newest` | -| `MAP_UPLOAD_MODE` | `multipart` | `multipart` · `base64json` | -| `MAP_ENDPOINT` | `/api/v1/fleet/ingest/{sn}/map` | path template | -| `STATE_DIR` | `/data/state` | upload-fingerprint state | +| `SERVER_URL` ✅ | — | fleet server base URL | +| `DEVICE_TOKEN` ✅ | — | bearer token (per robot) | +| `SN` | `_0000` | robot's real serial (server key) | +| `ROBOT_NAME` | `_` | friendly display name | +| `ROBOT_BRAND` / `ROBOT_TYPE` / `ROBOT_MODEL` | `unitree` / `humanoid`\|`dog` / `g1`\|`r1`\|`go2` | identity | +| `POLL_INTERVAL` | `2` | telemetry cadence (s) | +| `VERIFY_TLS` | `1` | verify server TLS (`0` for the local test server) | +| `HTTP_TIMEOUT` | `30` | per-request timeout (s) | +| `SOFTWARE_ROS` | `foxy` | reported in `software.ros` | -### R1 / Go2 telemetry +### DDS / telemetry | var | default | meaning | |---|---|---| -| `DDS_INTERFACE` | R1 `eth10` / Go2 `eth0` | NIC that sees robot DDS | +| `DDS_INTERFACE` | r1 `eth10` · else `eth0` | NIC that sees robot DDS | | `DDS_DOMAIN` | `0` | DDS domain id | | `MAC_INTERFACE` | = `DDS_INTERFACE` | NIC whose MAC is reported | -| `R1_READ_FSM` | `0` | (R1) read loco FSM for status — read-only GET RPC | -| `R1_POSITION_SOURCE` / `GO2_POSITION_SOURCE` | `none` | `none` · `rosbridge` (· `sportmode` Go2) | -| `ROSBRIDGE_URL` | `ws://127.0.0.1:9090` | position source when rosbridge | -| `LOW_SOC` | `15` | %→ `LOW_BATTERY` fault | -| `MOTOR_TEMP_MAX` | `85` | °C→ `MOTOR_OVERTEMP` fault | -| `TELEMETRY_ENDPOINT` | `/api/v1/fleet/ingest/telemetry` | path | +| `G1_READ_FSM` | `0` | read loco FSM for status (read-only GET RPC) | +| `POSITION_SOURCE` | `odom`\|`none` | `odom` (DDS) · `rosbridge` · `none` | +| `ROSBRIDGE_URL` | `ws://127.0.0.1:9090` | position source when `rosbridge` | +| `MOTOR_TEMP_MAX` | `85` | °C → `MOTOR_OVERTEMP` fault | + +### Maps + +| var | default | meaning | +|---|---|---| +| `MAPS_DIR` | `/data/maps` | primary SLAM/nav maps mount | +| `EXTRA_MAP_DIRS` | `/data/nav2_maps` | extra roots (colon-sep) for Nav2/Pudu `pgm+yaml` sets | +| `MAP_SELECT` | `all` | `all` · `active` · `newest` | +| `MAP_POLL_INTERVAL` | `30` | map check cadence (s) | +| `MAP_MAX_UPLOAD_MB` | `7` | skip `.db` larger than this (server cap) | +| `MAP_UPLOAD_MODE` | `multipart` | `multipart` · `base64json` | +| `DATA_DIR` | `/data/web_data` | per-map places (waypoints) | +| `STATE_DIR` | `/data/state` | upload-fingerprint + start-time state | + +### Alerts + +| var | default | meaning | +|---|---|---| +| `LOW_SOC` | `50` | battery % → `LOW_BATTERY` fault + alert | +| `ALERT_SCAN_INTERVAL` | `10` | project-log scan cadence (s) | +| `ALERT_LOG_COOLDOWN` | `300` | per-signature re-alert gap (s) | +| `ALERT_LOG_PATTERNS` | Gemini-billing + error/critical/traceback | `CODE=regex` entries split by `;;` (case-sensitive; use `(?i)` inline) | +| `ALERT_BACKFILL_BYTES` | `8 MiB` | startup scan window so active errors alert on boot | + +### Logs · remote · control + +| var | default | meaning | +|---|---|---| +| `LOGS_INTERVAL` | `60` | log-ship cadence (s) | +| `PROJECT_LOG_CONTAINER` | `auto` | robot's Sanad container to tail (`auto` discovers it) | +| `PROJECT_LOG_PATH` / `_LABEL` / `_BACKFILL` / `_EXCLUDE` | — / — / `100` / access-log filter | project-log overrides | +| `REMOTE_ENABLE` | `1` | register the dashboard URL + ssh | +| `REMOTE_PORTS` | `8001,8014,8011-8013,8000,8080` | dashboard ports to probe | +| `REMOTE_URL` | — | pin an explicit URL (e.g. a public tunnel) | +| `SSH_REGISTER` / `SSH_USER` | `1` / `unitree` | register `ssh @` | +| `CONTROL_ENABLE` | `0` | remote mode-**switch** (motion) — **off by design** | +| `CONTROL_STATUS_URL` | — | Sanad `/api/controller/status` for read-only loco mode | --- -## 9. Per-robot specifics +## 11. Per-robot specifics -### G1 (map) — real maps live inside the nav container +### R1 (live on production) +- DDS link is **`eth10`** (= `192.168.123.164`); WiFi is the fleet LAN. +- Battery from `rt/lf/bmsstate` (`BmsState_`); coexists with the running `sanadr1` + app (DDS allows many readers). +- Maps: VSLAM `~/r1_vslam_docker/data` (RTAB-Map `.db`) **and** the converted + **Pudu office map** in `~/r1_nav2_docker/maps` (mounted at `/data/nav2_maps`). -The G1's saved maps are stored **inside** the `p4_Foxy_sanad` (Package_4 nav) -container at `/home/unitree/marcus_nav2_test/maps`, which Package_4 does **not** -bind-mount to the host. So the uploader (which mounts a host dir) sees **0 maps** -until you do one of: +### G1 (⚠ STANDING HOLD — stage only, do not deploy) +- Do **not** deploy to the new G1 until the hold is lifted; stage maps only. +- Saved maps live **inside** the `p4_Foxy_sanad` nav container at + `~/marcus_nav2_test/maps` (not host-mounted by default) — add a host bind-mount + (and to `web/data/`) so the agent can read them, or use a shared volume. -1. Add a host bind-mount for `maps/` (and `web/data/`) to the Package_4 `nav` - service, then set the uploader's `MAPS_HOST_DIR`/`DATA_HOST_DIR` to those paths. -2. Put both containers on a shared named volume for the maps dir. - -The `test` command seeds a throwaway fixture map so the upload path is verified -regardless. - -### R1 (telemetry) - -- DDS link is **`eth10`** (= `192.168.123.164`); `wlan0` is the fleet LAN. -- Battery: `rt/lf/bmsstate` (`BmsState_`), `soc` 0–100, charging from `current` - sign. Coexists fine with the running `sanadr1` app (DDS allows many readers). - -### Go2 (telemetry) — ⚠️ unverified on hardware - -- Uses `unitree_go` DDS. Battery is nested in **`rt/lowstate.bms_state`** (Go2 has - no separate BMS topic). -- Written from the SDK layout; image builds and `unitree_go` imports, simulate is - correct, but **not yet run on a real Go2** — confirm the `bms_state` current sign - (charging polarity) and `sportmodestate` fields on the robot. +### Go2 (⚠ unverified on hardware) +- Uses `unitree_go`; battery is nested in **`rt/lowstate.bms_state`** (no separate + BMS topic). Image builds and imports, simulate is correct — but confirm the + `bms_state` current sign (charge polarity) and `sportmodestate` fields on a real Go2. --- -## 10. The fleet test server +## 12. The fleet test server -`fleet_test_server.py` stands in for the real fleet server so you can verify a -deploy end-to-end from your workstation. +`fleet_test_server.py` stands in for the real fleet server so you can verify a deploy +end-to-end from the workstation. ```bash PORT=8799 REQLOG=/tmp/fleet.jsonl python3 fleet_test_server.py ``` -- Binds `0.0.0.0:8799`. -- `GET /ping` → `200` (reachability check the installer uses). -- Accepts `POST …/map` and `POST …/telemetry`; logs each request to `REQLOG` and - prints a live summary. +- Binds `0.0.0.0:8799`; `GET /ping` → `200` (the reachability check the installer uses). +- Accepts every ingest POST, logs each to `REQLOG`, prints a live summary. -`./fleet_install.sh test …` starts it automatically, triggers a post from the -robot, asserts the payload arrived, and prints `PASS`/`FAIL`. +`./fleet_install.sh test …` starts it, triggers a post from the robot, asserts the +payload arrived, and prints **PASS**/**FAIL**. --- -## 11. How a deploy works internally +## 13. Fleet inventory -``` -install: - rsync agents// → unitree@:~/sanad_api_/ (--delete, minus .env/state) - write ~/sanad_api_/.env (SERVER_URL, TOKEN, SN, iface…) - ssh: docker build -t sanad-api-:latest . (native arm64) - ssh: docker create --name sanad-api- … (no docker restart policy) - write ~/.config/systemd/user/sanad-api-.service - ssh: loginctl enable-linger ; systemctl --user enable --now … - -run-args: --network host --env-file …/.env - (g1 also: -v maps:ro -v web_data:ro -v state) -``` - -`--network host` is required: G1 to reach `127.0.0.1:8765` (web_nav3) if used, and -R1/Go2 so the robot's DDS traffic is visible (DDS multicast doesn't cross a NAT -bridge). - ---- - -## 12. Fleet inventory - -| robot | agent(s) | IP | SSH | arch | DDS iface | SN | +| robot | IP | SSH | arch | DDS iface | SN (name) | status | |---|---|---|---|---|---|---| -| G1 | telemetry + map (`g1`) | `10.255.254.58` | `unitree` (key) | arm64 | `eth0` | `E21D6000PB89GF88` (name `g1_58`) | -| R1 | telemetry (`r1`) | `10.255.254.82` | `unitree` (key) | arm64 | `eth10` | `E39N4000Q6D7E70F` (name `r1_82`) | -| Go2 | telemetry (`go2`) | *(TBD)* | `unitree` | arm64 | `eth0` | — | +| R1 | `10.255.254.82` (wired `192.168.123.164`) | `unitree` (key) | arm64 | `eth10` | `E39N4000Q6D7E70F` (`r1_82`) | **live on prod** · office map uploaded | +| G1 | `10.255.254.58` | `unitree` (key) | arm64 | `eth0` | `E21D6000PB89GF88` (`g1_58`) | ⚠ hold + offline (battery) | +| Go2 | *(TBD)* | `unitree` | arm64 | `eth0` | — | unverified | Workstation (deploy host + test fleet server): **`10.255.254.83`** (`wlp4s0`). +Production fleet server: **`https://eco.yslootahrobotics.com`**. --- -## 13. Troubleshooting +## 14. Troubleshooting | symptom | cause / fix | |---|---| -| `robot cannot reach the workstation server (http 000)` during `test` | robot→workstation blocked. Check both are on the same subnet; open the port if a firewall is on (`sudo ufw allow 8799`). | -| G1 uploads nothing / "0 maps" | no saved map, or maps aren't mounted — see [§9 G1](#g1-map--real-maps-live-inside-the-nav-container). | -| R1/Go2 `telemetry POST failed … Connection refused` | the target server (e.g. the test server) is down. Point at the real server: `install … --server-ip --token `. | -| R1/Go2 `battery=null, status=offline` forever | DDS not seen. Wrong `DDS_INTERFACE` (R1 = `eth10`) or robot firmware down. Confirm with `ip -o addr` on the robot. | -| build is very slow (R1/Go2) | first build compiles CycloneDDS (minutes). Run detached and poll: `ssh … 'setsid bash -c "cd ~/sanad_api_r1 && docker build -t sanad-api-r1:latest . > build.log 2>&1" | grep Linger` → `Linger=yes`. The installer sets it. | -| `unbound variable` / weird prompt behavior | ensure you're on the current `fleet_install.sh` (uses `ssh -n` for command calls so SSH doesn't eat stdin). | +| `test`: robot can't reach the workstation (`http 000`) | same-subnet? open the port (`sudo ufw allow 8799`). | +| map: "0 maps" / nothing uploaded | no saved map, or maps aren't mounted — see [§4](#4-maps-discovery--upload) and [§11 G1](#g1--standing-hold--stage-only-do-not-deploy). | +| `telemetry POST failed … Connection refused` | target server down / wrong URL. Re-point: `install … --server-url `. | +| `battery=null, status=offline` forever | DDS not seen. Wrong `DDS_INTERFACE` (R1 = `eth10`) or robot firmware down — check `ip -o addr` on the robot. | +| build very slow (r1/go2) | first build compiles CycloneDDS (minutes). Run detached: `setsid bash -c 'cd ~/sanad_api_r1 && docker build -t sanad-api-r1:latest . >build.log 2>&1' \| grep Linger` → `Linger=yes` (the installer sets it). | +| r1/go2 out of sync with g1 | you edited g1 without regenerating — run `python3 tools/gen_agents.py`. | --- -## 14. Security notes +## 15. Security notes -- **Outbound only.** Agents open no inbound ports on the robot; they POST out over - HTTPS with a per-robot bearer token. -- **Least privilege.** Runs as the `unitree` user (docker group), user-level - systemd, no root. Read-only w.r.t. the robot (no motion commands). -- **Token handling.** `.env` holds the device token and is never rsynced back or - committed (`.gitignore`d). The installer writes it directly to the robot. +- **Outbound only.** No inbound port on the robot; POSTs out over HTTPS with a + per-robot bearer token. +- **Least privilege.** Runs as `unitree` (docker group), user-level systemd, no root. + Read-only toward the robot — **no motion commands**. +- **Token handling.** `.env` holds the device token, is never rsynced back, and is + `.gitignore`d. The installer writes it straight to the robot; it is never printed. - **TLS.** `VERIFY_TLS=1` in production; `0` only for the local test server. diff --git a/agents/g1/.env.example b/agents/g1/.env.example index 82d22b7..3ad6d1c 100644 --- a/agents/g1/.env.example +++ b/agents/g1/.env.example @@ -1,48 +1,74 @@ # sanad_api_g1 — copy to .env and fill in. docker compose reads it automatically. +# ONE agent = telemetry + map + logs + alerts + remote (see fleet/README.md). # ── fleet server (REQUIRED — YS Lootah gives you these two) ────────────────── SERVER_URL=https://fleet.example.com DEVICE_TOKEN=REPLACE_WITH_DEVICE_TOKEN # ── identity ───────────────────────────────────────────────────────────────── -# This robot's fleet id — used as {sn} in POST /api/v1/fleet/ingest/{sn}/map. -SN=g1_7892 +# The robot's REAL Unitree serial — keys the robot on the server ({sn} routes). +SN=E21D6000PB89GF88 +# Friendly display name shown on the dashboard (default _). +ROBOT_NAME=g1_58 +ROBOT_BRAND=unitree +ROBOT_TYPE=humanoid +ROBOT_MODEL=g1 # web_nav3 robot name: the maps subdir (//*.db) + X-Robot-Name. -# Must match web_nav3's robot_config.yaml robot_name (default: sanad). ROBOT=sanad +# Optional: Sanad data dir whose size is reported in storage (as /host/ +# when the installer's read-only /:/host mount is used). Empty = omit. +STORAGE_DATA_PATH= -# ── where the map files live on the HOST (bind-mounted read-only) ──────────── -# Point these at the robot's web_nav3 install. Defaults assume a Package_4 -# robot (nav container -> /home/unitree/marcus_nav2_test). For the workstation -# dev copy use .../Project/G1/Nav2_Projects/web_nav3/{maps,web/data}. -MAPS_HOST_DIR=/home/unitree/marcus_nav2_test/maps -DATA_HOST_DIR=/home/unitree/marcus_nav2_test/web/data -STATE_HOST_DIR=./data/state +# ── DDS (reading robot state) ──────────────────────────────────────────────── +# Use the interface that sees the robot's DDS traffic (G1 backpack: eth0). +DDS_INTERFACE=eth0 +DDS_DOMAIN=0 +# Which NIC's MAC to report as the robot identity (defaults to DDS_INTERFACE). +# MAC_INTERFACE=eth0 -# Optional legacy per-robot places file (older sanad setups). Leave blank if -# each map already has its own places under DATA_DIR//places/. -# LEGACY_PLACES=/data/legacy/places.json +# ── status / position options ──────────────────────────────────────────────── +# Read the loco FSM for a richer status (READ-ONLY GET RPC, never commands +# motion). 0 = derive status from battery + joint motion only (safe default). +G1_READ_FSM=0 +# Position source: odom (rt/lf/odommodestate over DDS) | rosbridge | none. +G1_POSITION_SOURCE=odom +ROSBRIDGE_URL=ws://127.0.0.1:9090 -# ── behaviour ──────────────────────────────────────────────────────────────── -# Which maps to upload: all | active | newest. active/newest need WEB_NAV3_URL. +# ── fault thresholds ───────────────────────────────────────────────────────── +LOW_SOC=50 +MOTOR_TEMP_MAX=85 + +# ── cadence / transport ────────────────────────────────────────────────────── +POLL_INTERVAL=2 +VERIFY_TLS=1 +HTTP_TIMEOUT=10 + +# ── map sync (uploaded ONCE per content; status shown in telemetry "map") ──── +# The installer mounts the robot's real maps dir at /data/maps (probed on the +# robot: rtabmap*.db / slam_toolbox yaml+pgm sets). Fallback = agent's own dir. +MAPS_DIR=/data/maps +# Extra map roots (colon-separated container paths) — Nav2/Pudu deploy maps +# (map.pgm + map.yaml). The installer mounts the robot's *_nav2_docker/maps +# here; safe to leave set even when nothing is mounted (missing dirs skipped). +EXTRA_MAP_DIRS=/data/nav2_maps +DATA_DIR=/data/web_data +STATE_DIR=/data/state +# Which maps to upload: all | active | newest. active needs WEB_NAV3_URL. MAP_SELECT=all # Optional — only used to learn which map is ACTIVE (MAP_SELECT=active). WEB_NAV3_URL=http://127.0.0.1:8765 - -# Wire format the server accepts on the map endpoint: -# multipart -> multipart/form-data: file field `db` + form field `meta` (JSON) -# base64json -> JSON body with the .db as `db_base64` + the same meta fields +# Optional legacy per-robot places file (older sanad setups). Leave blank if +# each map already has its own places under DATA_DIR//places/. +# LEGACY_PLACES=/data/legacy/places.json MAP_UPLOAD_MODE=multipart -# Endpoint path template ({sn} substituted). Change only if the server differs. -MAP_ENDPOINT=/api/v1/fleet/ingest/{sn}/map - -# Scan cadence for the loop (seconds). Maps rarely change, so 30s is plenty. -POLL_INTERVAL=30 -# Verify the fleet server's TLS cert (1 recommended; 0 only for self-signed dev). -VERIFY_TLS=1 -HTTP_TIMEOUT=30 +MAP_POLL_INTERVAL=30 +# Skip .db maps larger than this (the prod server 413s at ~8 MB). Raster +# (pgm+yaml) maps are tiny and never skipped. +MAP_MAX_UPLOAD_MB=7 # ── logs + alerts ───────────────────────────────────────────────────────────── +# agent log lines shipped to /{sn}/logs every LOGS_INTERVAL; new faults POSTed +# to /{sn}/alert immediately (rising edge). LOGS_INTERVAL=60 # ── project logs (shared alongside agent logs) ─────────────────────────────── @@ -66,7 +92,7 @@ REMOTE_INTERVAL=60 # ── control panel (READ-ONLY loco mode; remote SWITCH is off by design) ─────── # The agent reads the Sanad dashboard's /api/controller/status (no DDS, no -# motion) and reports the current mode (zero_torque/damp/lock/running) + the +# motion) and reports the current mode (G1: running/lock/squat/…) + the # switchable set in telemetry.control. CONTROL_ENABLE stays 0: remote mode- # SWITCHING is deliberately NOT built (it commands motion and can drop the robot; # switching is done in person on the Sanad dashboard). Leave URL blank to use the diff --git a/agents/g1/sanad_api_g1.py b/agents/g1/sanad_api_g1.py index 9644d8f..5261c2d 100644 --- a/agents/g1/sanad_api_g1.py +++ b/agents/g1/sanad_api_g1.py @@ -131,6 +131,7 @@ class Config: # map sync robot: str maps_dir: Path + extra_map_dirs: List[Path] # extra roots to scan for pgm+yaml sets (e.g. Nav2/Pudu maps) web_data_dir: Optional[Path] legacy_places: Optional[Path] web_nav3_url: str @@ -210,6 +211,9 @@ class Config: telemetry_endpoint=_env("TELEMETRY_ENDPOINT", "/api/v1/fleet/ingest/telemetry"), robot=_env("ROBOT", "sanad"), maps_dir=Path(_env("MAPS_DIR", "/data/maps")), + # colon-separated extra roots (mounted Nav2/Pudu map dirs). Any *.yaml+*.pgm + # set found here is rendered to PNG and uploaded like a slam_toolbox map. + extra_map_dirs=[Path(p) for p in _env("EXTRA_MAP_DIRS", "").split(":") if p.strip()], web_data_dir=Path(data_dir) if data_dir else None, legacy_places=Path(legacy) if legacy else None, web_nav3_url=_env("WEB_NAV3_URL", "").rstrip("/"), @@ -797,6 +801,7 @@ def _discover_slam_sets(cfg: Config) -> List[MapArtifact]: """Find slam_toolbox / map_server map sets: .yaml + .pgm (+ optional .posegraph/.data) under the maps roots and maps_slam/.""" roots = [cfg.maps_dir, cfg.maps_dir / cfg.robot, cfg.maps_dir / "maps_slam"] + roots += list(cfg.extra_map_dirs) # Nav2/Pudu map dirs mounted via EXTRA_MAP_DIRS seen: set = set() out: List[MapArtifact] = [] for root in roots: @@ -824,6 +829,11 @@ def _discover_slam_sets(cfg: Config) -> List[MapArtifact]: out.append(MapArtifact(path=y, name=y.name, stem=y.stem, size=size, mtime=mtime, fmt="slam_toolbox", files=files)) + # Pudu/Nav2 converter emits a plain map + a keepout-BAKED twin (obstacles baked + # in for Foxy, which has no KeepoutFilter). The baked one is the deploy map — drop + # the redundant plain twin so the fleet server gets one canonical map, not two. + baked = {m.stem[: -len("_keepout_baked")] for m in out if m.stem.endswith("_keepout_baked")} + out = [m for m in out if m.stem not in baked] return out @@ -1772,6 +1782,7 @@ def main(argv: Optional[List[str]] = None) -> int: ap = argparse.ArgumentParser(description="G1 fleet agent: telemetry + map sync") ap.add_argument("--simulate", action="store_true", help="synthetic DDS state (map scan stays real)") ap.add_argument("--once", action="store_true", help="one map pass + one telemetry post, then exit") + ap.add_argument("--map-only", action="store_true", help="upload discovered maps once (no DDS, no telemetry), then exit") ap.add_argument("--dry-run", action="store_true", help="build payloads, never POST") ap.add_argument("--force", action="store_true", help="re-upload maps even if unchanged") ap.add_argument("--list", action="store_true", help="list discovered maps and exit") @@ -1804,6 +1815,14 @@ def main(argv: Optional[List[str]] = None) -> int: cfg.position_source, cfg.maps_dir, cfg.map_poll_interval, " [SIMULATE]" if args.simulate else "") + session = requests.Session() + + if args.map_only: + # map upload only — search all map roots (incl. EXTRA_MAP_DIRS) and ship, + # without opening DDS or posting telemetry (won't disturb a live feed). + map_sync_once(cfg, session, force=args.force, dry_run=args.dry_run) + return 0 + reader = None pos = None if not args.simulate: @@ -1812,7 +1831,6 @@ def main(argv: Optional[List[str]] = None) -> int: pos = RosbridgePosition(cfg) time.sleep(1.0) - session = requests.Session() tick = 0 def one_telemetry() -> None: diff --git a/agents/go2/.env.example b/agents/go2/.env.example index 1ffb51d..ddc674a 100644 --- a/agents/go2/.env.example +++ b/agents/go2/.env.example @@ -40,11 +40,18 @@ STORAGE_DATA_PATH= # The installer mounts the robot's real maps dir at /data/maps (probed on the # robot: rtabmap*.db / slam_toolbox yaml+pgm sets). Fallback = agent's own dir. MAPS_DIR=/data/maps +# Extra map roots (colon-separated container paths) — Nav2/Pudu deploy maps +# (map.pgm + map.yaml). The installer mounts the robot's *_nav2_docker/maps +# here; safe to leave set even when nothing is mounted (missing dirs skipped). +EXTRA_MAP_DIRS=/data/nav2_maps DATA_DIR=/data/web_data STATE_DIR=/data/state MAP_SELECT=all MAP_UPLOAD_MODE=multipart MAP_POLL_INTERVAL=30 +# Skip .db maps larger than this (the prod server 413s at ~8 MB). Raster +# (pgm+yaml) maps are tiny and never skipped. +MAP_MAX_UPLOAD_MB=7 # ── logs + alerts ───────────────────────────────────────────────────────────── # agent log lines shipped to /{sn}/logs every LOGS_INTERVAL; new faults POSTed diff --git a/agents/go2/sanad_api_go2.py b/agents/go2/sanad_api_go2.py index 2ee9325..861f247 100644 --- a/agents/go2/sanad_api_go2.py +++ b/agents/go2/sanad_api_go2.py @@ -131,6 +131,7 @@ class Config: # map sync robot: str maps_dir: Path + extra_map_dirs: List[Path] # extra roots to scan for pgm+yaml sets (e.g. Nav2/Pudu maps) web_data_dir: Optional[Path] legacy_places: Optional[Path] web_nav3_url: str @@ -210,6 +211,9 @@ class Config: telemetry_endpoint=_env("TELEMETRY_ENDPOINT", "/api/v1/fleet/ingest/telemetry"), robot=_env("ROBOT", "sanad"), maps_dir=Path(_env("MAPS_DIR", "/data/maps")), + # colon-separated extra roots (mounted Nav2/Pudu map dirs). Any *.yaml+*.pgm + # set found here is rendered to PNG and uploaded like a slam_toolbox map. + extra_map_dirs=[Path(p) for p in _env("EXTRA_MAP_DIRS", "").split(":") if p.strip()], web_data_dir=Path(data_dir) if data_dir else None, legacy_places=Path(legacy) if legacy else None, web_nav3_url=_env("WEB_NAV3_URL", "").rstrip("/"), @@ -751,6 +755,7 @@ def _discover_slam_sets(cfg: Config) -> List[MapArtifact]: """Find slam_toolbox / map_server map sets: .yaml + .pgm (+ optional .posegraph/.data) under the maps roots and maps_slam/.""" roots = [cfg.maps_dir, cfg.maps_dir / cfg.robot, cfg.maps_dir / "maps_slam"] + roots += list(cfg.extra_map_dirs) # Nav2/Pudu map dirs mounted via EXTRA_MAP_DIRS seen: set = set() out: List[MapArtifact] = [] for root in roots: @@ -778,6 +783,11 @@ def _discover_slam_sets(cfg: Config) -> List[MapArtifact]: out.append(MapArtifact(path=y, name=y.name, stem=y.stem, size=size, mtime=mtime, fmt="slam_toolbox", files=files)) + # Pudu/Nav2 converter emits a plain map + a keepout-BAKED twin (obstacles baked + # in for Foxy, which has no KeepoutFilter). The baked one is the deploy map — drop + # the redundant plain twin so the fleet server gets one canonical map, not two. + baked = {m.stem[: -len("_keepout_baked")] for m in out if m.stem.endswith("_keepout_baked")} + out = [m for m in out if m.stem not in baked] return out @@ -1726,6 +1736,7 @@ def main(argv: Optional[List[str]] = None) -> int: ap = argparse.ArgumentParser(description="Go2 fleet agent: telemetry + map sync") ap.add_argument("--simulate", action="store_true", help="synthetic DDS state (map scan stays real)") ap.add_argument("--once", action="store_true", help="one map pass + one telemetry post, then exit") + ap.add_argument("--map-only", action="store_true", help="upload discovered maps once (no DDS, no telemetry), then exit") ap.add_argument("--dry-run", action="store_true", help="build payloads, never POST") ap.add_argument("--force", action="store_true", help="re-upload maps even if unchanged") ap.add_argument("--list", action="store_true", help="list discovered maps and exit") @@ -1758,6 +1769,14 @@ def main(argv: Optional[List[str]] = None) -> int: cfg.position_source, cfg.maps_dir, cfg.map_poll_interval, " [SIMULATE]" if args.simulate else "") + session = requests.Session() + + if args.map_only: + # map upload only — search all map roots (incl. EXTRA_MAP_DIRS) and ship, + # without opening DDS or posting telemetry (won't disturb a live feed). + map_sync_once(cfg, session, force=args.force, dry_run=args.dry_run) + return 0 + reader = None pos = None if not args.simulate: @@ -1766,7 +1785,6 @@ def main(argv: Optional[List[str]] = None) -> int: pos = RosbridgePosition(cfg) time.sleep(1.0) - session = requests.Session() tick = 0 def one_telemetry() -> None: diff --git a/agents/r1/.env.example b/agents/r1/.env.example index 95437f8..8744ddc 100644 --- a/agents/r1/.env.example +++ b/agents/r1/.env.example @@ -46,11 +46,18 @@ STORAGE_DATA_PATH= # The installer mounts the robot's real maps dir at /data/maps (probed on the # robot: rtabmap*.db / slam_toolbox yaml+pgm sets). Fallback = agent's own dir. MAPS_DIR=/data/maps +# Extra map roots (colon-separated container paths) — Nav2/Pudu deploy maps +# (map.pgm + map.yaml). The installer mounts the robot's *_nav2_docker/maps +# here; safe to leave set even when nothing is mounted (missing dirs skipped). +EXTRA_MAP_DIRS=/data/nav2_maps DATA_DIR=/data/web_data STATE_DIR=/data/state MAP_SELECT=all MAP_UPLOAD_MODE=multipart MAP_POLL_INTERVAL=30 +# Skip .db maps larger than this (the prod server 413s at ~8 MB). Raster +# (pgm+yaml) maps are tiny and never skipped. +MAP_MAX_UPLOAD_MB=7 # ── logs + alerts ───────────────────────────────────────────────────────────── # agent log lines shipped to /{sn}/logs every LOGS_INTERVAL; new faults POSTed diff --git a/agents/r1/sanad_api_r1.py b/agents/r1/sanad_api_r1.py index a751e92..d06c917 100644 --- a/agents/r1/sanad_api_r1.py +++ b/agents/r1/sanad_api_r1.py @@ -131,6 +131,7 @@ class Config: # map sync robot: str maps_dir: Path + extra_map_dirs: List[Path] # extra roots to scan for pgm+yaml sets (e.g. Nav2/Pudu maps) web_data_dir: Optional[Path] legacy_places: Optional[Path] web_nav3_url: str @@ -210,6 +211,9 @@ class Config: telemetry_endpoint=_env("TELEMETRY_ENDPOINT", "/api/v1/fleet/ingest/telemetry"), robot=_env("ROBOT", "sanad"), maps_dir=Path(_env("MAPS_DIR", "/data/maps")), + # colon-separated extra roots (mounted Nav2/Pudu map dirs). Any *.yaml+*.pgm + # set found here is rendered to PNG and uploaded like a slam_toolbox map. + extra_map_dirs=[Path(p) for p in _env("EXTRA_MAP_DIRS", "").split(":") if p.strip()], web_data_dir=Path(data_dir) if data_dir else None, legacy_places=Path(legacy) if legacy else None, web_nav3_url=_env("WEB_NAV3_URL", "").rstrip("/"), @@ -797,6 +801,7 @@ def _discover_slam_sets(cfg: Config) -> List[MapArtifact]: """Find slam_toolbox / map_server map sets: .yaml + .pgm (+ optional .posegraph/.data) under the maps roots and maps_slam/.""" roots = [cfg.maps_dir, cfg.maps_dir / cfg.robot, cfg.maps_dir / "maps_slam"] + roots += list(cfg.extra_map_dirs) # Nav2/Pudu map dirs mounted via EXTRA_MAP_DIRS seen: set = set() out: List[MapArtifact] = [] for root in roots: @@ -824,6 +829,11 @@ def _discover_slam_sets(cfg: Config) -> List[MapArtifact]: out.append(MapArtifact(path=y, name=y.name, stem=y.stem, size=size, mtime=mtime, fmt="slam_toolbox", files=files)) + # Pudu/Nav2 converter emits a plain map + a keepout-BAKED twin (obstacles baked + # in for Foxy, which has no KeepoutFilter). The baked one is the deploy map — drop + # the redundant plain twin so the fleet server gets one canonical map, not two. + baked = {m.stem[: -len("_keepout_baked")] for m in out if m.stem.endswith("_keepout_baked")} + out = [m for m in out if m.stem not in baked] return out @@ -1772,6 +1782,7 @@ def main(argv: Optional[List[str]] = None) -> int: ap = argparse.ArgumentParser(description="R1 fleet agent: telemetry + map sync") ap.add_argument("--simulate", action="store_true", help="synthetic DDS state (map scan stays real)") ap.add_argument("--once", action="store_true", help="one map pass + one telemetry post, then exit") + ap.add_argument("--map-only", action="store_true", help="upload discovered maps once (no DDS, no telemetry), then exit") ap.add_argument("--dry-run", action="store_true", help="build payloads, never POST") ap.add_argument("--force", action="store_true", help="re-upload maps even if unchanged") ap.add_argument("--list", action="store_true", help="list discovered maps and exit") @@ -1804,6 +1815,14 @@ def main(argv: Optional[List[str]] = None) -> int: cfg.position_source, cfg.maps_dir, cfg.map_poll_interval, " [SIMULATE]" if args.simulate else "") + session = requests.Session() + + if args.map_only: + # map upload only — search all map roots (incl. EXTRA_MAP_DIRS) and ship, + # without opening DDS or posting telemetry (won't disturb a live feed). + map_sync_once(cfg, session, force=args.force, dry_run=args.dry_run) + return 0 + reader = None pos = None if not args.simulate: @@ -1812,7 +1831,6 @@ def main(argv: Optional[List[str]] = None) -> int: pos = RosbridgePosition(cfg) time.sleep(1.0) - session = requests.Session() tick = 0 def one_telemetry() -> None: diff --git a/deploy.svg b/deploy.svg new file mode 100644 index 0000000..465082d --- /dev/null +++ b/deploy.svg @@ -0,0 +1,69 @@ + + + + + + + + + Deploy — workstation -> robot, over SSH (no sudo) + One command: ./fleet_install.sh install <type> <ip> --sn ... --token ... --server-url ... + + + + + + WORKSTATION + agents/<type>/ + single source + of truth + + + + rsync -> robot + ~/sanad_api_<type> + --delete + keeps .env, state + + + + docker build + native arm64 + on the robot + (DDS stack for r1/go2) + + + + docker create + container + --network host + -v /:/host:ro · maps + + + + systemd --user + + enable-linger + auto-start on boot + Restart=always + + + + RUNNING + -> HTTPS -> + fleet server + Bearer token + + + + + + + + + + + + + + on the robot  ·  ssh unitree@<ip>  ·  no root, no docker-compose + diff --git a/diagram.svg b/diagram.svg index af87dec..a019ba5 100644 --- a/diagram.svg +++ b/diagram.svg @@ -1,114 +1,109 @@ - + - - - - - + + - - + - - Sanad Fleet — Architecture & Data Pipeline - Robot state -> containerized agent -> fleet server · outbound HTTPS only + + Sanad Fleet — one agent, five endpoints + Robot state -> containerized agent -> YS Lootah fleet server · outbound HTTPS only, read-only toward the robot - - ON THE ROBOT (edge) - AGENT - YS LOOTAH FLEET + + + ROBOT · G1 · R1 · Go2 + unitree_hg (G1/R1) · unitree_go (Go2) - - - - - - G1 — map + telemetry - map: web_nav3 .db + places (files) - telem: rt/lf/bmsstate · rt/lowstate - position: rt/lf/odommodestate + + DDS topics + rt/lf/bmsstate-> battery, SoH + rt/lowstate-> temps, faults + rt/lf/odommodestate + -> position + sport RPC 7001-> loco mode (ro) + + + Files · via /host:ro mount + SLAM .db · Nav2/Pudu map.pgm+yaml + rendered to PNG for upload + + + Sanad app + docker json-log -> project logs + dashboard :8001 -> remote URL + + + + reads + + + + sanad_api_<type> + Docker · --network host · user systemd · no sudo + read-only · never commands motion + + + + + + Telemetry · 2s + battery · status · position · faults · control mode + + + Map sync · 30s + discover pgm+yaml / .db · upload once per change + + + Alert scan · 10s + battery<=50 · Gemini billing · any error + + + Log ship · 60s + agent lines + [sanadr1-logs] + + + Remote register · 60s + dashboard URL + ssh unitree@ip - - - - - R1 — telemetry - source: DDS (unitree_hg) - rt/lf/bmsstate · rt/lowstate - iface eth10 - - - - - - Go2 — telemetry - source: DDS (unitree_go) - rt/lowstate.bms_state - unverified on hardware + all three generated from agents/g1 · gen_agents.py + + + + + + + + - - - - - - - - - sanad_api_<type> - Docker · --network host - read-only · never moves - - - read -> build payload -> POST - - - - systemd --user service - auto-start on boot · no sudo - - - - heartbeat if source silent - -> robot stays "online" - - loop: 2 s telemetry · 30 s map (on change) - deployed via fleet_install.sh - (rsync + docker build on robot) - - - - - HTTPS POST - Bearer token - - - - - Fleet Server — ingest API - POST …/ingest/telemetry - ~2 s - POST …/ingest/{sn}/map - on change - - Authorization: Bearer <token> - keyed by sn · mac in telemetry - - - - + + + Fleet server + eco.yslootahrobotics.com + Authorization: Bearer <token> · keyed by sn - - Dashboard · Storage · Alerts - live status · map view · fault alerts - (YS Lootah side — already built) + + POST /ingest/telemetry + every ~2s · 27 fields + + POST /ingest/{sn}/map + on change · PNG + resolution + origin + + POST /ingest/{sn}/alert + on new fault (strings) + + POST /ingest/{sn}/logs + every 60s · ring-buffered lines + + POST /ingest/{sn}/remote + kind: web + ssh + also GET /ingest/{sn}/commands (+ ack) - - Outbound HTTPS only — no inbound port on the robot. Each robot type = its own Docker image, installed & managed over SSH by - fleet_install.sh · user-level systemd auto-start, no sudo. + + No inbound port on the robot · each robot type is its own Docker image · installed & managed over SSH by fleet_install.sh · user-level systemd auto-start, no sudo. diff --git a/fleet_install.sh b/fleet_install.sh index 1ae4a6b..8acd222 100755 --- a/fleet_install.sh +++ b/fleet_install.sh @@ -43,11 +43,12 @@ detect_server_ip(){ [ -n "$SERVER_IP" ] && { echo "$SERVER_IP"; return; } # Every agent now syncs maps: find where THIS robot's Sanad/SLAM stack keeps its # maps on the host. Falls back to the agent's own dirs (agent then reports # map: no_map until a map exists / is exposed). -MAPS_HOST=""; DATA_HOST="" +MAPS_HOST=""; DATA_HOST=""; EXTRA_MAPS_HOST="" probe_maps_dirs(){ local t="$1"; local rdir; rdir="$(rdir_of "$t")" MAPS_HOST="/home/$USER_/$rdir/maps" DATA_HOST="/home/$USER_/$rdir/web_data" + EXTRA_MAPS_HOST="" if [ "$t" = g1 ]; then for c in "/home/$USER_/marcus_nav2_test/maps" \ "/home/$USER_/sanad_deploy/Sanad_Package_4/nav/data/ros"; do @@ -63,15 +64,27 @@ probe_maps_dirs(){ found=$(rmt "find /home/$USER_ -maxdepth 4 \( -name 'rtabmap*.db' -o -name '*.posegraph' \) 2>/dev/null | head -1" 2>/dev/null) [ -n "$found" ] && MAPS_HOST="$(dirname "$found")" fi + # Nav2/Pudu deploy maps (map.pgm + map.yaml, incl. keepout-baked) usually live in + # a *_nav2_docker/maps dir SEPARATE from the SLAM db above. Find it and mount it + # as an extra map root so the agent discovers+uploads it too. Skip if it's the + # same as the primary maps mount (g1 already maps marcus_nav2_test/maps). + local e + e=$(rmt "for d in /home/$USER_/${t}_nav2_docker/maps /home/$USER_/marcus_nav2_test/maps \ + /home/$USER_/go2_nav2_docker/maps /home/$USER_/r1_nav2_docker/maps; do \ + ls \$d/*.yaml >/dev/null 2>&1 && { echo \$d; break; }; done" 2>/dev/null | head -1) + if [ -n "$e" ] && [ "$e" != "$MAPS_HOST" ]; then EXTRA_MAPS_HOST="$e"; fi } run_args(){ # /host:ro → real disk stats; maps/web_data ro + state rw → map sync local rdir; rdir="$(rdir_of "$1")" + # extra Nav2/Pudu map dir (mounted at /data/nav2_maps, matches EXTRA_MAP_DIRS) + local extra="" + [ -n "$EXTRA_MAPS_HOST" ] && extra=" -v ${EXTRA_MAPS_HOST}:/data/nav2_maps:ro" echo "--network host --env-file /home/$USER_/$rdir/.env -v /:/host:ro \ -v ${MAPS_HOST:-/home/$USER_/$rdir/maps}:/data/maps:ro \ -v ${DATA_HOST:-/home/$USER_/$rdir/web_data}:/data/web_data:ro \ - -v /home/$USER_/$rdir/state:/data/state" + -v /home/$USER_/$rdir/state:/data/state${extra}" } # Resolve the SERVER_URL + VERIFY_TLS for a deploy: a full --server-url (real @@ -114,6 +127,7 @@ POLL_INTERVAL=2 LOW_SOC=50 ROBOT=sanad MAPS_DIR=/data/maps +EXTRA_MAP_DIRS=/data/nav2_maps DATA_DIR=/data/web_data STATE_DIR=/data/state MAP_SELECT=all