# Sanad Fleet — Data Pipeline How robot state becomes a record on the YS Lootah fleet dashboard, end to end. This is the reference for **what flows, in what shape, when, and how failures are handled**. For install/ops, see [README.md](README.md). --- ## 1. Big picture ![Sanad Fleet — architecture and data pipeline](diagram.svg) ``` ROBOT (edge) │ YS LOOTAH (cloud) │ ┌───────────────┐ read ┌──────────────┐ │ HTTPS POST ┌──────────────┐ │ robot sources │ ─────────▶ │ sanad_api_* │ ─┼───────────────▶ │ fleet server │ │ DDS / files │ │ (Docker) │ │ Bearer token │ (ingest API) │ └───────────────┘ └──────────────┘ │ └──────┬───────┘ ▲ │ │ │ │ passive, read-only │ systemd │ ▼ │ (never commands motion) │ user svc │ ┌──────────────┐ ▼ │ │ dashboard │ stays "online" │ │ storage │ via heartbeat │ │ alerts │ │ └──────────────┘ ``` - **Direction:** outbound only. The robot opens no inbound ports; every call is an HTTPS `POST` to the fleet server. - **Transport:** HTTP/1.1 + JSON (telemetry) or `multipart/form-data` (map). - **Auth:** `Authorization: Bearer ` on every request. - **Identity:** each robot is keyed by `sn` (fleet id, e.g. `r1_82`); the hardware `mac` rides along in telemetry. --- ## 2. API contract The full spec (from `sanad-tasks-en.html`) defines these ingest endpoints. This repo currently implements the **bold** ones; the rest are documented for later. | endpoint | method | agent | status | |---|---|---|---| | **`/api/v1/fleet/ingest/telemetry`** | POST | G1, R1, Go2 | ✅ ~2 s; 27 fields incl. software/firmware/control + all status mirrors | | **`/api/v1/fleet/ingest/{sn}/map`** | POST | G1, R1, Go2 | ✅ once per content; rtabmap `.db` (≤~7 MB) + slam_toolbox `pgm/yaml`→PNG | | **`/api/v1/fleet/ingest/{sn}/alert`** | POST | G1, R1, Go2 | ✅ each NEW fault, rising edge, string body | | **`/api/v1/fleet/ingest/{sn}/logs`** | POST | G1, R1, Go2 | ✅ agent + `[sanadr1-logs]` project lines every `LOGS_INTERVAL` (60 s) | | **`/api/v1/fleet/ingest/{sn}/remote`** | POST | G1, R1, Go2 | ✅ registers `web` (Sanad dashboard URL) + `ssh` (`ssh unitree@`) every 60 s | | `/api/v1/fleet/ingest/{sn}/commands` | GET | — | ⏳ channel works; no command executor built (motion = deliberately out) | | `/api/v1/fleet/ingest/commands/{id}/ack` | POST | — | ⏳ | Auth header (all): `Authorization: Bearer `. --- ## 3. Telemetry pipeline (R1, Go2) **Goal:** a fresh status record every ~2 s; never go dark. ``` DDS topics agent (2 s loop) server ────────── ──────────────── ────── rt/lowstate ─┐ callbacks ┌─ snapshot() ─┐ build_telemetry() ┌ POST (LowState_) ├─────────────▶ │ battery ├──────────────────────▶│ /ingest/ rt/lf/bmsstate│ (background │ temps/dq │ derive status+faults │ telemetry (BmsState_) ─┘ threads) │ liveness/age │ read mac └ (JSON) └──────────────┘ │ no data? └────────▶ heartbeat (battery:null, offline) ``` ### 3.1 Sources per robot | field | R1 (`unitree_hg`) | Go2 (`unitree_go`) | |---|---|---| | `battery` (0–100) | `rt/lf/bmsstate` → `BmsState_.soc` | `rt/lowstate` → `LowState_.bms_state.soc` | | `charging` | `BmsState_.current` > +0.05 A | `bms_state.current` > +0.05 A | | `faults[]` | `rt/lowstate` motor temps + staleness | same | | `status` | derived (see 3.3); optional loco FSM `GET 7001` | derived | | `position` | optional `rosbridge /odom` | optional `rosbridge /odom` or `rt/lf/sportmodestate` | | `mac` | NIC (`/sys/class/net//address`) | NIC | DDS is initialized once (`ChannelFactoryInitialize(domain, interface)`); each topic has a subscriber whose callback updates a locked in-memory snapshot. The loop reads the snapshot — it never blocks on the network. ### 3.2 Payload ```json POST /api/v1/fleet/ingest/telemetry Authorization: Bearer Content-Type: application/json { // ── identity ── "sn": "E39N4000Q6D7E70F", // the robot's REAL Unitree serial (server key) "name": "r1_82", // friendly display name "mac": "4c:bb:47:51:25:9a", "brand": "unitree", "type": "humanoid", "model": "r1", // type: humanoid|dog // ── software / firmware ── "software": { "ros": "foxy", "os": "Ubuntu 20.04.5 LTS", "os_version": "20.04", "kernel": "5.10.104-tegra", "arch": "aarch64", "python": "3.10.20", "agent": "sanad_api_r1 2026.07.13" }, "firmware": { "board": "NVIDIA Orin NX Developer Kit", "l4t": "R35.3.1", "kernel": "5.10.104-tegra", "robot": "0.0", "bms": "0.44" }, // ── power / health ── "battery": 62, "charging": false, "battery_detail": { "voltage_v": 34.5, "current_a": -3.08, "temp_c": 42, "soh": 99, "cycles": 10 }, "motor_temp": { "max": 49.0, "avg": 37.8, "min": 32.0 }, // null = not receiving "storage": { "total_gb": 98.2, "free_gb": 58.5, "used_percent": 36.2, "data_kb": 770.0 }, // data_kb only when STORAGE_DATA_PATH set // ── state ── "status": "idle", "position": { "x": 12.4, "y": 3.1 }, // or null when no localization source "control": { "fsm_id": 0, "mode": "zero_torque", // zero_torque|damp|lock|running "armed": false, "walk_ready": false, "teleop_active": false, "switchable_modes": ["zero_torque","damp","lock","running"], "remote_switch_enabled": false }, // read-only; no remote motion "faults": [], // strings, e.g. "LOW_BATTERY: battery 12% (warning)" // ── sub-system status (each stream's health, mirrored here) ── "map": { "uploaded": true, "state": "uploaded", "maps_found": 2, "last_map": "rtabmap", "error": null }, "logs": { "last_sent": "…", "lines_sent": 105, "ok": true }, "project_logs": "sanadr1-logs", // the Sanad app whose logs are tailed (null=none) "remote": { "url": "http://10.255.254.82:8001", "port": 8001, "kind": "web", "ok": true, "ssh": "ssh unitree@10.255.254.82", "ssh_ok": true }, "alerts": { "sent": 0, "last": null, "last_time": null, "ok": null }, // ── time ── "time": "2026-07-13 17:00:22+04:00", // full local datetime (TZ_OFFSET_HOURS, default +4) "started_at": "2026-07-13 17:00:21+04:00", "last_start": "2026-07-13 16:57:43+04:00", // previous agent start (persisted) "uptime_s": 625, "ts": 1731000000 } ``` Sub-objects `map` / `logs` / `alerts` / `remote` / `control` are **status mirrors** — the real work happens on their own endpoints (below); telemetry just always shows whether each is healthy so the fleet UI never has to poll them separately. ### 3.3 Status derivation ``` if no rt/lowstate for >3 s AND never saw battery → "offline" elif charging (bms current > +0.05 A) → "charging" elif max |joint velocity| > 0.15 rad/s → "moving" elif R1_READ_FSM and FSM id known → FSM label (811 ready / 4 standing / 1 damping / 0 zero_torque) else → "idle" ``` ### 3.4 Faults Ordinary faults ride inside telemetry `faults[]` (the spec reserves the separate `/alert` endpoint for critical, immediate events like e-stop): | code | trigger | severity | |---|---|---| | `LOW_BATTERY` | `soc ≤ LOW_SOC` (default 15) | warning | | `MOTOR_OVERTEMP` | any motor temp ≥ `MOTOR_TEMP_MAX` (default 85 °C) | warning | | `COMMS_STALE` | no `rt/lowstate` for > 3 s | critical | ### 3.5 Heartbeat & failure handling - **No DDS / no state** → still POST with `battery:null`, `status:"offline"` so the dashboard shows the robot as reachable (the spec's "send a heartbeat" rule). - **unitree_sdk2py missing** → the agent logs a warning and runs in heartbeat mode (no crash). - **POST fails** (transport or non-2xx) → logged, loop continues; next tick retries in `POLL_INTERVAL` seconds. No back-pressure, no queue (latest state wins). - **Every tick is wrapped** — one bad read can't kill the loop. --- ## 4. Map pipeline (G1) **Goal:** keep the server's copy of the nav map current, sending only on change. ``` files on disk agent (30 s scan) server ───────────── ───────────────── ────── maps//*.db ─┐ discover ┌ fingerprint (size+mtime) ─┐ changed? ┌ POST maps_meta.json ├────────▶ │ if changed: sha256 ├──yes──────▶│ /ingest/ web/data// │ │ load places → points[] │ │ {sn}/map places/.json ┘ └ build meta + db bytes ─────┘ └ (multipart) │ unchanged └────────▶ skip (state/uploaded.json) ``` ### 4.1 Sources - **Map file:** RTAB-Map SQLite `.db` — `maps//.db` (web_nav3). - **Metadata:** `maps//maps_meta.json` → `{ "": {description, created_at} }`. - **Places → points:** `web/data//places/.json` (`{ "": {x, y, z, qx, qy, qz, qw} }`), converted to `{name, type:"waypoint", x, y, yaw}` (yaw computed from the quaternion). ### 4.2 Change detection 1. Cheap pre-check: `size + mtime` vs `STATE_DIR/uploaded.json`. 2. If different, compute `sha256` of the `.db` (content-true). 3. Upload; on success, record the new fingerprint. A restart re-reads state → no redundant re-upload. `MAP_SELECT` chooses scope: `all` (default), `active` (via web_nav3 `GET /api/status`), or `newest`. ### 4.3 Payload (two wire formats) `multipart/form-data` (default) — file part `db` + form field `meta`: ``` POST /api/v1/fleet/ingest/g1_7892/map Authorization: Bearer Content-Type: multipart/form-data db = (application/octet-stream) meta = { "sn":"g1_7892", "name":"floor-1", "file":"floor-1.db", "format":"rtabmap_db", "size_bytes":6994944, "sha256":"…", "mtime":1731000000, "description":"ground floor", "points":[ {"name":"dock","type":"waypoint","x":1.2,"y":3.4,"yaw":0.0} ] } ``` `base64json` (set `MAP_UPLOAD_MODE=base64json`) — same fields as JSON with the `.db` as `db_base64`. > **Server note:** this uploads the raw RTAB-Map `.db` (not a rendered PNG), so the > server must accept a `format:"rtabmap_db"` artifact. The spec's image-based map > body (`image_base64`/`resolution`/`origin`) would instead require rendering the > live `/map` OccupancyGrid over rosbridge — a different path, not used here. --- ## 5. Timing | stream | cadence | trigger | |---|---|---| | telemetry (R1/Go2) | every `POLL_INTERVAL` (2 s) | timer | | map (G1) | scan every `POLL_INTERVAL` (30 s) | uploads only on content change | | DDS reads | continuous (subscriber callbacks) | firmware publish rate | | heartbeat | same as telemetry cadence | when state is unreadable | --- ## 6. Where it runs ``` robot host └─ systemd --user └─ sanad-api-.service (Restart=always, enabled at boot via linger) └─ docker start -a sanad-api- └─ container (--network host) └─ python -u sanad_api_.py ← the loop above ``` - **`--network host`**: DDS multicast visibility (R1/Go2) and localhost access to web_nav3:8765 (G1); also the robot's real NIC MAC. - **Single owner**: the container is `docker create`d without a docker restart policy; systemd owns start/stop/restart. --- ## 7. End-to-end sequences ### Telemetry tick ``` loop DDS(sub) agent fleet server │ (2 s) │ │ │ │────────────▶│ snapshot │ │ │ │───────────▶│ build payload │ │ │ │──── POST JSON ────▶│ (Bearer) │ │ │◀──── 200 OK ───────│ │ │ │ log "telemetry ok" │ ``` ### Map change ``` scan(30s) disk agent fleet server │────────────▶│ discover │ │ │ │───────────▶│ fingerprint │ │ │ │ changed → sha256 │ │ │ │ load points │ │ │ │── POST multipart ─▶│ (db + meta) │ │ │◀──── 200 OK ───────│ │ │ │ save state │ ``` --- ## 8. Verified behavior (real hardware) Confirmed against the live fleet during bring-up: - **R1** streamed real telemetry every 2 s; battery read live and drained `97 → 80%` across the session; `mac`/`sn` correct; heartbeat kicked in when the server was down. - **G1** uploaded a map (`multipart`, `format:rtabmap_db`, points with correct yaw) to the server. - The fleet server received a **G1 map and R1 telemetry in the same window** — the multi-robot pipeline works concurrently. - **Go2**: pipeline code + image verified (builds, `unitree_go` imports, simulate payloads correct); **not yet run on a real Go2**.