268 lines
12 KiB
Markdown
268 lines
12 KiB
Markdown
# Sanad Fleet — Data Pipeline
|
||
|
||
How robot state becomes a record on the YS Lootah fleet dashboard, end to end.
|
||
This is the reference for **what flows, in what shape, when, and how failures are
|
||
handled**. For install/ops, see [README.md](README.md).
|
||
|
||
---
|
||
|
||
## 1. Big picture
|
||
|
||
|
||
|
||
```
|
||
ROBOT (edge) │ YS LOOTAH (cloud)
|
||
│
|
||
┌───────────────┐ read ┌──────────────┐ │ HTTPS POST ┌──────────────┐
|
||
│ robot sources │ ─────────▶ │ sanad_api_* │ ─┼───────────────▶ │ fleet server │
|
||
│ DDS / files │ │ (Docker) │ │ Bearer token │ (ingest API) │
|
||
└───────────────┘ └──────────────┘ │ └──────┬───────┘
|
||
▲ │ │ │
|
||
│ passive, read-only │ systemd │ ▼
|
||
│ (never commands motion) │ user svc │ ┌──────────────┐
|
||
▼ │ │ dashboard │
|
||
stays "online" │ │ storage │
|
||
via heartbeat │ │ alerts │
|
||
│ └──────────────┘
|
||
```
|
||
|
||
- **Direction:** outbound only. The robot opens no inbound ports; every call is an
|
||
HTTPS `POST` to the fleet server.
|
||
- **Transport:** HTTP/1.1 + JSON (telemetry) or `multipart/form-data` (map).
|
||
- **Auth:** `Authorization: Bearer <device_token>` on every request.
|
||
- **Identity:** each robot is keyed by `sn` (fleet id, e.g. `r1_82`); the hardware
|
||
`mac` rides along in telemetry.
|
||
|
||
---
|
||
|
||
## 2. API contract
|
||
|
||
The full spec (from `sanad-tasks-en.html`) defines these ingest endpoints. This
|
||
repo currently implements the **bold** ones; the rest are documented for later.
|
||
|
||
| endpoint | method | agent | status |
|
||
|---|---|---|---|
|
||
| **`/api/v1/fleet/ingest/telemetry`** | POST | R1, Go2 | ✅ implemented |
|
||
| **`/api/v1/fleet/ingest/{sn}/map`** | POST | G1 | ✅ implemented |
|
||
| `/api/v1/fleet/ingest/{sn}/commands` | GET | — | ⏳ spec'd, not built |
|
||
| `/api/v1/fleet/ingest/commands/{id}/ack` | POST | — | ⏳ |
|
||
| `/api/v1/fleet/ingest/{sn}/alert` | POST | — | ⏳ (critical faults; ordinary ones ride in telemetry `faults[]`) |
|
||
| `/api/v1/fleet/ingest/{sn}/logs` | POST | — | ⏳ |
|
||
| `/api/v1/fleet/ingest/{sn}/remote` | POST | — | ⏳ (tunnel/SSH registration) |
|
||
|
||
Auth header (all): `Authorization: Bearer <device_token>`.
|
||
|
||
---
|
||
|
||
## 3. Telemetry pipeline (R1, Go2)
|
||
|
||
**Goal:** a fresh status record every ~2 s; never go dark.
|
||
|
||
```
|
||
DDS topics agent (2 s loop) server
|
||
────────── ──────────────── ──────
|
||
rt/lowstate ─┐ callbacks ┌─ snapshot() ─┐ build_telemetry() ┌ POST
|
||
(LowState_) ├─────────────▶ │ battery ├──────────────────────▶│ /ingest/
|
||
rt/lf/bmsstate│ (background │ temps/dq │ derive status+faults │ telemetry
|
||
(BmsState_) ─┘ threads) │ liveness/age │ read mac └ (JSON)
|
||
└──────────────┘
|
||
│ no data?
|
||
└────────▶ heartbeat (battery:null, offline)
|
||
```
|
||
|
||
### 3.1 Sources per robot
|
||
|
||
| field | R1 (`unitree_hg`) | Go2 (`unitree_go`) |
|
||
|---|---|---|
|
||
| `battery` (0–100) | `rt/lf/bmsstate` → `BmsState_.soc` | `rt/lowstate` → `LowState_.bms_state.soc` |
|
||
| `charging` | `BmsState_.current` > +0.05 A | `bms_state.current` > +0.05 A |
|
||
| `faults[]` | `rt/lowstate` motor temps + staleness | same |
|
||
| `status` | derived (see 3.3); optional loco FSM `GET 7001` | derived |
|
||
| `position` | optional `rosbridge /odom` | optional `rosbridge /odom` or `rt/lf/sportmodestate` |
|
||
| `mac` | NIC (`/sys/class/net/<iface>/address`) | NIC |
|
||
|
||
DDS is initialized once (`ChannelFactoryInitialize(domain, interface)`); each
|
||
topic has a subscriber whose callback updates a locked in-memory snapshot. The
|
||
loop reads the snapshot — it never blocks on the network.
|
||
|
||
### 3.2 Payload
|
||
|
||
```json
|
||
POST /api/v1/fleet/ingest/telemetry
|
||
Authorization: Bearer <device_token>
|
||
Content-Type: application/json
|
||
|
||
{ "sn": "r1_82",
|
||
"mac": "4c:bb:47:51:25:9a",
|
||
"battery": 80,
|
||
"charging": false,
|
||
"status": "idle",
|
||
"position": { "x": 12.4, "y": 3.1 }, // or null when no localization source
|
||
"faults": [],
|
||
"ts": 1731000000 }
|
||
```
|
||
|
||
### 3.3 Status derivation
|
||
|
||
```
|
||
if no rt/lowstate for >3 s AND never saw battery → "offline"
|
||
elif charging (bms current > +0.05 A) → "charging"
|
||
elif max |joint velocity| > 0.15 rad/s → "moving"
|
||
elif R1_READ_FSM and FSM id known → FSM label (811 ready / 4 standing / 1 damping / 0 zero_torque)
|
||
else → "idle"
|
||
```
|
||
|
||
### 3.4 Faults
|
||
|
||
Ordinary faults ride inside telemetry `faults[]` (the spec reserves the separate
|
||
`/alert` endpoint for critical, immediate events like e-stop):
|
||
|
||
| code | trigger | severity |
|
||
|---|---|---|
|
||
| `LOW_BATTERY` | `soc ≤ LOW_SOC` (default 15) | warning |
|
||
| `MOTOR_OVERTEMP` | any motor temp ≥ `MOTOR_TEMP_MAX` (default 85 °C) | warning |
|
||
| `COMMS_STALE` | no `rt/lowstate` for > 3 s | critical |
|
||
|
||
### 3.5 Heartbeat & failure handling
|
||
|
||
- **No DDS / no state** → still POST with `battery:null`, `status:"offline"` so the
|
||
dashboard shows the robot as reachable (the spec's "send a heartbeat" rule).
|
||
- **unitree_sdk2py missing** → the agent logs a warning and runs in heartbeat mode
|
||
(no crash).
|
||
- **POST fails** (transport or non-2xx) → logged, loop continues; next tick retries
|
||
in `POLL_INTERVAL` seconds. No back-pressure, no queue (latest state wins).
|
||
- **Every tick is wrapped** — one bad read can't kill the loop.
|
||
|
||
---
|
||
|
||
## 4. Map pipeline (G1)
|
||
|
||
**Goal:** keep the server's copy of the nav map current, sending only on change.
|
||
|
||
```
|
||
files on disk agent (30 s scan) server
|
||
───────────── ───────────────── ──────
|
||
maps/<robot>/*.db ─┐ discover ┌ fingerprint (size+mtime) ─┐ changed? ┌ POST
|
||
maps_meta.json ├────────▶ │ if changed: sha256 ├──yes──────▶│ /ingest/
|
||
web/data/<robot>/ │ │ load places → points[] │ │ {sn}/map
|
||
places/<map>.json ┘ └ build meta + db bytes ─────┘ └ (multipart)
|
||
│ unchanged
|
||
└────────▶ skip (state/uploaded.json)
|
||
```
|
||
|
||
### 4.1 Sources
|
||
|
||
- **Map file:** RTAB-Map SQLite `.db` — `maps/<robot>/<name>.db` (web_nav3).
|
||
- **Metadata:** `maps/<robot>/maps_meta.json` → `{ "<db>": {description, created_at} }`.
|
||
- **Places → points:** `web/data/<robot>/places/<map>.json`
|
||
(`{ "<name>": {x, y, z, qx, qy, qz, qw} }`), converted to
|
||
`{name, type:"waypoint", x, y, yaw}` (yaw computed from the quaternion).
|
||
|
||
### 4.2 Change detection
|
||
|
||
1. Cheap pre-check: `size + mtime` vs `STATE_DIR/uploaded.json`.
|
||
2. If different, compute `sha256` of the `.db` (content-true).
|
||
3. Upload; on success, record the new fingerprint. A restart re-reads state → no
|
||
redundant re-upload.
|
||
|
||
`MAP_SELECT` chooses scope: `all` (default), `active` (via web_nav3
|
||
`GET /api/status`), or `newest`.
|
||
|
||
### 4.3 Payload (two wire formats)
|
||
|
||
`multipart/form-data` (default) — file part `db` + form field `meta`:
|
||
|
||
```
|
||
POST /api/v1/fleet/ingest/g1_7892/map
|
||
Authorization: Bearer <device_token>
|
||
Content-Type: multipart/form-data
|
||
|
||
db = <floor-1.db bytes> (application/octet-stream)
|
||
meta = { "sn":"g1_7892", "name":"floor-1", "file":"floor-1.db",
|
||
"format":"rtabmap_db", "size_bytes":6994944, "sha256":"…",
|
||
"mtime":1731000000, "description":"ground floor",
|
||
"points":[ {"name":"dock","type":"waypoint","x":1.2,"y":3.4,"yaw":0.0} ] }
|
||
```
|
||
|
||
`base64json` (set `MAP_UPLOAD_MODE=base64json`) — same fields as JSON with the
|
||
`.db` as `db_base64`.
|
||
|
||
> **Server note:** this uploads the raw RTAB-Map `.db` (not a rendered PNG), so the
|
||
> server must accept a `format:"rtabmap_db"` artifact. The spec's image-based map
|
||
> body (`image_base64`/`resolution`/`origin`) would instead require rendering the
|
||
> live `/map` OccupancyGrid over rosbridge — a different path, not used here.
|
||
|
||
---
|
||
|
||
## 5. Timing
|
||
|
||
| stream | cadence | trigger |
|
||
|---|---|---|
|
||
| telemetry (R1/Go2) | every `POLL_INTERVAL` (2 s) | timer |
|
||
| map (G1) | scan every `POLL_INTERVAL` (30 s) | uploads only on content change |
|
||
| DDS reads | continuous (subscriber callbacks) | firmware publish rate |
|
||
| heartbeat | same as telemetry cadence | when state is unreadable |
|
||
|
||
---
|
||
|
||
## 6. Where it runs
|
||
|
||
```
|
||
robot host
|
||
└─ systemd --user
|
||
└─ sanad-api-<type>.service (Restart=always, enabled at boot via linger)
|
||
└─ docker start -a sanad-api-<type>
|
||
└─ container (--network host)
|
||
└─ python -u sanad_api_<type>.py ← the loop above
|
||
```
|
||
|
||
- **`--network host`**: DDS multicast visibility (R1/Go2) and localhost access to
|
||
web_nav3:8765 (G1); also the robot's real NIC MAC.
|
||
- **Single owner**: the container is `docker create`d without a docker restart
|
||
policy; systemd owns start/stop/restart.
|
||
|
||
---
|
||
|
||
## 7. End-to-end sequences
|
||
|
||
### Telemetry tick
|
||
|
||
```
|
||
loop DDS(sub) agent fleet server
|
||
│ (2 s) │ │ │
|
||
│────────────▶│ snapshot │ │
|
||
│ │───────────▶│ build payload │
|
||
│ │ │──── POST JSON ────▶│ (Bearer)
|
||
│ │ │◀──── 200 OK ───────│
|
||
│ │ │ log "telemetry ok" │
|
||
```
|
||
|
||
### Map change
|
||
|
||
```
|
||
scan(30s) disk agent fleet server
|
||
│────────────▶│ discover │ │
|
||
│ │───────────▶│ fingerprint │
|
||
│ │ │ changed → sha256 │
|
||
│ │ │ load points │
|
||
│ │ │── POST multipart ─▶│ (db + meta)
|
||
│ │ │◀──── 200 OK ───────│
|
||
│ │ │ save state │
|
||
```
|
||
|
||
---
|
||
|
||
## 8. Verified behavior (real hardware)
|
||
|
||
Confirmed against the live fleet during bring-up:
|
||
|
||
- **R1** streamed real telemetry every 2 s; battery read live and drained
|
||
`97 → 80%` across the session; `mac`/`sn` correct; heartbeat kicked in when the
|
||
server was down.
|
||
- **G1** uploaded a map (`multipart`, `format:rtabmap_db`, points with correct
|
||
yaw) to the server.
|
||
- The fleet server received a **G1 map and R1 telemetry in the same window** — the
|
||
multi-robot pipeline works concurrently.
|
||
- **Go2**: pipeline code + image verified (builds, `unitree_go` imports, simulate
|
||
payloads correct); **not yet run on a real Go2**.
|