kassam/fleet
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update 2026-07-10 11:52:56

Browse Source
This commit is contained in:
kassam 2026-07-10 11:52:58 +04:00
parent db757d0b55
commit 6a91ca7c45
2 changed files with 611 additions and 62 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

265
PIPELINE.md Normal file
View File

@ -0,0 +1,265 @@
# Sanad Fleet — Data Pipeline
How robot state becomes a record on the YS Lootah fleet dashboard, end to end.
This is the reference for **what flows, in what shape, when, and how failures are
handled**. For install/ops, see [README.md](README.md).
---
## 1. Big picture
```
ROBOT (edge) │ YS LOOTAH (cloud)
│
┌───────────────┐ read ┌──────────────┐ │ HTTPS POST ┌──────────────┐
│ robot sources │ ─────────▶ │ sanad_api_* │ ─┼───────────────▶ │ fleet server │
│ DDS / files │ │ (Docker) │ │ Bearer token │ (ingest API) │
└───────────────┘ └──────────────┘ │ └──────┬───────┘
▲ │ │ │
│ passive, read-only │ systemd │ ▼
│ (never commands motion) │ user svc │ ┌──────────────┐
▼ │ │ dashboard │
stays "online" │ │ storage │
via heartbeat │ │ alerts │
│ └──────────────┘
```
- **Direction:** outbound only. The robot opens no inbound ports; every call is an
HTTPS `POST` to the fleet server.
- **Transport:** HTTP/1.1 + JSON (telemetry) or `multipart/form-data` (map).
- **Auth:** `Authorization: Bearer <device_token>` on every request.
- **Identity:** each robot is keyed by `sn` (fleet id, e.g. `r1_82`); the hardware
`mac` rides along in telemetry.
---
## 2. API contract
The full spec (from `sanad-tasks-en.html`) defines these ingest endpoints. This
repo currently implements the **bold** ones; the rest are documented for later.
| endpoint | method | agent | status |
|---|---|---|---|
| **`/api/v1/fleet/ingest/telemetry`** | POST | R1, Go2 | ✅ implemented |
| **`/api/v1/fleet/ingest/{sn}/map`** | POST | G1 | ✅ implemented |
| `/api/v1/fleet/ingest/{sn}/commands` | GET | — | ⏳ spec'd, not built |
| `/api/v1/fleet/ingest/commands/{id}/ack` | POST | — | ⏳ |
| `/api/v1/fleet/ingest/{sn}/alert` | POST | — | ⏳ (critical faults; ordinary ones ride in telemetry `faults[]`) |
| `/api/v1/fleet/ingest/{sn}/logs` | POST | — | ⏳ |
| `/api/v1/fleet/ingest/{sn}/remote` | POST | — | ⏳ (tunnel/SSH registration) |
Auth header (all): `Authorization: Bearer <device_token>`.
---
## 3. Telemetry pipeline (R1, Go2)
**Goal:** a fresh status record every ~2 s; never go dark.
```
DDS topics agent (2 s loop) server
────────── ──────────────── ──────
rt/lowstate ─┐ callbacks ┌─ snapshot() ─┐ build_telemetry() ┌ POST
(LowState_) ├─────────────▶ │ battery ├──────────────────────▶│ /ingest/
rt/lf/bmsstate│ (background │ temps/dq │ derive status+faults │ telemetry
(BmsState_) ─┘ threads) │ liveness/age │ read mac └ (JSON)
└──────────────┘
│ no data?
└────────▶ heartbeat (battery:null, offline)
```
### 3.1 Sources per robot
| field | R1 (`unitree_hg`) | Go2 (`unitree_go`) |
|---|---|---|
| `battery` (0–100) | `rt/lf/bmsstate` → `BmsState_.soc` | `rt/lowstate` → `LowState_.bms_state.soc` |
| `charging` | `BmsState_.current` > +0.05 A | `bms_state.current` > +0.05 A |
| `faults[]` | `rt/lowstate` motor temps + staleness | same |
| `status` | derived (see 3.3); optional loco FSM `GET 7001` | derived |
| `position` | optional `rosbridge /odom` | optional `rosbridge /odom` or `rt/lf/sportmodestate` |
| `mac` | NIC (`/sys/class/net/<iface>/address`) | NIC |
DDS is initialized once (`ChannelFactoryInitialize(domain, interface)`); each
topic has a subscriber whose callback updates a locked in-memory snapshot. The
loop reads the snapshot — it never blocks on the network.
### 3.2 Payload
```json
POST /api/v1/fleet/ingest/telemetry
Authorization: Bearer <device_token>
Content-Type: application/json
{ "sn": "r1_82",
"mac": "4c:bb:47:51:25:9a",
"battery": 80,
"charging": false,
"status": "idle",
"position": { "x": 12.4, "y": 3.1 }, // or null when no localization source
"faults": [],
"ts": 1731000000 }
```
### 3.3 Status derivation
```
if no rt/lowstate for >3 s AND never saw battery → "offline"
elif charging (bms current > +0.05 A) → "charging"
elif max |joint velocity| > 0.15 rad/s → "moving"
elif R1_READ_FSM and FSM id known → FSM label (811 ready / 4 standing / 1 damping / 0 zero_torque)
else → "idle"
```
### 3.4 Faults
Ordinary faults ride inside telemetry `faults[]` (the spec reserves the separate
`/alert` endpoint for critical, immediate events like e-stop):
| code | trigger | severity |
|---|---|---|
| `LOW_BATTERY` | `soc ≤ LOW_SOC` (default 15) | warning |
| `MOTOR_OVERTEMP` | any motor temp ≥ `MOTOR_TEMP_MAX` (default 85 °C) | warning |
| `COMMS_STALE` | no `rt/lowstate` for > 3 s | critical |
### 3.5 Heartbeat & failure handling
- **No DDS / no state** → still POST with `battery:null`, `status:"offline"` so the
dashboard shows the robot as reachable (the spec's "send a heartbeat" rule).
- **unitree_sdk2py missing** → the agent logs a warning and runs in heartbeat mode
(no crash).
- **POST fails** (transport or non-2xx) → logged, loop continues; next tick retries
in `POLL_INTERVAL` seconds. No back-pressure, no queue (latest state wins).
- **Every tick is wrapped** — one bad read can't kill the loop.
---
## 4. Map pipeline (G1)
**Goal:** keep the server's copy of the nav map current, sending only on change.
```
files on disk agent (30 s scan) server
───────────── ───────────────── ──────
maps/<robot>/*.db ─┐ discover ┌ fingerprint (size+mtime) ─┐ changed? ┌ POST
maps_meta.json ├────────▶ │ if changed: sha256 ├──yes──────▶│ /ingest/
web/data/<robot>/ │ │ load places → points[] │ │ {sn}/map
places/<map>.json ┘ └ build meta + db bytes ─────┘ └ (multipart)
│ unchanged
└────────▶ skip (state/uploaded.json)
```
### 4.1 Sources
- **Map file:** RTAB-Map SQLite `.db` — `maps/<robot>/<name>.db` (web_nav3).
- **Metadata:** `maps/<robot>/maps_meta.json` → `{ "<db>": {description, created_at} }`.
- **Places → points:** `web/data/<robot>/places/<map>.json`
(`{ "<name>": {x, y, z, qx, qy, qz, qw} }`), converted to
`{name, type:"waypoint", x, y, yaw}` (yaw computed from the quaternion).
### 4.2 Change detection
1. Cheap pre-check: `size + mtime` vs `STATE_DIR/uploaded.json`.
2. If different, compute `sha256` of the `.db` (content-true).
3. Upload; on success, record the new fingerprint. A restart re-reads state → no
redundant re-upload.
`MAP_SELECT` chooses scope: `all` (default), `active` (via web_nav3
`GET /api/status`), or `newest`.
### 4.3 Payload (two wire formats)
`multipart/form-data` (default) — file part `db` + form field `meta`:
```
POST /api/v1/fleet/ingest/g1_7892/map
Authorization: Bearer <device_token>
Content-Type: multipart/form-data
db = <floor-1.db bytes> (application/octet-stream)
meta = { "sn":"g1_7892", "name":"floor-1", "file":"floor-1.db",
"format":"rtabmap_db", "size_bytes":6994944, "sha256":"…",
"mtime":1731000000, "description":"ground floor",
"points":[ {"name":"dock","type":"waypoint","x":1.2,"y":3.4,"yaw":0.0} ] }
```
`base64json` (set `MAP_UPLOAD_MODE=base64json`) — same fields as JSON with the
`.db` as `db_base64`.
> **Server note:** this uploads the raw RTAB-Map `.db` (not a rendered PNG), so the
> server must accept a `format:"rtabmap_db"` artifact. The spec's image-based map
> body (`image_base64`/`resolution`/`origin`) would instead require rendering the
> live `/map` OccupancyGrid over rosbridge — a different path, not used here.
---
## 5. Timing
| stream | cadence | trigger |
|---|---|---|
| telemetry (R1/Go2) | every `POLL_INTERVAL` (2 s) | timer |
| map (G1) | scan every `POLL_INTERVAL` (30 s) | uploads only on content change |
| DDS reads | continuous (subscriber callbacks) | firmware publish rate |
| heartbeat | same as telemetry cadence | when state is unreadable |
---
## 6. Where it runs
```
robot host
└─ systemd --user
└─ sanad-api-<type>.service (Restart=always, enabled at boot via linger)
└─ docker start -a sanad-api-<type>
└─ container (--network host)
└─ python -u sanad_api_<type>.py ← the loop above
```
- **`--network host`**: DDS multicast visibility (R1/Go2) and localhost access to
web_nav3:8765 (G1); also the robot's real NIC MAC.
- **Single owner**: the container is `docker create`d without a docker restart
policy; systemd owns start/stop/restart.
---
## 7. End-to-end sequences
### Telemetry tick
```
loop DDS(sub) agent fleet server
│ (2 s) │ │ │
│────────────▶│ snapshot │ │
│ │───────────▶│ build payload │
│ │ │──── POST JSON ────▶│ (Bearer)
│ │ │◀──── 200 OK ───────│
│ │ │ log "telemetry ok" │
```
### Map change
```
scan(30s) disk agent fleet server
│────────────▶│ discover │ │
│ │───────────▶│ fingerprint │
│ │ │ changed → sha256 │
│ │ │ load points │
│ │ │── POST multipart ─▶│ (db + meta)
│ │ │◀──── 200 OK ───────│
│ │ │ save state │
```
---
## 8. Verified behavior (real hardware)
Confirmed against the live fleet during bring-up:
- **R1** streamed real telemetry every 2 s; battery read live and drained
`97 → 80%` across the session; `mac`/`sn` correct; heartbeat kicked in when the
server was down.
- **G1** uploaded a map (`multipart`, `format:rtabmap_db`, points with correct
yaw) to the server.
- The fleet server received a **G1 map and R1 telemetry in the same window** — the
multi-robot pipeline works concurrently.
- **Go2**: pipeline code + image verified (builds, `unitree_go` imports, simulate
payloads correct); **not yet run on a real Go2**.

408
README.md
View File

@ -1,94 +1,378 @@
# Sanad Fleet Agents
Per-robot agents that report to the YS Lootah fleet server, each shipped as
**full Docker** with a user-level **systemd auto-start service**. Deploy, manage,
and remove them over SSH with one interactive script.
On-robot agents that report each robot's state to the **YS Lootah fleet server**.
Every robot type ships as its own self-contained **Docker** image with a
user-level **systemd auto-start service**, deployed, managed, and removed over
SSH by a single script. No `docker-compose` on the robot, **no `sudo`** required.
> For the end-to-end data flow (topics → agent → HTTP → server), see
> [PIPELINE.md](PIPELINE.md).
---
## Table of contents
1. [What this is](#1-what-this-is)
2. [Directory layout](#2-directory-layout)
3. [The three agents](#3-the-three-agents)
4. [Prerequisites](#4-prerequisites)
5. [Quick start](#5-quick-start)
6. [Installer reference](#6-installer-reference)
7. [The auto-start service (systemd)](#7-the-auto-start-service-systemd)
8. [Configuration reference](#8-configuration-reference)
9. [Per-robot specifics](#9-per-robot-specifics)
10. [The fleet test server](#10-the-fleet-test-server)
11. [How a deploy works internally](#11-how-a-deploy-works-internally)
12. [Fleet inventory](#12-fleet-inventory)
13. [Troubleshooting](#13-troubleshooting)
14. [Security notes](#14-security-notes)
---
## 1. What this is
The fleet server (built by YS Lootah — "their side") needs each robot to push its
status outbound over HTTPS. This repo is **the robot side**: a small agent per
robot type that reads local state and POSTs it to documented endpoints, plus the
tooling to install/manage it across the fleet.
- **G1** → uploads the **navigation map**.
- **R1** / **Go2** → stream **telemetry** (battery, charging, status, position, faults).
Each is a drop-in Docker image; install it on any new robot of that type and it
just works.
---
## 2. Directory layout
```
Project/fleet/
fleet_install.sh # interactive + scriptable deploy/manage tool
fleet_test_server.py # your workstation standing in as the fleet server (tests)
agents/
g1/ sanad_api_g1.py — MAP uploader (POST …/{sn}/map)
r1/ sanad_api_r1.py — TELEMETRY unitree_hg (POST …/telemetry)
go2/ sanad_api_go2.py — TELEMETRY unitree_go (POST …/telemetry) [unverified on hw]
├── README.md ← this file
├── PIPELINE.md ← end-to-end data-flow reference
├── fleet_install.sh ← deploy / manage / remove over SSH (interactive + scriptable)
├── fleet_test_server.py ← workstation stand-in for the fleet server (for tests)
└── agents/
├── g1/ ← MAP uploader
│ ├── sanad_api_g1.py
│ ├── Dockerfile (lean: python + requests)
│ ├── requirements.txt
│ ├── docker-compose.yml (manual local use only)
│ └── .env.example
├── r1/ ← TELEMETRY (unitree_hg)
│ ├── sanad_api_r1.py
│ ├── Dockerfile (DDS: CycloneDDS + unitree_sdk2py)
│ ├── vendor/ (unitree_sdk2py wheel + crc libs)
│ └── .env.example
└── go2/ ← TELEMETRY (unitree_go) [unverified on hardware]
├── sanad_api_go2.py
├── Dockerfile
├── vendor/
└── .env.example
```
All three are self-contained (own `Dockerfile`, `.env.example`). Deploy uses
plain `docker build`/`docker run` — **no docker-compose needed on the robot**
(R1/Go2 don't have it). Source is pushed from this canonical workstation copy via
`rsync`; images build natively on each arm64 robot; a **user systemd unit**
(linger-enabled, **no sudo**) owns start/stop/auto-start-on-boot.
`agents/` is the **single source of truth**. The installer rsyncs the selected
agent to the robot; you never edit files on the robot.
## Install / manage
---
Interactive (asks robot type → IP → detects installed-or-not → right actions):
## 3. The three agents
| agent | robot | endpoint | what it sends | cadence |
|---|---|---|---|---|
| `sanad_api_g1` | Unitree G1 | `POST /api/v1/fleet/ingest/{sn}/map` | RTAB-Map `.db` + places (points) | on change (~30 s scan) |
| `sanad_api_r1` | Unitree R1 EDU | `POST /api/v1/fleet/ingest/telemetry` | battery, charging, status, position, faults, mac | every ~2 s |
| `sanad_api_go2` | Unitree Go2 | `POST /api/v1/fleet/ingest/telemetry` | same as R1 | every ~2 s |
All requests carry `Authorization: Bearer <device_token>`. Full payload schemas
and the data pipeline are in [PIPELINE.md](PIPELINE.md).
**Design principles**
- **No ROS in the agents.** G1 reads map files directly; R1/Go2 read DDS via
`unitree_sdk2py`. This keeps images small and portable.
- **Read-only.** Telemetry agents never command motion (only passive DDS reads +,
optionally, the read-only `GET_FSM_ID` RPC).
- **Never crash the loop.** Every tick is wrapped; transient errors are logged and
retried. Telemetry sends a heartbeat when it can't read state so the robot stays
"online".
- **Change-detected uploads.** The map is only re-sent when its content hash
changes.
---
## 4. Prerequisites
**Workstation (the machine you deploy from):**
- `bash`, `ssh`, `rsync`, `python3`.
- SSH **key** access to each robot (`ssh unitree@<ip>` must work without a
password — the installer uses `BatchMode=yes`).
- On the same network as the robots (they POST back to the workstation during
`test`).
**Robot:**
- Docker (Engine ≥ 20). The `unitree` user must be in the `docker` group.
- Internet at build time (pulls the base image; R1/Go2 also `apt`/`pip` the DDS
stack).
- `systemd` with a user bus (standard on Ubuntu 20.04+). **No sudo needed.**
- Architecture: arm64 (Jetson / backpack). Images build natively on the robot.
---
## 5. Quick start
```bash
cd Project/fleet
# Interactive — asks robot type, IP, and (if new) robot name + server:
./fleet_install.sh
# …or scripted:
./fleet_install.sh install r1 10.255.254.82 --sn r1_82 \
--server-ip 10.255.254.83 --port 8799 --token <device-token>
# See what it's sending, tail logs, check the service:
./fleet_install.sh data r1 10.255.254.82
./fleet_install.sh logs r1 10.255.254.82
./fleet_install.sh status r1 10.255.254.82
# End-to-end test against your workstation acting as the server:
./fleet_install.sh test r1 10.255.254.82
# Remove everything:
./fleet_install.sh uninstall r1 10.255.254.82
```
---
## 6. Installer reference
### Interactive mode (no arguments)
```
./fleet_install.sh
```
Scriptable:
Flow:
1. **Which robot?** `1) g1 2) r1 3) go2`
2. **Robot IP** and **SSH user** (default `unitree`; it verifies SSH works).
3. It **detects whether the agent is already installed** (systemd unit file *or*
container present) and branches:
- **Installed** → menu: `1) show data 2) status 3) logs 4) reinstall
5) UNINSTALL 6) quit`.
- **Not installed** → prompts **Robot name (SN)**, **Fleet server IP**
(auto-detected default), **port**, **device token**, then installs.
### Scriptable commands
```
./fleet_install.sh <command> <g1|r1|go2> <ip> [options]
```
| command | action |
|---|---|
| `install` | rsync agent → robot, build image, create container, install + enable systemd service |
| `uninstall` | disable/remove service, remove container, image, and `~/sanad_api_<type>` |
| `status` | systemd service state + container state |
| `data` | recent telemetry/map log lines (what it's currently sending) |
| `logs` | `docker logs -f` (live tail) |
| `test` | start the workstation server, push a real post from the robot, verify receipt (PASS/FAIL) |
### Options
| option | default | meaning |
|---|---|---|
| `--sn NAME` | `<type>_<last-octet>` | robot's fleet id (the `sn` field) |
| `--server-ip IP` | auto (route toward robot) | fleet server the robot posts to |
| `--port N` | `8799` | fleet server port |
| `--token TOK` | `test-token` | device bearer token |
| `--user USER` | `unitree` | SSH user on the robot |
| `--keep-server` | off | (test) leave the workstation test server running |
---
## 7. The auto-start service (systemd)
Because the robots have **no passwordless sudo**, the agent runs as a **user-level
systemd service** (no root needed):
- Unit file: `~/.config/systemd/user/sanad-api-<type>.service`
- Boot auto-start: enabled via `loginctl enable-linger <user>` (allowed without
sudo) so the user manager starts at boot before login.
- The service owns the container lifecycle:
```ini
[Service]
Type=simple
Restart=always
RestartSec=5
ExecStart=/usr/bin/docker start -a sanad-api-<type>
ExecStop=/usr/bin/docker stop -t 10 sanad-api-<type>
```
- The container is created with `docker create` (no docker restart policy) so
**systemd is the single owner** — `Restart=always` also covers the boot race
where the user manager starts before `dockerd` is ready (it retries every 5 s).
**Managing it on the robot:**
```bash
./fleet_install.sh install <g1|r1|go2> <ip> [--sn NAME] [--server-ip IP] [--port N] [--token TOK]
./fleet_install.sh data <g1|r1|go2> <ip> # show what it's currently sending
./fleet_install.sh status <g1|r1|go2> <ip>
./fleet_install.sh logs <g1|r1|go2> <ip>
./fleet_install.sh test <g1|r1|go2> <ip> # end-to-end vs workstation server
./fleet_install.sh uninstall <g1|r1|go2> <ip> # service + container + image + dir
systemctl --user status sanad-api-r1
systemctl --user restart sanad-api-r1
systemctl --user stop sanad-api-r1
journalctl --user -u sanad-api-r1 -f # or: docker logs -f sanad-api-r1
```
`install`/`uninstall` are detected automatically in interactive mode: if the
agent is already on the robot it offers **data / status / logs / reinstall /
uninstall**; otherwise it prompts for **robot name (SN)**, server IP, port, token
and installs.
---
### Auto-start service
## 8. Configuration reference
Each install writes `~/.config/systemd/user/sanad-api-<type>.service`, enables
linger (`loginctl enable-linger`, no sudo), and `systemctl --user enable --now`s
it. It starts on boot and restarts on crash. Manage with:
Config is env-only. The installer writes `~/sanad_api_<type>/.env` on the robot;
`.env.example` in each agent dir documents every key.
### Common (all agents)
| var | required | default | meaning |
|---|---|---|---|
| `SERVER_URL` | ✅ | — | fleet server base URL |
| `DEVICE_TOKEN` | ✅ | — | bearer token (per robot) |
| `SN` | — | `<type>_0000` | robot fleet id |
| `VERIFY_TLS` | — | `1` | verify server TLS cert (`0` for self-signed dev) |
| `HTTP_TIMEOUT` | — | 10–30 | per-request timeout (s) |
| `POLL_INTERVAL` | — | 2 (tel) / 30 (map) | loop cadence (s) |
### G1 map uploader
| var | default | meaning |
|---|---|---|
| `ROBOT` | `sanad` | web_nav3 robot name → maps subdir + `X-Robot-Name` |
| `MAPS_DIR` | `/data/maps` | mounted web_nav3 `maps/` (`<robot>/*.db`) |
| `DATA_DIR` | — | mounted web_nav3 `web/data/` (per-map places) |
| `LEGACY_PLACES` | — | optional legacy `places.json` |
| `WEB_NAV3_URL` | — | optional `http://127.0.0.1:8765` (to learn the active map) |
| `MAP_SELECT` | `all` | `all` · `active` · `newest` |
| `MAP_UPLOAD_MODE` | `multipart` | `multipart` · `base64json` |
| `MAP_ENDPOINT` | `/api/v1/fleet/ingest/{sn}/map` | path template |
| `STATE_DIR` | `/data/state` | upload-fingerprint state |
### R1 / Go2 telemetry
| var | default | meaning |
|---|---|---|
| `DDS_INTERFACE` | R1 `eth10` / Go2 `eth0` | NIC that sees robot DDS |
| `DDS_DOMAIN` | `0` | DDS domain id |
| `MAC_INTERFACE` | = `DDS_INTERFACE` | NIC whose MAC is reported |
| `R1_READ_FSM` | `0` | (R1) read loco FSM for status — read-only GET RPC |
| `R1_POSITION_SOURCE` / `GO2_POSITION_SOURCE` | `none` | `none` · `rosbridge` (· `sportmode` Go2) |
| `ROSBRIDGE_URL` | `ws://127.0.0.1:9090` | position source when rosbridge |
| `LOW_SOC` | `15` | %→ `LOW_BATTERY` fault |
| `MOTOR_TEMP_MAX` | `85` | °C→ `MOTOR_OVERTEMP` fault |
| `TELEMETRY_ENDPOINT` | `/api/v1/fleet/ingest/telemetry` | path |
---
## 9. Per-robot specifics
### G1 (map) — real maps live inside the nav container
The G1's saved maps are stored **inside** the `p4_Foxy_sanad` (Package_4 nav)
container at `/home/unitree/marcus_nav2_test/maps`, which Package_4 does **not**
bind-mount to the host. So the uploader (which mounts a host dir) sees **0 maps**
until you do one of:
1. Add a host bind-mount for `maps/` (and `web/data/`) to the Package_4 `nav`
service, then set the uploader's `MAPS_HOST_DIR`/`DATA_HOST_DIR` to those paths.
2. Put both containers on a shared named volume for the maps dir.
The `test` command seeds a throwaway fixture map so the upload path is verified
regardless.
### R1 (telemetry)
- DDS link is **`eth10`** (= `192.168.123.164`); `wlan0` is the fleet LAN.
- Battery: `rt/lf/bmsstate` (`BmsState_`), `soc` 0–100, charging from `current`
sign. Coexists fine with the running `sanadr1` app (DDS allows many readers).
### Go2 (telemetry) — ⚠️ unverified on hardware
- Uses `unitree_go` DDS. Battery is nested in **`rt/lowstate.bms_state`** (Go2 has
no separate BMS topic).
- Written from the SDK layout; image builds and `unitree_go` imports, simulate is
correct, but **not yet run on a real Go2** — confirm the `bms_state` current sign
(charging polarity) and `sportmodestate` fields on the robot.
---
## 10. The fleet test server
`fleet_test_server.py` stands in for the real fleet server so you can verify a
deploy end-to-end from your workstation.
```bash
ssh unitree@<ip> 'systemctl --user status sanad-api-r1'
ssh unitree@<ip> 'systemctl --user restart sanad-api-r1'
PORT=8799 REQLOG=/tmp/fleet.jsonl python3 fleet_test_server.py
```
## Fleet (this deployment)
- Binds `0.0.0.0:8799`.
- `GET /ping` → `200` (reachability check the installer uses).
- Accepts `POST …/map` and `POST …/telemetry`; logs each request to `REQLOG` and
prints a live summary.
| robot | agent | ip | ssh user | arch | DDS iface |
|---|---|---|---|---|---|
| G1 | map | `10.255.254.58` | `unitree` (key) | arm64 | — |
| R1 | telemetry | `10.255.254.82` | `unitree` (key) | arm64 | `eth10` |
| Go2 | telemetry | *(when available)* | `unitree` | arm64 | `eth0` |
`./fleet_install.sh test …` starts it automatically, triggers a post from the
robot, asserts the payload arrived, and prints `PASS`/`FAIL`.
Workstation fleet server (for tests): **10.255.254.83** via `fleet_test_server.py`.
---
## What each agent sends, and when
## 11. How a deploy works internally
**G1 map** → `POST /api/v1/fleet/ingest/{sn}/map` — on change (scan ~30 s;
size+mtime→sha256). Sends the RTAB-Map `.db` + places as `multipart` (file `db` +
`meta` JSON `{sn,name,format:"rtabmap_db",size_bytes,sha256,points[…]}`) or
`base64json`. Reads web_nav3 `maps/<robot>/*.db` + `web/data/<robot>/places/*.json`.
**R1 / Go2 telemetry** → `POST /api/v1/fleet/ingest/telemetry` — every ~2 s;
heartbeat (`battery:null,status:offline`) when DDS is silent so it stays online:
```json
{ "sn":"r1_82", "mac":"…", "battery":74, "charging":false,
"status":"idle", "position":{"x":…,"y":…}|null, "faults":[], "ts":… }
```
- R1 (`unitree_hg`): battery `rt/lf/bmsstate`, faults/liveness `rt/lowstate`.
- Go2 (`unitree_go`): battery from **`rt/lowstate.bms_state`** (nested), no separate BMS topic.
- Both: `status` derived (charging/moving/idle/offline); position optional; read-only (never moves).
install:
rsync agents/<type>/ → unitree@<ip>:~/sanad_api_<type>/ (--delete, minus .env/state)
write ~/sanad_api_<type>/.env (SERVER_URL, TOKEN, SN, iface…)
ssh: docker build -t sanad-api-<type>:latest . (native arm64)
ssh: docker create --name sanad-api-<type> <run-args> … (no docker restart policy)
write ~/.config/systemd/user/sanad-api-<type>.service
ssh: loginctl enable-linger ; systemctl --user enable --now …
Auth on every request: `Authorization: Bearer <device_token>`.
run-args: --network host --env-file …/.env
(g1 also: -v maps:ro -v web_data:ro -v state)
```
## Notes / follow-ups
`--network host` is required: G1 to reach `127.0.0.1:8765` (web_nav3) if used, and
R1/Go2 so the robot's DDS traffic is visible (DDS multicast doesn't cross a NAT
bridge).
- **Point at the real fleet server:** re-run `install … --server-ip <fleet> --token <real> --sn <id>`.
- **G1 real maps** live *inside* the `p4_Foxy_sanad` nav container (not host-mounted).
Add a `maps/` bind-mount to Package_4's nav service, or share a volume, so the
uploader sees real maps (it reports 0 until then; `test` seeds a fixture).
- **Go2** agent is written from the `unitree_go` SDK layout but **not yet run on a
Go2** — confirm the `bms_state` current sign (charging) and sportmodestate fields.
---
## 12. Fleet inventory
| robot | agent | IP | SSH | arch | DDS iface | SN |
|---|---|---|---|---|---|---|
| G1 | map | `10.255.254.58` | `unitree` (key) | arm64 | — | `g1_7892` |
| R1 | telemetry | `10.255.254.82` | `unitree` (key) | arm64 | `eth10` | `r1_82` |
| Go2 | telemetry | *(TBD)* | `unitree` | arm64 | `eth0` | — |
Workstation (deploy host + test fleet server): **`10.255.254.83`** (`wlp4s0`).
---
## 13. Troubleshooting
| symptom | cause / fix |
|---|---|
| `robot cannot reach the workstation server (http 000)` during `test` | robot→workstation blocked. Check both are on the same subnet; open the port if a firewall is on (`sudo ufw allow 8799`). |
| G1 uploads nothing / "0 maps" | no saved map, or maps aren't mounted — see [§9 G1](#g1-map--real-maps-live-inside-the-nav-container). |
| R1/Go2 `telemetry POST failed … Connection refused` | the target server (e.g. the test server) is down. Point at the real server: `install … --server-ip <fleet> --token <real>`. |
| R1/Go2 `battery=null, status=offline` forever | DDS not seen. Wrong `DDS_INTERFACE` (R1 = `eth10`) or robot firmware down. Confirm with `ip -o addr` on the robot. |
| build is very slow (R1/Go2) | first build compiles CycloneDDS (minutes). Run detached and poll: `ssh … 'setsid bash -c "cd ~/sanad_api_r1 && docker build -t sanad-api-r1:latest . > build.log 2>&1" </dev/null &'` then watch `build.log`. Docker layer cache survives reboots. |
| service didn't come up after reboot | linger must be on: `loginctl show-user <user> | grep Linger` → `Linger=yes`. The installer sets it. |
| `unbound variable` / weird prompt behavior | ensure you're on the current `fleet_install.sh` (uses `ssh -n` for command calls so SSH doesn't eat stdin). |
---
## 14. Security notes
- **Outbound only.** Agents open no inbound ports on the robot; they POST out over
HTTPS with a per-robot bearer token.
- **Least privilege.** Runs as the `unitree` user (docker group), user-level
systemd, no root. Read-only w.r.t. the robot (no motion commands).
- **Token handling.** `.env` holds the device token and is never rsynced back or
committed (`.gitignore`d). The installer writes it directly to the robot.
- **TLS.** `VERIFY_TLS=1` in production; `0` only for the local test server.