fleet/README.md

# Sanad Fleet Agents

On-robot agents that report each Unitree robot's state to the **YS Lootah fleet
server**. One agent per robot type (**G1 · R1 · Go2**), each a self-contained
**Docker** image with a user-level **systemd auto-start service**, deployed and
managed over SSH by a single script — **no `docker-compose`, no `sudo`** on the robot.

![Sanad Fleet — one agent, five endpoints](diagram.svg)

> Full topic-by-topic data flow and payload schema: **[PIPELINE.md](PIPELINE.md)**.

---

## At a glance

- **One agent = one service = everything.** Each robot runs a single container that
  streams telemetry **and** uploads maps, ships logs, raises alerts, and registers
  its dashboard — across **five endpoints**.
- **Read-only toward the robot.** It reads DDS + files; it **never commands motion**.
- **Outbound HTTPS only.** No inbound port is opened on the robot; every request
  carries `Authorization: Bearer <token>` and is keyed by the robot's serial (`sn`).
- **One source of truth.** `agents/g1/` is canonical; `r1` and `go2` are **generated**
  from it (`tools/gen_agents.py`). You never hand-edit r1/go2, and never edit on the robot.

| endpoint | when | what |
|---|---|---|
| `POST …/ingest/telemetry` | every ~2 s | 27 fields — identity, software/firmware, battery(+detail), motor temps, storage, status, position, **control mode**, faults, + status of map/logs/alerts/remote |
| `POST …/ingest/{sn}/map` | on change | saved nav map → PNG + resolution + origin (also RTAB-Map `.db`) |
| `POST …/ingest/{sn}/alert` | on new fault | battery ≤ 50 %, Gemini-billing, any error (as strings) |
| `POST …/ingest/{sn}/logs` | every 60 s | the agent's own lines **+ the robot's Sanad app logs** |
| `POST …/ingest/{sn}/remote` | every 60 s | dashboard URL (`web`) + `ssh unitree@<ip>` (`ssh`) |

*(A `GET …/ingest/{sn}/commands` channel also exists; remote mode-**switching** is
deliberately **not** built — status only. See [§2](#2-the-agent).)*

---

## Table of contents

1. [Directory layout](#1-directory-layout)
2. [The agent](#2-the-agent)
3. [What it reports (the 27 fields)](#3-what-it-reports-the-27-fields)
4. [Maps: discovery & upload](#4-maps-discovery--upload)
5. [Prerequisites](#5-prerequisites)
6. [Quick start](#6-quick-start)
7. [How a deploy works](#7-how-a-deploy-works)
8. [Installer reference](#8-installer-reference)
9. [The auto-start service (systemd)](#9-the-auto-start-service-systemd)
10. [Configuration reference](#10-configuration-reference)
11. [Per-robot specifics](#11-per-robot-specifics)
12. [The fleet test server](#12-the-fleet-test-server)
13. [Fleet inventory](#13-fleet-inventory)
14. [Troubleshooting](#14-troubleshooting)
15. [Security notes](#15-security-notes)

---

## 1. Directory layout

```
Project/fleet/
├── README.md                 ← this file
├── PIPELINE.md               ← end-to-end data-flow + payload schema
├── diagram.svg  deploy.svg   ← the two diagrams in this doc
├── fleet_install.sh          ← deploy / manage / remove over SSH (interactive + scriptable)
├── fleet_test_server.py      ← workstation stand-in for the fleet server (for tests)
├── tools/
│   └── gen_agents.py         ← regenerates r1 + go2 from the canonical g1
├── tests/                    ← mock capture + one-shot map export helpers
└── agents/
    ├── g1/                   ← CANONICAL source (edit here)
    │   ├── sanad_api_g1.py
    │   ├── Dockerfile        (lean: python + requests)
    │   ├── requirements.txt · docker-compose.yml (local use only) · .env.example
    ├── r1/                   ← generated · unitree_hg · R1 FSM ids · eth10
    │   ├── sanad_api_r1.py · Dockerfile (DDS: CycloneDDS + unitree_sdk2py) · vendor/ · .env.example
    └── go2/                  ← generated · unitree_go · ⚠ unverified on hardware
        └── sanad_api_go2.py · Dockerfile · vendor/ · .env.example
```

> **Golden rule:** edit `agents/g1/`, then run `python3 tools/gen_agents.py` to
> regenerate `r1` and `go2`. The installer rsyncs the selected agent to the robot;
> files on the robot are never edited by hand.

---

## 2. The agent

| agent (type) | robot | DDS family | notes |
|---|---|---|---|
| `sanad_api_g1` (`g1`) | Unitree G1 | `unitree_hg` | **canonical source** |
| `sanad_api_r1` (`r1`) | Unitree R1 EDU | `unitree_hg` | generated from g1 · R1 FSM ids `0/1/4/811` · `eth10` |
| `sanad_api_go2` (`go2`) | Unitree Go2 | `unitree_go` | generated from g1 · battery nested in `LowState.bms_state` · ⚠ unverified on hardware |

All three run the **same five loops** (telemetry 2 s · map 30 s · alert scan 10 s ·
log ship 60 s · remote register 60 s). They differ only in the DDS layer and a few
robot-specific ids — which is exactly what the generator handles.

**Design principles**

- **No ROS.** Maps are read from files; robot state from DDS via `unitree_sdk2py`.
- **Read-only.** The control panel reports the loco mode but **cannot switch it** —
  mode switching can drop the robot, so it is intentionally not built (`CONTROL_ENABLE=0`).
- **Never crash the loop.** Every tick is wrapped; a heartbeat keeps the robot
  "online" when state is momentarily unreadable.
- **Idempotent uploads.** Maps re-send only when their content changes.
- **Resilient shipping.** Failed log ships are requeued (last ~400 lines) until the
  server accepts them.

---

## 3. What it reports (the 27 fields)

Every telemetry POST is one JSON object. Grouped for readability:

| group | fields |
|---|---|
| **identity** | `sn` · `name` · `mac` · `brand` · `type` · `model` |
| **system** | `software{ros,os,os_version,kernel,arch,python,agent}` · `firmware{board,l4t,kernel,robot,bms}` |
| **power** | `battery` · `charging` · `battery_detail{voltage_v,current_a,temp_c,soh,cycles}` |
| **health** | `motor_temp{max,avg,min}` *(null = not receiving)* · `storage{total_gb,free_gb,used_percent}` · `faults[]` *(strings)* |
| **state** | `status` *(idle/moving/charging/offline)* · `position{x,y}` · `control{fsm_id,mode,armed,walk_ready,teleop_active,…}` |
| **sub-status** | `map{…}` · `logs{…}` · `project_logs` · `remote{…}` · `alerts{…}` |
| **timing** | `time` · `started_at` · `last_start` · `uptime_s` · `ts` |

`control.mode` is a friendly label of the loco FSM (R1: `zero_torque`/`damp`/`lock`/
`running`; G1: `running`/`lock`/`squat`/…), read from the Sanad dashboard status API
— no DDS command, no motion. Full schema in **[PIPELINE.md](PIPELINE.md)**.

---

## 4. Maps: discovery & upload

The agent finds saved maps on the robot and uploads each **once per content change**.
It understands two formats:

- **`slam_toolbox` / Nav2 / Pudu sets** — `map.pgm` + `map.yaml` → rendered to an
  image JSON (pure-stdlib PGM→PNG + resolution + origin). Small; always uploads.
- **RTAB-Map `.db`** — sent as-is, but **skipped if larger than the server's ~8 MB
  cap** (`MAP_MAX_UPLOAD_MB`, default 7) with a note in `map.error`.

**Where it looks** (each existing dir is scanned for `*.pgm` + `*.yaml` and `*.db`):

- `MAPS_DIR` (`/data/maps`) — the robot's primary SLAM/nav maps dir (installer probes it).
- `EXTRA_MAP_DIRS` (`/data/nav2_maps`) — a **separate Nav2/Pudu deploy-maps dir**
  (e.g. `~/r1_nav2_docker/maps`) mounted alongside. The installer auto-detects and
  mounts it, so converted **Pudu office maps** are discovered and uploaded too.

Two conveniences:

- The Pudu converter emits a plain map **and** a keepout-**baked** twin; when both
  exist the agent keeps only the baked one, so the server gets **one** canonical map.
- `--map-only` uploads discovered maps once with **no DDS and no telemetry** — handy
  to push a new map without disturbing a live feed:
  ```bash
  docker run --rm --network host --env-file .env \
      -e EXTRA_MAP_DIRS=/data/nav2_maps -v <host-maps>:/data/nav2_maps:ro \
      sanad-api-<type>:latest --map-only --force
  ```

Map status is mirrored into every telemetry post as
`map:{uploaded,state,maps_found,last_map,error,checked_ts}`.

---

## 5. Prerequisites

**Workstation (deploy host)**
- `bash`, `ssh`, `rsync`, `python3`.
- SSH **key** access to each robot — `ssh unitree@<ip>` must work without a password
  (the installer uses `BatchMode=yes`).
- On the same network as the robots (they POST back to the workstation during `test`).

**Robot**
- Docker (Engine ≥ 20); the `unitree` user in the `docker` group.
- Internet at build time (base image; r1/go2 also build the DDS stack).
- `systemd` **user** bus (standard on Ubuntu 20.04+). **No sudo needed.**
- Architecture: arm64 (Jetson / backpack). Images build natively on the robot.

---

## 6. Quick start

```bash
cd Project/fleet

# Interactive — asks robot type, IP, and (if new) name + token + server:
./fleet_install.sh

# …or scripted against the real fleet server:
./fleet_install.sh install r1 10.255.254.82 \
    --sn E39N4000Q6D7E70F --name r1_82 \
    --token <device-token> --server-url https://eco.yslootahrobotics.com

# Inspect / manage:
./fleet_install.sh data    r1 10.255.254.82   # what it's currently sending
./fleet_install.sh logs    r1 10.255.254.82   # live docker logs
./fleet_install.sh status  r1 10.255.254.82   # systemd + container state
./fleet_install.sh test    r1 10.255.254.82   # e2e against the workstation server
./fleet_install.sh uninstall r1 10.255.254.82 # remove everything
```

---

## 7. How a deploy works

![Deploy — workstation to robot over SSH](deploy.svg)

```
rsync agents/<type>/  →  unitree@<ip>:~/sanad_api_<type>/   (--delete; keeps .env + state)
write ~/sanad_api_<type>/.env                               (SERVER_URL, TOKEN, SN, iface, …)
ssh: docker build  -t sanad-api-<type>:latest .             (native arm64)
ssh: docker create --name sanad-api-<type> <run-args>       (no docker restart policy)
write ~/.config/systemd/user/sanad-api-<type>.service
ssh: loginctl enable-linger ; systemctl --user enable --now sanad-api-<type>
```

**run-args** (all agents): `--network host --env-file …/.env` plus read-only mounts:
`-v /:/host:ro` (real disk/firmware + Sanad json-log), `-v <maps>:/data/maps:ro`,
`-v <web_data>:/data/web_data:ro`, `-v <nav2 maps>:/data/nav2_maps:ro` *(when present)*,
and `-v …/state:/data/state` (rw, upload fingerprints).

`--network host` is required so the robot's DDS multicast is visible (it doesn't cross
a NAT bridge) and so the agent can reach the Sanad dashboard on `127.0.0.1`.

---

## 8. Installer reference

### Interactive (no arguments)

```
./fleet_install.sh
```

1. **Which robot?** `1) g1  2) r1  3) go2`
2. **Robot IP** + **SSH user** (default `unitree`; verifies SSH works).
3. **Detects whether it's already installed** (unit file *or* container) and branches:
   - **Installed** → menu: show data · status · logs · reinstall · **uninstall** · quit.
   - **Not installed** → prompts **SN (required)**, **server** (URL or auto-detected IP),
     **token**, then installs.

### Scriptable

```
./fleet_install.sh <command> <g1|r1|go2> <ip> [options]
```

| command | action |
|---|---|
| `install` | rsync → build → create → install + enable the systemd service |
| `uninstall` | disable/remove service, container, image, and `~/sanad_api_<type>` |
| `status` | systemd service state + container state |
| `data` | recent telemetry/map log lines (what it's sending) |
| `logs` | `docker logs -f` (live tail) |
| `test` | start the workstation server, push a real post from the robot, verify (PASS/FAIL) |

`install` requires the essentials to be explicit:

```bash
./fleet_install.sh install <type> <ip> --sn <serial> --token <token> \
    --server-url https://eco.yslootahrobotics.com [--post <path>] [--name <display>]
```

| option | default | meaning |
|---|---|---|
| `--sn SERIAL` | **required** | robot's REAL serial — keys it on the server (e.g. `E39N4000Q6D7E70F`) |
| `--name NAME` | `<model>_<last-octet>` | friendly display name (`r1_82`, `g1_58`) |
| `--token TOK` | `test-token` | device bearer token |
| `--server-url URL` | — | full HTTPS fleet server → `VERIFY_TLS=1` |
| `--post PATH` | agent default | override the telemetry ingest path |
| `--server-ip IP` | auto (route to robot) | use the local test server instead → `VERIFY_TLS=0` |
| `--port N` | `8799` | test-server port (with `--server-ip`) |
| `--user USER` | `unitree` | SSH user on the robot |
| `--keep-server` | off | (test) leave the workstation test server running |

---

## 9. The auto-start service (systemd)

Because the robots have **no passwordless sudo**, the agent runs as a **user-level**
systemd service (no root):

- Unit: `~/.config/systemd/user/sanad-api-<type>.service`
- Boot auto-start via `loginctl enable-linger <user>` (allowed without sudo).
- systemd owns the container lifecycle:
  ```ini
  [Service]
  Type=simple
  Restart=always
  RestartSec=5
  ExecStart=/usr/bin/docker start -a sanad-api-<type>
  ExecStop=/usr/bin/docker stop -t 10 sanad-api-<type>
  ```
- Created with `docker create` (no docker restart policy) so **systemd is the single
  owner**; `Restart=always` also covers the boot race where the user manager starts
  before `dockerd` is ready.

**On the robot:**
```bash
systemctl --user status  sanad-api-r1
systemctl --user restart sanad-api-r1
journalctl --user -u sanad-api-r1 -f      # or: docker logs -f sanad-api-r1
```

---

## 10. Configuration reference

Config is env-only; the installer writes `~/sanad_api_<type>/.env`. Every key is
documented in each agent's `.env.example`.

### Core (all agents)

| var | default | meaning |
|---|---|---|
| `SERVER_URL` ✅ | — | fleet server base URL |
| `DEVICE_TOKEN` ✅ | — | bearer token (per robot) |
| `SN` | `<type>_0000` | robot's real serial (server key) |
| `ROBOT_NAME` | `<model>_<octet>` | friendly display name |
| `ROBOT_BRAND` / `ROBOT_TYPE` / `ROBOT_MODEL` | `unitree` / `humanoid`\|`dog` / `g1`\|`r1`\|`go2` | identity |
| `POLL_INTERVAL` | `2` | telemetry cadence (s) |
| `VERIFY_TLS` | `1` | verify server TLS (`0` for the local test server) |
| `HTTP_TIMEOUT` | `30` | per-request timeout (s) |
| `SOFTWARE_ROS` | `foxy` | reported in `software.ros` |

### DDS / telemetry

| var | default | meaning |
|---|---|---|
| `DDS_INTERFACE` | r1 `eth10` · else `eth0` | NIC that sees robot DDS |
| `DDS_DOMAIN` | `0` | DDS domain id |
| `MAC_INTERFACE` | = `DDS_INTERFACE` | NIC whose MAC is reported |
| `G1_READ_FSM` | `0` | read loco FSM for status (read-only GET RPC) |
| `POSITION_SOURCE` | `odom`\|`none` | `odom` (DDS) · `rosbridge` · `none` |
| `ROSBRIDGE_URL` | `ws://127.0.0.1:9090` | position source when `rosbridge` |
| `MOTOR_TEMP_MAX` | `85` | °C → `MOTOR_OVERTEMP` fault |

### Maps

| var | default | meaning |
|---|---|---|
| `MAPS_DIR` | `/data/maps` | primary SLAM/nav maps mount |
| `EXTRA_MAP_DIRS` | `/data/nav2_maps` | extra roots (colon-sep) for Nav2/Pudu `pgm+yaml` sets |
| `MAP_SELECT` | `all` | `all` · `active` · `newest` |
| `MAP_POLL_INTERVAL` | `30` | map check cadence (s) |
| `MAP_MAX_UPLOAD_MB` | `7` | skip `.db` larger than this (server cap) |
| `MAP_UPLOAD_MODE` | `multipart` | `multipart` · `base64json` |
| `DATA_DIR` | `/data/web_data` | per-map places (waypoints) |
| `STATE_DIR` | `/data/state` | upload-fingerprint + start-time state |

### Alerts

| var | default | meaning |
|---|---|---|
| `LOW_SOC` | `50` | battery % → `LOW_BATTERY` fault + alert |
| `ALERT_SCAN_INTERVAL` | `10` | project-log scan cadence (s) |
| `ALERT_LOG_COOLDOWN` | `300` | per-signature re-alert gap (s) |
| `ALERT_LOG_PATTERNS` | Gemini-billing + error/critical/traceback | `CODE=regex` entries split by `;;` (case-sensitive; use `(?i)` inline) |
| `ALERT_BACKFILL_BYTES` | `8 MiB` | startup scan window so active errors alert on boot |

### Logs · remote · control

| var | default | meaning |
|---|---|---|
| `LOGS_INTERVAL` | `60` | log-ship cadence (s) |
| `PROJECT_LOG_CONTAINER` | `auto` | robot's Sanad container to tail (`auto` discovers it) |
| `PROJECT_LOG_PATH` / `_LABEL` / `_BACKFILL` / `_EXCLUDE` | — / — / `100` / access-log filter | project-log overrides |
| `REMOTE_ENABLE` | `1` | register the dashboard URL + ssh |
| `REMOTE_PORTS` | `8001,8014,8011-8013,8000,8080` | dashboard ports to probe |
| `REMOTE_URL` | — | pin an explicit URL (e.g. a public tunnel) |
| `SSH_REGISTER` / `SSH_USER` | `1` / `unitree` | register `ssh <user>@<ip>` |
| `CONTROL_ENABLE` | `0` | remote mode-**switch** (motion) — **off by design** |
| `CONTROL_STATUS_URL` | — | Sanad `/api/controller/status` for read-only loco mode |

---

## 11. Per-robot specifics

### R1 (live on production)
- DDS link is **`eth10`** (= `192.168.123.164`); WiFi is the fleet LAN.
- Battery from `rt/lf/bmsstate` (`BmsState_`); coexists with the running `sanadr1`
  app (DDS allows many readers).
- Maps: VSLAM `~/r1_vslam_docker/data` (RTAB-Map `.db`) **and** the converted
  **Pudu office map** in `~/r1_nav2_docker/maps` (mounted at `/data/nav2_maps`).

### G1 (⚠ STANDING HOLD — stage only, do not deploy)
- Do **not** deploy to the new G1 until the hold is lifted; stage maps only.
- Saved maps live **inside** the `p4_Foxy_sanad` nav container at
  `~/marcus_nav2_test/maps` (not host-mounted by default) — add a host bind-mount
  (and to `web/data/`) so the agent can read them, or use a shared volume.

### Go2 (⚠ unverified on hardware)
- Uses `unitree_go`; battery is nested in **`rt/lowstate.bms_state`** (no separate
  BMS topic). Image builds and imports, simulate is correct — but confirm the
  `bms_state` current sign (charge polarity) and `sportmodestate` fields on a real Go2.

---

## 12. The fleet test server

`fleet_test_server.py` stands in for the real fleet server so you can verify a deploy
end-to-end from the workstation.

```bash
PORT=8799 REQLOG=/tmp/fleet.jsonl python3 fleet_test_server.py
```

- Binds `0.0.0.0:8799`; `GET /ping` → `200` (the reachability check the installer uses).
- Accepts every ingest POST, logs each to `REQLOG`, prints a live summary.

`./fleet_install.sh test …` starts it, triggers a post from the robot, asserts the
payload arrived, and prints **PASS**/**FAIL**.

---

## 13. Fleet inventory

| robot | IP | SSH | arch | DDS iface | SN (name) | status |
|---|---|---|---|---|---|---|
| R1 | `10.255.254.82` (wired `192.168.123.164`) | `unitree` (key) | arm64 | `eth10` | `E39N4000Q6D7E70F` (`r1_82`) | **live on prod** · office map uploaded |
| G1 | `10.255.254.58` | `unitree` (key) | arm64 | `eth0` | `E21D6000PB89GF88` (`g1_58`) | ⚠ hold + offline (battery) |
| Go2 | *(TBD)* | `unitree` | arm64 | `eth0` | — | unverified |

Workstation (deploy host + test fleet server): **`10.255.254.83`** (`wlp4s0`).
Production fleet server: **`https://eco.yslootahrobotics.com`**.

---

## 14. Troubleshooting

| symptom | cause / fix |
|---|---|
| `test`: robot can't reach the workstation (`http 000`) | same-subnet? open the port (`sudo ufw allow 8799`). |
| map: "0 maps" / nothing uploaded | no saved map, or maps aren't mounted — see [§4](#4-maps-discovery--upload) and [§11 G1](#g1--standing-hold--stage-only-do-not-deploy). |
| `telemetry POST failed … Connection refused` | target server down / wrong URL. Re-point: `install … --server-url <real>`. |
| `battery=null, status=offline` forever | DDS not seen. Wrong `DDS_INTERFACE` (R1 = `eth10`) or robot firmware down — check `ip -o addr` on the robot. |
| build very slow (r1/go2) | first build compiles CycloneDDS (minutes). Run detached: `setsid bash -c 'cd ~/sanad_api_r1 && docker build -t sanad-api-r1:latest . >build.log 2>&1' </dev/null &` then watch `build.log`. Layer cache survives reboots. |
| service didn't start after reboot | linger must be on: `loginctl show-user <user> \| grep Linger` → `Linger=yes` (the installer sets it). |
| r1/go2 out of sync with g1 | you edited g1 without regenerating — run `python3 tools/gen_agents.py`. |

---

## 15. Security notes

- **Outbound only.** No inbound port on the robot; POSTs out over HTTPS with a
  per-robot bearer token.
- **Least privilege.** Runs as `unitree` (docker group), user-level systemd, no root.
  Read-only toward the robot — **no motion commands**.
- **Token handling.** `.env` holds the device token, is never rsynced back, and is
  `.gitignore`d. The installer writes it straight to the robot; it is never printed.
- **TLS.** `VERIFY_TLS=1` in production; `0` only for the local test server.