fleet/README.md

# Sanad Fleet Agents

On-robot agents that report each robot's state to the **YS Lootah fleet server**.
Every robot type ships as its own self-contained **Docker** image with a
user-level **systemd auto-start service**, deployed, managed, and removed over
SSH by a single script. No `docker-compose` on the robot, **no `sudo`** required.

![Sanad Fleet — architecture and data pipeline](diagram.svg)

> For the end-to-end data flow (topics → agent → HTTP → server), see
> [PIPELINE.md](PIPELINE.md).

---

## Table of contents

1. [What this is](#1-what-this-is)
2. [Directory layout](#2-directory-layout)
3. [The three agents](#3-the-three-agents)
4. [Prerequisites](#4-prerequisites)
5. [Quick start](#5-quick-start)
6. [Installer reference](#6-installer-reference)
7. [The auto-start service (systemd)](#7-the-auto-start-service-systemd)
8. [Configuration reference](#8-configuration-reference)
9. [Per-robot specifics](#9-per-robot-specifics)
10. [The fleet test server](#10-the-fleet-test-server)
11. [How a deploy works internally](#11-how-a-deploy-works-internally)
12. [Fleet inventory](#12-fleet-inventory)
13. [Troubleshooting](#13-troubleshooting)
14. [Security notes](#14-security-notes)

---

## 1. What this is

The fleet server (built by YS Lootah — "their side") needs each robot to push its
status outbound over HTTPS. This repo is **the robot side**: a small agent per
robot type that reads local state and POSTs it to documented endpoints, plus the
tooling to install/manage it across the fleet.

- **G1** → uploads the **navigation map**.
- **R1** / **Go2** → stream **telemetry** (battery, charging, status, position, faults).

Each is a drop-in Docker image; install it on any new robot of that type and it
just works.

---

## 2. Directory layout

```
Project/fleet/
├── README.md                 ← this file
├── PIPELINE.md               ← end-to-end data-flow reference
├── fleet_install.sh          ← deploy / manage / remove over SSH (interactive + scriptable)
├── fleet_test_server.py      ← workstation stand-in for the fleet server (for tests)
└── agents/
    ├── g1/                   ← MAP uploader
    │   ├── sanad_api_g1.py
    │   ├── Dockerfile        (lean: python + requests)
    │   ├── requirements.txt
    │   ├── docker-compose.yml (manual local use only)
    │   └── .env.example
    ├── g1t/                  ← TELEMETRY (G1, unitree_hg)
    │   ├── sanad_api_g1t.py
    │   ├── Dockerfile        (DDS) + vendor/
    │   └── .env.example
    ├── r1/                   ← TELEMETRY (unitree_hg)
    │   ├── sanad_api_r1.py
    │   ├── Dockerfile        (DDS: CycloneDDS + unitree_sdk2py)
    │   ├── vendor/           (unitree_sdk2py wheel + crc libs)
    │   └── .env.example
    └── go2/                  ← TELEMETRY (unitree_go)  [unverified on hardware]
        ├── sanad_api_go2.py
        ├── Dockerfile
        ├── vendor/
        └── .env.example
```

`agents/` is the **single source of truth**. The installer rsyncs the selected
agent to the robot; you never edit files on the robot.

---

## 3. The three agents

| agent (type) | robot | endpoint | what it sends | cadence |
|---|---|---|---|---|
| `sanad_api_g1` (`g1`) | Unitree G1 | `POST /api/v1/fleet/ingest/{sn}/map` | RTAB-Map `.db` + places (points) | on change (~30 s scan) |
| `sanad_api_g1t` (`g1t`) | Unitree G1 | `POST /api/v1/fleet/ingest/telemetry` | battery, charging, status, position, faults, mac | every ~2 s |
| `sanad_api_r1` (`r1`) | Unitree R1 EDU | `POST /api/v1/fleet/ingest/telemetry` | same telemetry | every ~2 s |
| `sanad_api_go2` (`go2`) | Unitree Go2 | `POST /api/v1/fleet/ingest/telemetry` | same telemetry | every ~2 s |

The G1 runs **both** `g1` (map) and `g1t` (telemetry) — two independent
containers/services, both keyed by the **same `sn`** (same physical robot). `g1t`
uses the same `unitree_hg` DDS as R1, and reads `position` from the firmware odom
topic (`rt/lf/odommodestate`) over DDS.

All requests carry `Authorization: Bearer <device_token>`. Full payload schemas
and the data pipeline are in [PIPELINE.md](PIPELINE.md).

**Design principles**

- **No ROS in the agents.** G1 reads map files directly; R1/Go2 read DDS via
  `unitree_sdk2py`. This keeps images small and portable.
- **Read-only.** Telemetry agents never command motion (only passive DDS reads +,
  optionally, the read-only `GET_FSM_ID` RPC).
- **Never crash the loop.** Every tick is wrapped; transient errors are logged and
  retried. Telemetry sends a heartbeat when it can't read state so the robot stays
  "online".
- **Change-detected uploads.** The map is only re-sent when its content hash
  changes.

---

## 4. Prerequisites

**Workstation (the machine you deploy from):**
- `bash`, `ssh`, `rsync`, `python3`.
- SSH **key** access to each robot (`ssh unitree@<ip>` must work without a
  password — the installer uses `BatchMode=yes`).
- On the same network as the robots (they POST back to the workstation during
  `test`).

**Robot:**
- Docker (Engine ≥ 20). The `unitree` user must be in the `docker` group.
- Internet at build time (pulls the base image; R1/Go2 also `apt`/`pip` the DDS
  stack).
- `systemd` with a user bus (standard on Ubuntu 20.04+). **No sudo needed.**
- Architecture: arm64 (Jetson / backpack). Images build natively on the robot.

---

## 5. Quick start

```bash
cd Project/fleet

# Interactive — asks robot type, IP, and (if new) robot name + server:
./fleet_install.sh

# …or scripted:
./fleet_install.sh install r1 10.255.254.82 --sn r1_82 \
    --server-ip 10.255.254.83 --port 8799 --token <device-token>

# See what it's sending, tail logs, check the service:
./fleet_install.sh data   r1 10.255.254.82
./fleet_install.sh logs   r1 10.255.254.82
./fleet_install.sh status r1 10.255.254.82

# End-to-end test against your workstation acting as the server:
./fleet_install.sh test   r1 10.255.254.82

# Remove everything:
./fleet_install.sh uninstall r1 10.255.254.82
```

---

## 6. Installer reference

### Interactive mode (no arguments)

```
./fleet_install.sh
```

Flow:
1. **Which robot?** `1) g1  2) r1  3) go2`
2. **Robot IP** and **SSH user** (default `unitree`; it verifies SSH works).
3. It **detects whether the agent is already installed** (systemd unit file *or*
   container present) and branches:
   - **Installed** → menu: `1) show data  2) status  3) logs  4) reinstall
     5) UNINSTALL  6) quit`.
   - **Not installed** → prompts **Robot name (SN)**, **Fleet server IP**
     (auto-detected default), **port**, **device token**, then installs.

### Scriptable commands

```
./fleet_install.sh <command> <g1|r1|go2> <ip> [options]
```

| command | action |
|---|---|
| `install` | rsync agent → robot, build image, create container, install + enable systemd service |
| `uninstall` | disable/remove service, remove container, image, and `~/sanad_api_<type>` |
| `status` | systemd service state + container state |
| `data` | recent telemetry/map log lines (what it's currently sending) |
| `logs` | `docker logs -f` (live tail) |
| `test` | start the workstation server, push a real post from the robot, verify receipt (PASS/FAIL) |

### Options

| option | default | meaning |
|---|---|---|
| `--sn NAME` | `<type>_<last-octet>` | robot's fleet id (the `sn` field) |
| `--server-ip IP` | auto (route toward robot) | fleet server the robot posts to |
| `--port N` | `8799` | fleet server port |
| `--token TOK` | `test-token` | device bearer token |
| `--user USER` | `unitree` | SSH user on the robot |
| `--keep-server` | off | (test) leave the workstation test server running |

---

## 7. The auto-start service (systemd)

Because the robots have **no passwordless sudo**, the agent runs as a **user-level
systemd service** (no root needed):

- Unit file: `~/.config/systemd/user/sanad-api-<type>.service`
- Boot auto-start: enabled via `loginctl enable-linger <user>` (allowed without
  sudo) so the user manager starts at boot before login.
- The service owns the container lifecycle:
  ```ini
  [Service]
  Type=simple
  Restart=always
  RestartSec=5
  ExecStart=/usr/bin/docker start -a sanad-api-<type>
  ExecStop=/usr/bin/docker stop -t 10 sanad-api-<type>
  ```
- The container is created with `docker create` (no docker restart policy) so
  **systemd is the single owner** — `Restart=always` also covers the boot race
  where the user manager starts before `dockerd` is ready (it retries every 5 s).

**Managing it on the robot:**

```bash
systemctl --user status  sanad-api-r1
systemctl --user restart sanad-api-r1
systemctl --user stop    sanad-api-r1
journalctl --user -u sanad-api-r1 -f      # or: docker logs -f sanad-api-r1
```

---

## 8. Configuration reference

Config is env-only. The installer writes `~/sanad_api_<type>/.env` on the robot;
`.env.example` in each agent dir documents every key.

### Common (all agents)

| var | required | default | meaning |
|---|---|---|---|
| `SERVER_URL` | ✅ | — | fleet server base URL |
| `DEVICE_TOKEN` | ✅ | — | bearer token (per robot) |
| `SN` | — | `<type>_0000` | robot fleet id |
| `VERIFY_TLS` | — | `1` | verify server TLS cert (`0` for self-signed dev) |
| `HTTP_TIMEOUT` | — | 10–30 | per-request timeout (s) |
| `POLL_INTERVAL` | — | 2 (tel) / 30 (map) | loop cadence (s) |

### G1 map uploader

| var | default | meaning |
|---|---|---|
| `ROBOT` | `sanad` | web_nav3 robot name → maps subdir + `X-Robot-Name` |
| `MAPS_DIR` | `/data/maps` | mounted web_nav3 `maps/` (`<robot>/*.db`) |
| `DATA_DIR` | — | mounted web_nav3 `web/data/` (per-map places) |
| `LEGACY_PLACES` | — | optional legacy `places.json` |
| `WEB_NAV3_URL` | — | optional `http://127.0.0.1:8765` (to learn the active map) |
| `MAP_SELECT` | `all` | `all` · `active` · `newest` |
| `MAP_UPLOAD_MODE` | `multipart` | `multipart` · `base64json` |
| `MAP_ENDPOINT` | `/api/v1/fleet/ingest/{sn}/map` | path template |
| `STATE_DIR` | `/data/state` | upload-fingerprint state |

### R1 / Go2 telemetry

| var | default | meaning |
|---|---|---|
| `DDS_INTERFACE` | R1 `eth10` / Go2 `eth0` | NIC that sees robot DDS |
| `DDS_DOMAIN` | `0` | DDS domain id |
| `MAC_INTERFACE` | = `DDS_INTERFACE` | NIC whose MAC is reported |
| `R1_READ_FSM` | `0` | (R1) read loco FSM for status — read-only GET RPC |
| `R1_POSITION_SOURCE` / `GO2_POSITION_SOURCE` | `none` | `none` · `rosbridge` (· `sportmode` Go2) |
| `ROSBRIDGE_URL` | `ws://127.0.0.1:9090` | position source when rosbridge |
| `LOW_SOC` | `15` | %→ `LOW_BATTERY` fault |
| `MOTOR_TEMP_MAX` | `85` | °C→ `MOTOR_OVERTEMP` fault |
| `TELEMETRY_ENDPOINT` | `/api/v1/fleet/ingest/telemetry` | path |

---

## 9. Per-robot specifics

### G1 (map) — real maps live inside the nav container

The G1's saved maps are stored **inside** the `p4_Foxy_sanad` (Package_4 nav)
container at `/home/unitree/marcus_nav2_test/maps`, which Package_4 does **not**
bind-mount to the host. So the uploader (which mounts a host dir) sees **0 maps**
until you do one of:

1. Add a host bind-mount for `maps/` (and `web/data/`) to the Package_4 `nav`
   service, then set the uploader's `MAPS_HOST_DIR`/`DATA_HOST_DIR` to those paths.
2. Put both containers on a shared named volume for the maps dir.

The `test` command seeds a throwaway fixture map so the upload path is verified
regardless.

### R1 (telemetry)

- DDS link is **`eth10`** (= `192.168.123.164`); `wlan0` is the fleet LAN.
- Battery: `rt/lf/bmsstate` (`BmsState_`), `soc` 0–100, charging from `current`
  sign. Coexists fine with the running `sanadr1` app (DDS allows many readers).

### Go2 (telemetry) — ⚠️ unverified on hardware

- Uses `unitree_go` DDS. Battery is nested in **`rt/lowstate.bms_state`** (Go2 has
  no separate BMS topic).
- Written from the SDK layout; image builds and `unitree_go` imports, simulate is
  correct, but **not yet run on a real Go2** — confirm the `bms_state` current sign
  (charging polarity) and `sportmodestate` fields on the robot.

---

## 10. The fleet test server

`fleet_test_server.py` stands in for the real fleet server so you can verify a
deploy end-to-end from your workstation.

```bash
PORT=8799 REQLOG=/tmp/fleet.jsonl python3 fleet_test_server.py
```

- Binds `0.0.0.0:8799`.
- `GET /ping` → `200` (reachability check the installer uses).
- Accepts `POST …/map` and `POST …/telemetry`; logs each request to `REQLOG` and
  prints a live summary.

`./fleet_install.sh test …` starts it automatically, triggers a post from the
robot, asserts the payload arrived, and prints `PASS`/`FAIL`.

---

## 11. How a deploy works internally

```
install:
  rsync agents/<type>/  →  unitree@<ip>:~/sanad_api_<type>/   (--delete, minus .env/state)
  write ~/sanad_api_<type>/.env                               (SERVER_URL, TOKEN, SN, iface…)
  ssh: docker build -t sanad-api-<type>:latest .              (native arm64)
  ssh: docker create --name sanad-api-<type> <run-args> …     (no docker restart policy)
  write ~/.config/systemd/user/sanad-api-<type>.service
  ssh: loginctl enable-linger ; systemctl --user enable --now …

run-args:  --network host --env-file …/.env
           (g1 also: -v maps:ro -v web_data:ro -v state)
```

`--network host` is required: G1 to reach `127.0.0.1:8765` (web_nav3) if used, and
R1/Go2 so the robot's DDS traffic is visible (DDS multicast doesn't cross a NAT
bridge).

---

## 12. Fleet inventory

| robot | agent(s) | IP | SSH | arch | DDS iface | SN |
|---|---|---|---|---|---|---|
| G1 | map (`g1`) + telemetry (`g1t`) | `10.255.254.58` | `unitree` (key) | arm64 | `eth0` | `g1_7892` |
| R1 | telemetry (`r1`) | `10.255.254.82` | `unitree` (key) | arm64 | `eth10` | `r1_82` |
| Go2 | telemetry (`go2`) | *(TBD)* | `unitree` | arm64 | `eth0` | — |

Workstation (deploy host + test fleet server): **`10.255.254.83`** (`wlp4s0`).

---

## 13. Troubleshooting

| symptom | cause / fix |
|---|---|
| `robot cannot reach the workstation server (http 000)` during `test` | robot→workstation blocked. Check both are on the same subnet; open the port if a firewall is on (`sudo ufw allow 8799`). |
| G1 uploads nothing / "0 maps" | no saved map, or maps aren't mounted — see [§9 G1](#g1-map--real-maps-live-inside-the-nav-container). |
| R1/Go2 `telemetry POST failed … Connection refused` | the target server (e.g. the test server) is down. Point at the real server: `install … --server-ip <fleet> --token <real>`. |
| R1/Go2 `battery=null, status=offline` forever | DDS not seen. Wrong `DDS_INTERFACE` (R1 = `eth10`) or robot firmware down. Confirm with `ip -o addr` on the robot. |
| build is very slow (R1/Go2) | first build compiles CycloneDDS (minutes). Run detached and poll: `ssh … 'setsid bash -c "cd ~/sanad_api_r1 && docker build -t sanad-api-r1:latest . > build.log 2>&1" </dev/null &'` then watch `build.log`. Docker layer cache survives reboots. |
| service didn't come up after reboot | linger must be on: `loginctl show-user <user> | grep Linger` → `Linger=yes`. The installer sets it. |
| `unbound variable` / weird prompt behavior | ensure you're on the current `fleet_install.sh` (uses `ssh -n` for command calls so SSH doesn't eat stdin). |

---

## 14. Security notes

- **Outbound only.** Agents open no inbound ports on the robot; they POST out over
  HTTPS with a per-robot bearer token.
- **Least privilege.** Runs as the `unitree` user (docker group), user-level
  systemd, no root. Read-only w.r.t. the robot (no motion commands).
- **Token handling.** `.env` holds the device token and is never rsynced back or
  committed (`.gitignore`d). The installer writes it directly to the robot.
- **TLS.** `VERIFY_TLS=1` in production; `0` only for the local test server.