engineai_fleet/README.md

# EngineAI Fleet — EngineAI PM01 agent

On-robot agent that reports the **EngineAI PM01** ("T800")'s live state to the
**YS Lootah fleet server / Eco system**.

Same concept, same payload schema and same endpoints as the AGIBOT X2 agent
(`agi_fleet/`), retargeted to the PM01 — and extended to post the identical
payload to **more than one fleet server at once**.

> **Status: LIVE.** Running on `10.210.136.150` as the system service
> `sanad-api-eng`, posting to `https://eco-dev.yslootahrobotics.com` every 2 s.
> `https://eco.yslootahrobotics.com` is configured and one line away from
> enabled — see §5.

```
telemetry ok: battery=21 charging=False status=idle mode=pd_sitdown pos=None
              motor_max=55.0 faults=1 map=no_map -> eco-dev=200
```

---

## 1. Layout

```
engineai_fleet/
├── README.md                  ← this file
├── agent/
│   ├── sanad_api_eng.py       ← THE AGENT (deployed as-is)
│   ├── .env.example           ← every setting, documented
│   ├── requirements.txt       ← requests (rclpy comes from the robot's ROS)
│   ├── sanad-api-eng.service  ← systemd unit (root, survives reboot)
│   └── AGENT_README.md        ← agent internals: backends, field mapping
├── tools/
│   └── probe_eng.sh           ← read-only robot discovery
└── docs/
    └── PM01_INTERFACE.md      ← what the PM01 actually exposes
```

---

## 2. The robot

| | |
|---|---|
| Host | `10.210.136.150` (NIC `wlP1p1s0`) |
| SSH | `ubuntu@10.210.136.150` |
| Display name | `pm01_150` |
| Hardware | NVIDIA Jetson AGX Orin, arm64, L4T R36.4.3 |
| Board serial | `1421326045624` |
| OS | Ubuntu 22.04.5 LTS, kernel `5.15.148-6-engine-tegra` |
| ROS | Humble, **`ROS_DOMAIN_ID=69`**, CycloneDDS pinned to `eth1` |
| Product tag | `t800` (config dir `pm01`) |
| Install dir | `/opt/sanad_api_eng/` |
| Service | `systemctl sanad-api-eng` (**system** unit, root) |

---

## 3. How it gets its data

Four ROS 2 topics from the robot's own stack, all read-only:

| what | source |
|---|---|
| battery, voltage, current, charging | `/hardware/power_info` (`interface_protocol/msg/PowerInfo`) |
| **motor + MOSFET temperatures, motor faults** | `/hardware/motor_debug` (`MotorDebug`) — 25 motors |
| locomotion → `status: moving` | `/hardware/joint_state` (`JointState`) velocities |
| motion mode + allowed transitions | `/motion/motion_state` (`MotionState`) |
| OS, kernel, arch, board, L4T, storage, MAC | read directly from the host |
| project logs | docker json-log of the `sanad-t8` container |

### The field mapping actually in use

```ini
ENG_SOURCE=ros2
ENG_TOPIC_POWER=/hardware/power_info    ENG_TYPE_POWER=interface_protocol/msg/PowerInfo
ENG_TOPIC_MOTORS=/hardware/motor_debug  ENG_TYPE_MOTORS=interface_protocol/msg/MotorDebug
ENG_TOPIC_JOINTS=/hardware/joint_state  ENG_TYPE_JOINTS=interface_protocol/msg/JointState
ENG_TOPIC_MOTION=/motion/motion_state   ENG_TYPE_MOTION=interface_protocol/msg/MotionState

ENG_FIELD_SOC=percentage       ENG_FIELD_VOLTAGE=voltage
ENG_FIELD_CURRENT=current      ENG_FIELD_TEMPS=motor_temperature
ENG_FIELD_MOS_TEMPS=mos_temperature
ENG_FIELD_MOTION=current_motion_task
ENG_SOC_SCALE=percent          ENG_CURRENT_SIGN=-1     # ← measured, see §4
ENG_ROS_QOS=best_effort        ROS_DOMAIN_ID=69

ENG_JOINT_MIN_PERIOD=0.05      # 500 Hz → 20 Hz  (protects the robot's CPU)
ENG_MOTOR_MIN_PERIOD=0.2       # 100 Hz →  5 Hz

MAC_INTERFACE=wlP1p1s0
REMOTE_PORTS=8014,8001,9002,9003,8000,8080
```

Nothing above is hard-coded in the agent — it is all `.env`. Retargeting is an
`.env` edit + restart, never a code change.

---

## 4. What it sends

One JSON object every 2 s to `POST /api/v1/fleet/ingest/telemetry` with
`Authorization: Bearer <token>`, plus `/{sn}/alert`, `/{sn}/logs`, `/{sn}/map`
and `/{sn}/remote` — **to every enabled server**. Verified field-by-field
against the robot's own readings:

| robot ground truth | agent sends |
|---|---|
| `power_info percentage: 21.0` | `battery: 21` |
| `power_info voltage: 53.3` | `voltage_v: 53.3` |
| `power_info current: +2.47` (discharging) | `current_a: -2.47`, `charging: false` |
| `power_info current_limit: 90.0` | `current_limit_a: 90.0` |
| `motor_debug motor_temperature[25]` | `motor_temp: {max:55.1, avg:29.5, min:24.0, count:25}` |
| `motor_debug mos_temperature[25]` | `motor_temp.mos_max: 45.6, mos_avg: 29.9` |
| `motion_state current_motion_task` | `control.mode: "pd_sitdown"` |
| `motion_state available_transition_motions` | `control.switchable_modes: [...]` |
| `df /` → 250.6 GB, 201 GB free | `storage: {total_gb:250.64, free_gb:201.3, used_percent:14.6}` |

Also registers the Sanad Dashboard (`http://10.210.136.150:8014`) and
`ssh ubuntu@10.210.136.150` for the fleet UI, and ships the Sanad app's logs as
`project_logs: "sanad-t8-logs"`.

### The battery sign — the one trap on this robot

`PowerInfo.current` stays **positive while the pack drains**. Measured over
15 minutes, idle and off-charger: `27% → 22%`, `54.74 V → 53.3 V`, current
`+1.7 … +2.5 A` throughout. So positive means *discharging* here — the opposite
of the ROS `BatteryState` convention.

Hence `ENG_CURRENT_SIGN=-1`. With the default `+1` the robot would report
`charging: true` and `status: "charging"` forever while its battery went flat —
a failure that looks exactly like healthy telemetry.

### Fields that are `null` — and why

- **`position`** — the PM01 publishes **no odometry topic at all**
  (no `/odom`, no `/tf`, no `amcl_pose`); its motion stack is a whole-body
  controller, not a navigation stack, and the Sanad nav bringup is not running.
  `null` means "not available", never `{x: 0, y: 0}` — a fabricated origin would
  park the robot in the corner of the fleet map and look like real data.
  One `.env` line turns it on the day localisation runs (`ENG_POSITION_SOURCE`).
- **`battery_detail.temp_c` / `soh` / `cycles`** — `PowerInfo` carries no pack
  temperature, state-of-health or cycle count. Left unmapped rather than pointed
  at a plausible-looking wrong field.
- **`map`** — `no_map`; there are no saved maps on this robot yet.

### What this robot reports that the X2 could not

`motor_temp` (the X2 publishes no per-motor temperature at all — permanently
null there), `control.mode` by name plus the real transition set (the X2 had no
documented FSM id scheme and reported `"unknown"`), and three hardware fault
channels: `POWER_FAULT`/`POWER_DISABLED` from `PowerInfo.error_code`/`enable`,
and `MOTOR_FAULT`/`MOTOR_OFFLINE` from `MotorDebug.error_code[]`/`offline[]`.

---

## 5. Two fleet servers, two tokens

The X2 agent posts to one server. This one keeps a **list** of endpoints and
sends the identical payload to each:

```ini
SERVER_URL=https://eco-dev.yslootahrobotics.com     # server 1
DEVICE_TOKEN=<eco-dev token>
SERVER_ENABLE=1

SERVER_2_URL=https://eco.yslootahrobotics.com       # server 2
SERVER_2_TOKEN=                                     # ← paste eco's token
SERVER_2_ENABLE=0                                   # ← then flip to 1
```

**The supplied token works on `eco-dev` and is rejected by `eco`.** Tested from
the robot: `eco-dev` returns `200 {"ok":true,"robot_id":162}`; `eco` returns
`401` for that token, for a garbage token and for no token at all — i.e. it is
the application's auth layer, not a firewall (the host is reachable; `/` answers
`302`). The two deployments maintain independent token stores.

So server 2 ships **configured but disabled**. To turn it on:

```bash
ssh ubuntu@10.210.136.150
sudo sed -i 's#^SERVER_2_TOKEN=.*#SERVER_2_TOKEN=<eco-token>#; \
             s#^SERVER_2_ENABLE=.*#SERVER_2_ENABLE=1#' /opt/sanad_api_eng/.env
sudo systemctl restart sanad-api-eng
```

Every log line then carries both results:

```
telemetry ok: battery=21 … -> eco-dev=200 eco=200
```

A failing server never blocks the other — each POST is independent, and repeated
failures are reported once per minute with a suppressed-count rather than once
every 2 s:

```
ERROR telemetry -> eco FAILED: HTTP 401 Unauthorized  <- token rejected by eco;
      each fleet server issues its OWN token …  [+3 more since last report]
INFO  telemetry ok: battery=21 … -> eco-dev=200 eco=401
```

Maps track upload state **per server**, so enabling `eco` later uploads every
existing map to it rather than finding them already marked "uploaded".

---

## 6. Operating it

```bash
# live log
ssh ubuntu@10.210.136.150 'sudo journalctl -u sanad-api-eng -f'

# what it is sending right now
ssh ubuntu@10.210.136.150 \
  "sudo journalctl -u sanad-api-eng -n 20 --no-pager | grep -oE 'telemetry ok:.*'"

# service control
ssh ubuntu@10.210.136.150 'sudo systemctl restart sanad-api-eng'
ssh ubuntu@10.210.136.150 'sudo systemctl status  sanad-api-eng'

# change a setting (then restart)
ssh ubuntu@10.210.136.150 'sudo nano /opt/sanad_api_eng/.env && sudo systemctl restart sanad-api-eng'
```

### Updating the agent code

```bash
scp agent/sanad_api_eng.py ubuntu@10.210.136.150:/tmp/
ssh ubuntu@10.210.136.150 \
  'sudo install -m755 /tmp/sanad_api_eng.py /opt/sanad_api_eng/ && sudo systemctl restart sanad-api-eng'
```

`.env` is never overwritten by this, so the tokens stay put.

### Changing the serial

`SN` is the primary key for every `/{sn}/` route. Changing it creates a **new**
robot entry on the server rather than renaming the existing one:

```bash
ssh ubuntu@10.210.136.150 \
  "sudo sed -i 's#^SN=.*#SN=<real-serial>#' /opt/sanad_api_eng/.env && \
   sudo systemctl restart sanad-api-eng"
```

### Re-discovering the robot's interface

```bash
bash tools/probe_eng.sh 10.210.136.150 ubuntu
```

Read-only — no install, no writes, publishes to no topic, safe on a live robot.
Every line it prints is labelled with the `.env` variable it feeds.

### Verifying without touching the live feed

```bash
sudo -E python3 /opt/sanad_api_eng/sanad_api_eng.py --dry-run   # builds payloads, never POSTs
sudo -E python3 /opt/sanad_api_eng/sanad_api_eng.py --once      # one real post, then exits
```

(both need the ROS overlay: `set +u; . /app/applications/install/bringup/ros_env.sh`)

---

## 7. Why it runs as a root system service

The X2 agent runs as a `--user` unit and has one open item: `linger` is off, so
it does **not** come back after a power cycle. That is fixed here by running a
**system** unit, which also solves a second problem:

- **Reboot survival** — `systemctl enable`d, no `loginctl enable-linger`
  needed. Verified `enabled` + `active`.
- **Project logs** — the Sanad app runs in the `sanad-t8` container and its log
  lives at `/var/lib/docker/containers/<id>/<id>-json.log`, root-owned mode 600.
  As `ubuntu` that read is `EACCES` and `project_logs` would stay null forever.

The unit sources the robot's **own** environment file so the agent follows
EngineAI's settings instead of restating them:

```ini
ExecStart=/bin/bash -c 'set +u; \
  . /opt/ros/humble/setup.bash >/dev/null 2>&1 || true; \
  . /app/applications/install/bringup/ros_env.sh >/dev/null 2>&1 || true; \
  exec /usr/bin/python3 -u /opt/sanad_api_eng/sanad_api_eng.py'
```

`set +u` is required — ROS's `setup.bash` reads unbound variables and would
abort the unit under `set -u`. `ros_env.sh` is what sets `ROS_DOMAIN_ID=69`,
`RMW_IMPLEMENTATION=rmw_cyclonedds_cpp` and the CycloneDDS interface pinning;
without it `ros2 topic list` shows 2 topics instead of 44 and every field
silently reports null.

The robot's own ROS apps run under **supervisord**, not systemd. The agent
deliberately does not join that group: a restart of the agent must never be able
to disturb the robot's motion stack.

### Verified behaviour

| | |
|---|---|
| `kill -9` | auto-restarts in ~5 s (`Restart=always`), `NRestarts: 1` |
| `systemctl restart` | `received SIGINT — stopping` → `Deactivated successfully` — clean, no abort |
| `systemctl is-enabled` | `enabled` — starts at boot |

The clean stop matters: rclpy's CycloneDDS threads abort the process at
interpreter teardown (`terminate called without an active exception`), which
systemd records as a failed exit and which would mask a real crash. The agent
shuts ROS down explicitly on `SIGINT`/`SIGTERM`.

---

## 8. Security

- **Outbound only.** No inbound port is opened on the robot; every call is an
  HTTPS POST carrying `Authorization: Bearer <token>`, keyed by `sn`.
- **Read-only toward the robot.** It subscribes to four topics and reads two
  local HTTP status pages. It never publishes, never calls a service, and never
  commands motion. `/motion/motion_state`'s transition list is *reported*, never
  *requested*; `CONTROL_ENABLE=0`.
- **Token handling.** Tokens live only in `/opt/sanad_api_eng/.env` on the
  robot, mode `600` root-owned, and are `.gitignore`d here. Only `.env.example`
  (placeholders) is in this repo.
- **TLS verified** (`VERIFY_TLS=1`).
- **Rate-limited.** The 500 Hz and 100 Hz streams are gated to 20 Hz / 5 Hz and
  subscribed `raw`, so a dropped message is never deserialized. Measured cost:
  **~24% of one core out of 12** (~2% of the machine), on the application
  processor — the realtime motion controller is a separate board. See
  `agent/AGENT_README.md` for the per-subscription breakdown and why it is not
  optimised further.

---

## 9. Relationship to the other fleet agents

`sanad_api_eng.py` is standalone. It shares no files with `agi_fleet/`
(the X2 agent) or with the Unitree `fleet/` agents (G1 / R1 / Go2), and contains
zero AgiBot or Unitree code. The three can be changed independently.

It does deliberately keep the X2 agent's **telemetry schema, endpoint paths,
env-var discipline and map/logs/alerts/remote loops** unchanged, so the same
fleet server ingests all of them identically — the only structural difference is
the multi-server endpoint list described in §5.