Sedra/engineai_fleet
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update 2026-08-27 16:24

Browse Source
This commit is contained in:
Sidra 2026-08-27 16:24:12 +04:00
commit 5ce59ef443
11 changed files with 4139 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
.gitignore vendored Normal file
View File

@ -0,0 +1,5 @@
# the real .env holds the device tokens — never commit it
agent/.env
.env
__pycache__/
*.pyc

343
README.md Normal file
View File

@ -0,0 +1,343 @@
# EngineAI Fleet — EngineAI PM01 agent
On-robot agent that reports the **EngineAI PM01** ("T800")'s live state to the
**YS Lootah fleet server / Eco system**.
Same concept, same payload schema and same endpoints as the AGIBOT X2 agent
(`agi_fleet/`), retargeted to the PM01 — and extended to post the identical
payload to **more than one fleet server at once**.
> **Status: LIVE.** Running on `10.210.136.150` as the system service
> `sanad-api-eng`, posting to `https://eco-dev.yslootahrobotics.com` every 2 s.
> `https://eco.yslootahrobotics.com` is configured and one line away from
> enabled — see §5.
```
telemetry ok: battery=21 charging=False status=idle mode=pd_sitdown pos=None
motor_max=55.0 faults=1 map=no_map -> eco-dev=200
```
---
## 1. Layout
```
engineai_fleet/
├── README.md ← this file
├── agent/
│ ├── sanad_api_eng.py ← THE AGENT (deployed as-is)
│ ├── .env.example ← every setting, documented
│ ├── requirements.txt ← requests (rclpy comes from the robot's ROS)
│ ├── sanad-api-eng.service ← systemd unit (root, survives reboot)
│ └── AGENT_README.md ← agent internals: backends, field mapping
├── tools/
│ └── probe_eng.sh ← read-only robot discovery
└── docs/
└── PM01_INTERFACE.md ← what the PM01 actually exposes
```
---
## 2. The robot
| | |
|---|---|
| Host | `10.210.136.150` (NIC `wlP1p1s0`) |
| SSH | `ubuntu@10.210.136.150` |
| Display name | `pm01_150` |
| Hardware | NVIDIA Jetson AGX Orin, arm64, L4T R36.4.3 |
| Board serial | `1421326045624` |
| OS | Ubuntu 22.04.5 LTS, kernel `5.15.148-6-engine-tegra` |
| ROS | Humble, **`ROS_DOMAIN_ID=69`**, CycloneDDS pinned to `eth1` |
| Product tag | `t800` (config dir `pm01`) |
| Install dir | `/opt/sanad_api_eng/` |
| Service | `systemctl sanad-api-eng` (**system** unit, root) |
---
## 3. How it gets its data
Four ROS 2 topics from the robot's own stack, all read-only:
| what | source |
|---|---|
| battery, voltage, current, charging | `/hardware/power_info` (`interface_protocol/msg/PowerInfo`) |
| **motor + MOSFET temperatures, motor faults** | `/hardware/motor_debug` (`MotorDebug`) — 25 motors |
| locomotion → `status: moving` | `/hardware/joint_state` (`JointState`) velocities |
| motion mode + allowed transitions | `/motion/motion_state` (`MotionState`) |
| OS, kernel, arch, board, L4T, storage, MAC | read directly from the host |
| project logs | docker json-log of the `sanad-t8` container |
### The field mapping actually in use
```ini
ENG_SOURCE=ros2
ENG_TOPIC_POWER=/hardware/power_info ENG_TYPE_POWER=interface_protocol/msg/PowerInfo
ENG_TOPIC_MOTORS=/hardware/motor_debug ENG_TYPE_MOTORS=interface_protocol/msg/MotorDebug
ENG_TOPIC_JOINTS=/hardware/joint_state ENG_TYPE_JOINTS=interface_protocol/msg/JointState
ENG_TOPIC_MOTION=/motion/motion_state ENG_TYPE_MOTION=interface_protocol/msg/MotionState
ENG_FIELD_SOC=percentage ENG_FIELD_VOLTAGE=voltage
ENG_FIELD_CURRENT=current ENG_FIELD_TEMPS=motor_temperature
ENG_FIELD_MOS_TEMPS=mos_temperature
ENG_FIELD_MOTION=current_motion_task
ENG_SOC_SCALE=percent ENG_CURRENT_SIGN=-1 # ← measured, see §4
ENG_ROS_QOS=best_effort ROS_DOMAIN_ID=69
ENG_JOINT_MIN_PERIOD=0.05 # 500 Hz → 20 Hz (protects the robot's CPU)
ENG_MOTOR_MIN_PERIOD=0.2 # 100 Hz → 5 Hz
MAC_INTERFACE=wlP1p1s0
REMOTE_PORTS=8014,8001,9002,9003,8000,8080
```
Nothing above is hard-coded in the agent — it is all `.env`. Retargeting is an
`.env` edit + restart, never a code change.
---
## 4. What it sends
One JSON object every 2 s to `POST /api/v1/fleet/ingest/telemetry` with
`Authorization: Bearer <token>`, plus `/{sn}/alert`, `/{sn}/logs`, `/{sn}/map`
and `/{sn}/remote` — **to every enabled server**. Verified field-by-field
against the robot's own readings:
| robot ground truth | agent sends |
|---|---|
| `power_info percentage: 21.0` | `battery: 21` |
| `power_info voltage: 53.3` | `voltage_v: 53.3` |
| `power_info current: +2.47` (discharging) | `current_a: -2.47`, `charging: false` |
| `power_info current_limit: 90.0` | `current_limit_a: 90.0` |
| `motor_debug motor_temperature[25]` | `motor_temp: {max:55.1, avg:29.5, min:24.0, count:25}` |
| `motor_debug mos_temperature[25]` | `motor_temp.mos_max: 45.6, mos_avg: 29.9` |
| `motion_state current_motion_task` | `control.mode: "pd_sitdown"` |
| `motion_state available_transition_motions` | `control.switchable_modes: [...]` |
| `df /` → 250.6 GB, 201 GB free | `storage: {total_gb:250.64, free_gb:201.3, used_percent:14.6}` |
Also registers the Sanad Dashboard (`http://10.210.136.150:8014`) and
`ssh ubuntu@10.210.136.150` for the fleet UI, and ships the Sanad app's logs as
`project_logs: "sanad-t8-logs"`.
### The battery sign — the one trap on this robot
`PowerInfo.current` stays **positive while the pack drains**. Measured over
15 minutes, idle and off-charger: `27% → 22%`, `54.74 V → 53.3 V`, current
`+1.7 … +2.5 A` throughout. So positive means *discharging* here — the opposite
of the ROS `BatteryState` convention.
Hence `ENG_CURRENT_SIGN=-1`. With the default `+1` the robot would report
`charging: true` and `status: "charging"` forever while its battery went flat —
a failure that looks exactly like healthy telemetry.
### Fields that are `null` — and why
- **`position`** — the PM01 publishes **no odometry topic at all**
(no `/odom`, no `/tf`, no `amcl_pose`); its motion stack is a whole-body
controller, not a navigation stack, and the Sanad nav bringup is not running.
`null` means "not available", never `{x: 0, y: 0}` — a fabricated origin would
park the robot in the corner of the fleet map and look like real data.
One `.env` line turns it on the day localisation runs (`ENG_POSITION_SOURCE`).
- **`battery_detail.temp_c` / `soh` / `cycles`** — `PowerInfo` carries no pack
temperature, state-of-health or cycle count. Left unmapped rather than pointed
at a plausible-looking wrong field.
- **`map`** — `no_map`; there are no saved maps on this robot yet.
### What this robot reports that the X2 could not
`motor_temp` (the X2 publishes no per-motor temperature at all — permanently
null there), `control.mode` by name plus the real transition set (the X2 had no
documented FSM id scheme and reported `"unknown"`), and three hardware fault
channels: `POWER_FAULT`/`POWER_DISABLED` from `PowerInfo.error_code`/`enable`,
and `MOTOR_FAULT`/`MOTOR_OFFLINE` from `MotorDebug.error_code[]`/`offline[]`.
---
## 5. Two fleet servers, two tokens
The X2 agent posts to one server. This one keeps a **list** of endpoints and
sends the identical payload to each:
```ini
SERVER_URL=https://eco-dev.yslootahrobotics.com # server 1
DEVICE_TOKEN=<eco-dev token>
SERVER_ENABLE=1
SERVER_2_URL=https://eco.yslootahrobotics.com # server 2
SERVER_2_TOKEN= # ← paste eco's token
SERVER_2_ENABLE=0 # ← then flip to 1
```
**The supplied token works on `eco-dev` and is rejected by `eco`.** Tested from
the robot: `eco-dev` returns `200 {"ok":true,"robot_id":162}`; `eco` returns
`401` for that token, for a garbage token and for no token at all — i.e. it is
the application's auth layer, not a firewall (the host is reachable; `/` answers
`302`). The two deployments maintain independent token stores.
So server 2 ships **configured but disabled**. To turn it on:
```bash
ssh ubuntu@10.210.136.150
sudo sed -i 's#^SERVER_2_TOKEN=.*#SERVER_2_TOKEN=<eco-token>#; \
s#^SERVER_2_ENABLE=.*#SERVER_2_ENABLE=1#' /opt/sanad_api_eng/.env
sudo systemctl restart sanad-api-eng
```
Every log line then carries both results:
```
telemetry ok: battery=21 … -> eco-dev=200 eco=200
```
A failing server never blocks the other — each POST is independent, and repeated
failures are reported once per minute with a suppressed-count rather than once
every 2 s:
```
ERROR telemetry -> eco FAILED: HTTP 401 Unauthorized <- token rejected by eco;
each fleet server issues its OWN token … [+3 more since last report]
INFO telemetry ok: battery=21 … -> eco-dev=200 eco=401
```
Maps track upload state **per server**, so enabling `eco` later uploads every
existing map to it rather than finding them already marked "uploaded".
---
## 6. Operating it
```bash
# live log
ssh ubuntu@10.210.136.150 'sudo journalctl -u sanad-api-eng -f'
# what it is sending right now
ssh ubuntu@10.210.136.150 \
"sudo journalctl -u sanad-api-eng -n 20 --no-pager | grep -oE 'telemetry ok:.*'"
# service control
ssh ubuntu@10.210.136.150 'sudo systemctl restart sanad-api-eng'
ssh ubuntu@10.210.136.150 'sudo systemctl status sanad-api-eng'
# change a setting (then restart)
ssh ubuntu@10.210.136.150 'sudo nano /opt/sanad_api_eng/.env && sudo systemctl restart sanad-api-eng'
```
### Updating the agent code
```bash
scp agent/sanad_api_eng.py ubuntu@10.210.136.150:/tmp/
ssh ubuntu@10.210.136.150 \
'sudo install -m755 /tmp/sanad_api_eng.py /opt/sanad_api_eng/ && sudo systemctl restart sanad-api-eng'
```
`.env` is never overwritten by this, so the tokens stay put.
### Changing the serial
`SN` is the primary key for every `/{sn}/` route. Changing it creates a **new**
robot entry on the server rather than renaming the existing one:
```bash
ssh ubuntu@10.210.136.150 \
"sudo sed -i 's#^SN=.*#SN=<real-serial>#' /opt/sanad_api_eng/.env && \
sudo systemctl restart sanad-api-eng"
```
### Re-discovering the robot's interface
```bash
bash tools/probe_eng.sh 10.210.136.150 ubuntu
```
Read-only — no install, no writes, publishes to no topic, safe on a live robot.
Every line it prints is labelled with the `.env` variable it feeds.
### Verifying without touching the live feed
```bash
sudo -E python3 /opt/sanad_api_eng/sanad_api_eng.py --dry-run # builds payloads, never POSTs
sudo -E python3 /opt/sanad_api_eng/sanad_api_eng.py --once # one real post, then exits
```
(both need the ROS overlay: `set +u; . /app/applications/install/bringup/ros_env.sh`)
---
## 7. Why it runs as a root system service
The X2 agent runs as a `--user` unit and has one open item: `linger` is off, so
it does **not** come back after a power cycle. That is fixed here by running a
**system** unit, which also solves a second problem:
- **Reboot survival** — `systemctl enable`d, no `loginctl enable-linger`
needed. Verified `enabled` + `active`.
- **Project logs** — the Sanad app runs in the `sanad-t8` container and its log
lives at `/var/lib/docker/containers/<id>/<id>-json.log`, root-owned mode 600.
As `ubuntu` that read is `EACCES` and `project_logs` would stay null forever.
The unit sources the robot's **own** environment file so the agent follows
EngineAI's settings instead of restating them:
```ini
ExecStart=/bin/bash -c 'set +u; \
. /opt/ros/humble/setup.bash >/dev/null 2>&1 || true; \
. /app/applications/install/bringup/ros_env.sh >/dev/null 2>&1 || true; \
exec /usr/bin/python3 -u /opt/sanad_api_eng/sanad_api_eng.py'
```
`set +u` is required — ROS's `setup.bash` reads unbound variables and would
abort the unit under `set -u`. `ros_env.sh` is what sets `ROS_DOMAIN_ID=69`,
`RMW_IMPLEMENTATION=rmw_cyclonedds_cpp` and the CycloneDDS interface pinning;
without it `ros2 topic list` shows 2 topics instead of 44 and every field
silently reports null.
The robot's own ROS apps run under **supervisord**, not systemd. The agent
deliberately does not join that group: a restart of the agent must never be able
to disturb the robot's motion stack.
### Verified behaviour
| | |
|---|---|
| `kill -9` | auto-restarts in ~5 s (`Restart=always`), `NRestarts: 1` |
| `systemctl restart` | `received SIGINT — stopping` → `Deactivated successfully` — clean, no abort |
| `systemctl is-enabled` | `enabled` — starts at boot |
The clean stop matters: rclpy's CycloneDDS threads abort the process at
interpreter teardown (`terminate called without an active exception`), which
systemd records as a failed exit and which would mask a real crash. The agent
shuts ROS down explicitly on `SIGINT`/`SIGTERM`.
---
## 8. Security
- **Outbound only.** No inbound port is opened on the robot; every call is an
HTTPS POST carrying `Authorization: Bearer <token>`, keyed by `sn`.
- **Read-only toward the robot.** It subscribes to four topics and reads two
local HTTP status pages. It never publishes, never calls a service, and never
commands motion. `/motion/motion_state`'s transition list is *reported*, never
*requested*; `CONTROL_ENABLE=0`.
- **Token handling.** Tokens live only in `/opt/sanad_api_eng/.env` on the
robot, mode `600` root-owned, and are `.gitignore`d here. Only `.env.example`
(placeholders) is in this repo.
- **TLS verified** (`VERIFY_TLS=1`).
- **Rate-limited.** The 500 Hz and 100 Hz streams are gated to 20 Hz / 5 Hz and
subscribed `raw`, so a dropped message is never deserialized. Measured cost:
**~24% of one core out of 12** (~2% of the machine), on the application
processor — the realtime motion controller is a separate board. See
`agent/AGENT_README.md` for the per-subscription breakdown and why it is not
optimised further.
---
## 9. Relationship to the other fleet agents
`sanad_api_eng.py` is standalone. It shares no files with `agi_fleet/`
(the X2 agent) or with the Unitree `fleet/` agents (G1 / R1 / Go2), and contains
zero AgiBot or Unitree code. The three can be changed independently.
It does deliberately keep the X2 agent's **telemetry schema, endpoint paths,
env-var discipline and map/logs/alerts/remote loops** unchanged, so the same
fleet server ingests all of them identically — the only structural difference is
the multi-server endpoint list described in §5.

154
agent/.env.deployed Normal file
View File

@ -0,0 +1,154 @@
# ============================================================================
# .env.deployed — a REDACTED snapshot of the .env actually running on the
# robot at /opt/sanad_api_eng/.env (mode 600, root).
#
# This is the real deployed configuration, captured so the folder is a
# complete record of what is live. THE TOKENS HAVE BEEN REMOVED:
#
# DEVICE_TOKEN=REPLACE_WITH_ECO_DEV_TOKEN <- was the working eco-dev token
# SERVER_2_TOKEN= <- was already empty (eco 401s)
#
# Every other value is exactly what the live service is using.
#
# To rebuild the robot from this file:
# scp agent/.env.deployed ubuntu@10.210.136.150:/tmp/.env
# ssh ubuntu@10.210.136.150 'sudo install -m600 -o root -g root /tmp/.env \
# /opt/sanad_api_eng/.env && rm /tmp/.env'
# # then put the real token back:
# ssh ubuntu@10.210.136.150 "sudo sed -i \
# 's#^DEVICE_TOKEN=.*#DEVICE_TOKEN=<token>#' /opt/sanad_api_eng/.env && \
# sudo systemctl restart sanad-api-eng"
# ============================================================================
# sanad_api_eng — EngineAI PM01 (10.210.136.150). LIVE CONFIG.
# Generated from .env.example; every value below was verified on this robot.
# ── fleet servers (same payload to every enabled one) ────────────────────────
SERVER_URL=https://eco-dev.yslootahrobotics.com
DEVICE_TOKEN=REPLACE_WITH_ECO_DEV_TOKEN
SERVER_NAME=eco-dev
SERVER_ENABLE=1
# eco (production) rejected the token above with 401 — identically to a garbage
# token, so it is the app's auth layer, not a firewall. eco issues its own
# tokens. Paste it below and set SERVER_2_ENABLE=1, then restart the service.
SERVER_2_URL=https://eco.yslootahrobotics.com
SERVER_2_TOKEN=
SERVER_2_NAME=eco
SERVER_2_ENABLE=0
SERVER_2_SHARE_TOKEN=0
# ── identity ─────────────────────────────────────────────────────────────────
SN=PM01-1421326045624
ROBOT_NAME=pm01_150
ROBOT_BRAND=engineai
ROBOT_TYPE=humanoid
ROBOT_MODEL=pm01
ROBOT=sanad
STORAGE_DATA_PATH=/home/ubuntu/sanad_t8/data
# ── PM01 state source ────────────────────────────────────────────────────────
ENG_SOURCE=ros2
ENG_TOPIC_POWER=/hardware/power_info
ENG_TYPE_POWER=interface_protocol/msg/PowerInfo
ENG_TOPIC_MOTORS=/hardware/motor_debug
ENG_TYPE_MOTORS=interface_protocol/msg/MotorDebug
ENG_TOPIC_JOINTS=/hardware/joint_state
ENG_TYPE_JOINTS=interface_protocol/msg/JointState
ENG_TOPIC_MOTION=/motion/motion_state
ENG_TYPE_MOTION=interface_protocol/msg/MotionState
ENG_TOPIC_ODOM=
ENG_TYPE_ODOM=nav_msgs/msg/Odometry
ROS_DOMAIN_ID=69
ENG_ROS_QOS=best_effort
ENG_JOINT_MIN_PERIOD=0.05
ENG_MOTOR_MIN_PERIOD=0.2
# ── field mapping ────────────────────────────────────────────────────────────
ENG_FIELD_SOC=percentage
ENG_FIELD_VOLTAGE=voltage
ENG_FIELD_CURRENT=current
ENG_FIELD_CURRENT_LIMIT=current_limit
ENG_FIELD_POWER_ERR=error_code
ENG_FIELD_POWER_ENABLE=enable
ENG_FIELD_TEMP=
ENG_FIELD_SOH=
ENG_FIELD_CYCLES=
ENG_FIELD_TEMPS=motor_temperature
ENG_FIELD_MOS_TEMPS=mos_temperature
ENG_FIELD_MOTOR_ERR=error_code
ENG_FIELD_MOTOR_OFFLINE=offline
ENG_FIELD_VEL=velocity
ENG_FIELD_MOTION=current_motion_task
ENG_FIELD_TRANSITIONS=available_transition_motions
ENG_FIELD_X=pose.pose.position.x
ENG_FIELD_Y=pose.pose.position.y
ENG_FIELD_FW=
# ── unit conventions (measured on this robot) ────────────────────────────────
ENG_SOC_SCALE=percent
ENG_CURRENT_SIGN=-1
ENG_VOLTAGE_SCALE=1
ENG_CURRENT_SCALE=1
# ── position (no odometry on the PM01 — reports null, never a fake origin) ───
ENG_POSITION_SOURCE=none
ENG_POSITION_URL=http://127.0.0.1:8014/api/nav/status
ENG_POSITION_FIELD_X=pose.x
ENG_POSITION_FIELD_Y=pose.y
ENG_POSITION_INTERVAL=2
ROSBRIDGE_URL=ws://127.0.0.1:9090
# ── identity / networking ────────────────────────────────────────────────────
MAC_INTERFACE=wlP1p1s0
# ── fault thresholds ─────────────────────────────────────────────────────────
LOW_SOC=50
MOTOR_TEMP_MAX=85
MOS_TEMP_MAX=100
MOVING_VEL=0.15
# ── cadence / transport ──────────────────────────────────────────────────────
POLL_INTERVAL=2
VERIFY_TLS=1
HTTP_TIMEOUT=30
TZ_OFFSET_HOURS=4
# ── map sync ─────────────────────────────────────────────────────────────────
MAPS_DIR=/home/ubuntu/sanad_t8/data/maps
EXTRA_MAP_DIRS=
DATA_DIR=/home/ubuntu/sanad_t8/data
STATE_DIR=/var/lib/sanad_api_eng
MAP_SELECT=all
MAP_UPLOAD_MODE=multipart
MAP_POLL_INTERVAL=30
MAP_MAX_UPLOAD_MB=7
WEB_NAV3_URL=
# ── logs + alerts ────────────────────────────────────────────────────────────
LOGS_INTERVAL=60
PROJECT_LOG_CONTAINER=auto
PROJECT_LOG_PATH=
PROJECT_LOG_LABEL=
PROJECT_LOG_BACKFILL=100
ALERT_SCAN_INTERVAL=10
ALERT_LOG_COOLDOWN=300
# ── remote ───────────────────────────────────────────────────────────────────
REMOTE_ENABLE=1
REMOTE_PORTS=8014,8001,9002,9003,8000,8080
REMOTE_KIND=web
REMOTE_INTERVAL=60
REMOTE_URL=
REMOTE_HOST=
SSH_REGISTER=1
SSH_USER=ubuntu
SSH_PORT=22
# ── control (READ-ONLY) ──────────────────────────────────────────────────────
CONTROL_STATUS_URL=http://127.0.0.1:8014/api/controller/status
CONTROL_ENABLE=0
# raw subscriptions for the 500Hz/100Hz topics (see AGENT_README)
ENG_RAW_SUBSCRIBE=1
ERROR_LOG_COOLDOWN=60

235
agent/.env.example Normal file
View File

@ -0,0 +1,235 @@
# sanad_api_eng — EngineAI PM01. Copy to .env and fill in.
# ONE agent = telemetry + map + logs + alerts + remote, posted to EVERY server.
#
# Every topic name and field path used to read robot state is an env var in this
# file. Nothing is hard-coded in the agent. Discover the real ones with:
#
# bash tools/probe_eng.sh 10.210.136.150 ubuntu
#
# then set them below and restart — no code change, no rebuild.
# ═════════════════════════════════════════════════════════════════════════════
# FLEET SERVERS — the same payload is posted to EVERY enabled server
# ═════════════════════════════════════════════════════════════════════════════
# Each server has its OWN token. eco and eco-dev maintain INDEPENDENT token
# stores: a token minted on one is rejected with 401 by the other. Reusing a
# single token across both silently 401s forever on whichever server did not
# issue it — which is why the token is per-slot, not global.
# ── server 1 (primary) ───────────────────────────────────────────────────────
SERVER_URL=https://eco-dev.yslootahrobotics.com
DEVICE_TOKEN=REPLACE_WITH_ECO_DEV_TOKEN
SERVER_NAME=eco-dev
SERVER_ENABLE=1
# ── server 2 ─────────────────────────────────────────────────────────────────
# Configured and ready. Paste the token eco issues for this robot and flip
# SERVER_2_ENABLE to 1 — that is the whole change, then restart the service.
SERVER_2_URL=https://eco.yslootahrobotics.com
SERVER_2_TOKEN=
SERVER_2_NAME=eco
SERVER_2_ENABLE=0
# Set to 1 ONLY if eco is ever configured to accept the primary DEVICE_TOKEN.
SERVER_2_SHARE_TOKEN=0
# ── servers 3..5 (unused; same three keys each) ──────────────────────────────
#SERVER_3_URL=
#SERVER_3_TOKEN=
#SERVER_3_ENABLE=0
# ── identity ─────────────────────────────────────────────────────────────────
# The robot's serial — keys the robot on the server ({sn} routes). Changing it
# later creates a SECOND robot entry on the server rather than renaming this one.
SN=REPLACE_WITH_PM01_SERIAL
# Friendly display name shown on the dashboard.
ROBOT_NAME=pm01_150
ROBOT_BRAND=engineai
ROBOT_TYPE=humanoid
ROBOT_MODEL=pm01
# Maps subdir (<MAPS_DIR>/<ROBOT>/…) + X-Robot-Name header.
ROBOT=sanad
# Optional: app data dir whose size is reported inside storage. Empty = omit.
STORAGE_DATA_PATH=/home/ubuntu/sanad_t8/data
# ═════════════════════════════════════════════════════════════════════════════
# PM01 STATE SOURCE — the one robot-specific section
# ═════════════════════════════════════════════════════════════════════════════
# auto = ros2 if rclpy imports, else http if ENG_STATE_URL is set, else none
# (heartbeat: battery null, status offline).
# ros2 = subscribe ENG_TOPIC_* — the deployed setting.
# http = poll ENG_STATE_URL for one JSON object; map fields with ENG_FIELD_*.
# none = never read; always heartbeat.
ENG_SOURCE=ros2
# ── backend: ros2 ────────────────────────────────────────────────────────────
# Verified live on the robot with `ros2 topic list -t` (see docs/PM01_INTERFACE.md).
# The types are the vendor's own interface_protocol messages; they import only
# after the ROS overlay is sourced, which the systemd unit does via ros_env.sh.
ENG_TOPIC_POWER=/hardware/power_info
ENG_TYPE_POWER=interface_protocol/msg/PowerInfo
ENG_TOPIC_MOTORS=/hardware/motor_debug
ENG_TYPE_MOTORS=interface_protocol/msg/MotorDebug
ENG_TOPIC_JOINTS=/hardware/joint_state
ENG_TYPE_JOINTS=interface_protocol/msg/JointState
ENG_TOPIC_MOTION=/motion/motion_state
ENG_TYPE_MOTION=interface_protocol/msg/MotionState
# The PM01 publishes NO odometry topic — leave empty (position reports null).
# Fill this in the day a nav stack starts publishing one.
ENG_TOPIC_ODOM=
ENG_TYPE_ODOM=nav_msgs/msg/Odometry
# Must match the robot's domain or ROS 2 discovery silently sees nothing.
# The PM01 stack runs on 69 with CycloneDDS pinned to eth1 — the unit sources
# /app/applications/install/bringup/ros_env.sh, which sets all three.
ROS_DOMAIN_ID=69
# Subscription reliability. Keep best_effort: a RELIABLE subscriber receives
# NOTHING from a BEST_EFFORT publisher (the subscription is created, no error is
# raised, and the field silently stays null forever), while a BEST_EFFORT
# subscriber reads from either kind.
ENG_ROS_QOS=best_effort
# ── decimation (protects the robot's own CPU) ────────────────────────────────
# /hardware/joint_state publishes at 500 Hz and /hardware/motor_debug at 100 Hz.
# Telemetry resamples every 2 s, so running a Python callback on every message
# would burn the robot's compute for nothing. Seconds between PROCESSED messages:
ENG_JOINT_MIN_PERIOD=0.05
ENG_MOTOR_MIN_PERIOD=0.2
# ── field mapping (applies to EVERY backend) ─────────────────────────────────
# Dotted paths into the message/JSON. They work over both ROS message objects
# and plain dicts, support list indices ("cell_temp[0]") and wildcards
# ("joints[*].temp"), and yield null for any missing link — a wrong path
# degrades a field, it never crashes the agent. Empty = field unavailable.
ENG_FIELD_SOC=percentage
ENG_FIELD_VOLTAGE=voltage
ENG_FIELD_CURRENT=current
ENG_FIELD_CURRENT_LIMIT=current_limit
ENG_FIELD_POWER_ERR=error_code
ENG_FIELD_POWER_ENABLE=enable
# PowerInfo carries NO pack temperature / SOH / cycle count — leaving these
# empty reports null. Filling them with a wrong path would fabricate data.
ENG_FIELD_TEMP=
ENG_FIELD_SOH=
ENG_FIELD_CYCLES=
# MotorDebug — the PM01 DOES publish per-motor temperatures (25 of them), plus
# the driver MOSFET temperatures, plus per-motor fault/offline flags.
ENG_FIELD_TEMPS=motor_temperature
ENG_FIELD_MOS_TEMPS=mos_temperature
ENG_FIELD_MOTOR_ERR=error_code
ENG_FIELD_MOTOR_OFFLINE=offline
# JointState velocities → the "moving" status.
ENG_FIELD_VEL=velocity
# MotionState → control.mode + control.switchable_modes (read-only).
ENG_FIELD_MOTION=current_motion_task
ENG_FIELD_TRANSITIONS=available_transition_motions
ENG_FIELD_X=pose.pose.position.x
ENG_FIELD_Y=pose.pose.position.y
ENG_FIELD_FW=
# ── unit conventions (verified on the real robot) ────────────────────────────
# PowerInfo.percentage is already 0..100 → percent (never scale).
ENG_SOC_SCALE=percent
# +1 = positive current means CHARGING (the ROS BatteryState convention).
# MEASURED on this PM01: current stays POSITIVE (~2 A) while the pack drains
# (27%→26%, 54.74→54.58 V) — i.e. positive = DISCHARGE. Hence -1. Setting this
# to +1 would report a discharging robot as "charging" forever.
ENG_CURRENT_SIGN=-1
# PowerInfo already publishes volts and amps.
ENG_VOLTAGE_SCALE=1
ENG_CURRENT_SCALE=1
# ── position ─────────────────────────────────────────────────────────────────
# none | ros2 | rosbridge | http.
# The PM01's motion stack is a whole-body controller, not a navigation stack:
# there is NO odometry topic and no localisation running, so position is null.
# That is "not available", never a fabricated origin.
# ros2 — set ENG_TOPIC_ODOM too, the day a nav bringup publishes one
# http — read the Sanad nav API (works as soon as its bringup is alive)
# rosbridge — read /odom over the rosbridge websocket
ENG_POSITION_SOURCE=none
ENG_POSITION_URL=http://127.0.0.1:8014/api/nav/status
ENG_POSITION_FIELD_X=pose.x
ENG_POSITION_FIELD_Y=pose.y
ENG_POSITION_INTERVAL=2
ROSBRIDGE_URL=ws://127.0.0.1:9090
# ── identity / networking ────────────────────────────────────────────────────
# Which NIC's MAC is reported as the robot identity. wlP1p1s0 is the wifi NIC
# that carries the robot's LAN address.
MAC_INTERFACE=wlP1p1s0
# ── fault thresholds ─────────────────────────────────────────────────────────
LOW_SOC=50
MOTOR_TEMP_MAX=85
MOS_TEMP_MAX=100
# max |joint velocity| above which status becomes "moving"
MOVING_VEL=0.15
# ── cadence / transport ──────────────────────────────────────────────────────
POLL_INTERVAL=2
VERIFY_TLS=1
HTTP_TIMEOUT=30
TZ_OFFSET_HOURS=4
# ── map sync (uploaded ONCE per content PER SERVER; status in telemetry "map") ──
# pgm+yaml sets are rendered to PNG with pure stdlib; RTAB-Map .db is sent
# as-is (skipped above the server's upload cap).
MAPS_DIR=/home/ubuntu/sanad_t8/data/maps
EXTRA_MAP_DIRS=
DATA_DIR=/home/ubuntu/sanad_t8/data
STATE_DIR=/var/lib/sanad_api_eng
MAP_SELECT=all
MAP_UPLOAD_MODE=multipart
MAP_POLL_INTERVAL=30
MAP_MAX_UPLOAD_MB=7
WEB_NAV3_URL=
# ── logs + alerts ────────────────────────────────────────────────────────────
LOGS_INTERVAL=60
# auto = find a RUNNING sanad* docker container and tail its json-log. On this
# robot that is "sanad-t8". Reading it needs root, which the system service has.
PROJECT_LOG_CONTAINER=auto
PROJECT_LOG_PATH=
PROJECT_LOG_LABEL=
PROJECT_LOG_BACKFILL=100
ALERT_SCAN_INTERVAL=10
ALERT_LOG_COOLDOWN=300
# ── remote (register a dashboard URL + ssh for the fleet UI) ─────────────────
# 8014 is the Sanad Dashboard on this robot. 9002/9003 are EngineAI's own
# dashboard and Foxglove — kept as fallbacks if Sanad is ever stopped.
REMOTE_ENABLE=1
REMOTE_PORTS=8014,8001,9002,9003,8000,8080
REMOTE_KIND=web
REMOTE_INTERVAL=60
REMOTE_URL=
REMOTE_HOST=
SSH_REGISTER=1
# The robot's LOGIN user. Left empty, ssh registration is skipped rather than
# registering a wrong user that silently fails for whoever tries it.
SSH_USER=ubuntu
SSH_PORT=22
# ── control panel (READ-ONLY; remote mode-SWITCH is off by design) ───────────
# control.mode and control.switchable_modes come from /motion/motion_state — the
# robot's own words, no id table to guess. This URL only adds arm/teleop detail.
CONTROL_STATUS_URL=http://127.0.0.1:8014/api/controller/status
CONTROL_ENABLE=0
# ── CPU: raw subscriptions for the high-rate topics ─────────────────────────
# 1 (default) subscribes /hardware/joint_state and /hardware/motor_debug with
# raw=True, so the rate gate above runs BEFORE deserialization and a dropped
# message is never turned into a Python object. Measured 27.2% -> 23.8% of one
# core. Set 0 to use ordinary subscriptions (identical data, more CPU).
#
# Measured cost of each subscription, as % of ONE core (12 available):
# all four topics 26.5% | without joint_state 6.5% | without both 2.2%
# i.e. the 500 Hz joint_state is ~20% on its own. If you ever need that back,
# set ENG_TOPIC_JOINTS= (empty): the agent drops to ~6% and "status" stops
# reporting "moving" (control.mode still shows the real motion task).
ENG_RAW_SUBSCRIBE=1
# How often a repeated POST failure to the same server is reported in full.
# Suppressed occurrences are counted and shown on the next line that prints.
ERROR_LOG_COOLDOWN=60

204
agent/AGENT_README.md Normal file
View File

@ -0,0 +1,204 @@
# `sanad_api_eng.py` — internals
One process, five loops, one payload schema. Read `docs/PM01_INTERFACE.md` first
for what the robot exposes; this file is about how the agent is put together.
---
## Loops
| loop | cadence | what |
|---|---|---|
| telemetry | `POLL_INTERVAL` (2 s) | build payload → POST `/ingest/telemetry` → rising-edge alerts |
| map | `MAP_POLL_INTERVAL` (30 s) | scan `MAPS_DIR` → upload new content once per server |
| logs | `LOGS_INTERVAL` (60 s) | drain the agent's log ring + project-log tail → POST `/{sn}/logs` |
| alert scan | `ALERT_SCAN_INTERVAL` (10 s) | regex the project log → POST `/{sn}/alert` |
| remote | `REMOTE_INTERVAL` (60 s) | discover the Sanad dashboard → POST `/{sn}/remote` |
All four background loops are daemon threads; the telemetry loop is the main
thread. Any loop raising is caught and logged — one failing subsystem never
stops telemetry.
---
## The seam: `EngineAiSource.snapshot()`
The only robot-specific class. Whatever the backend, it fills one dict, and that
dict is the entire interface between "this robot" and the shared
telemetry / fault / status pipeline:
```python
{ "bms": {soc, current_a, voltage_v, temp_c, soh, cycles, current_limit_a} | None,
"state_age": float | None, # seconds since ANY topic last updated
"temps": [float], # motor_temperature[]
"mos_temps": [float], # mos_temperature[]
"motor_faults": ["joint7=0x2"], # error_code[] != 0
"motor_offline": [int], # offline[] != 0
"power_err": int | None, # PowerInfo.error_code
"power_enabled": bool | None, # PowerInfo.enable
"max_vel": float, # max |JointState.velocity|
"xy": {x, y} | None,
"motion": str | None, # MotionState.current_motion_task
"transitions": [str], # available_transition_motions
"fw": {} }
```
Porting to another EngineAI model means changing `.env` topic names, not this
class. Porting to a different vendor means writing one new class with the same
`snapshot()` contract.
Every ingest method is wrapped in `try/except: pass`. A wrong field mapping
degrades that one field to null; it never kills the subscription or the loop.
---
## `_dig(obj, "a.b[0].c")`
Walks a dotted path over **both** ROS message objects (`getattr`) and plain
dicts (`.get`) — so the same `ENG_FIELD_*` mapping works for the `ros2` and
`http` backends. Supports list indices (`cell_temp[0]`) and a wildcard
(`joints[*].temp`) that collects a field from every element and flattens.
Returns `None` for any missing link. That is the whole reason a mistyped
`.env` path shows up as one null field on the dashboard instead of a crash loop.
---
## Multi-server fan-out
`Config.endpoints` is a list of `Endpoint(name, url, token, enabled)`. Every
ingest call goes through `_post_each()`, which POSTs to each enabled endpoint
and returns `{server_name: {ok, code, error}}`.
Design points that are load-bearing:
- **Per-endpoint tokens.** `eco` and `eco-dev` maintain independent token
stores; a token minted on one 401s on the other. A single shared
`DEVICE_TOKEN` would silently fail on whichever server did not issue it.
- **Independent failure.** One server being down, slow or unauthorised cannot
block or fail the others — each POST is its own try/except.
- **Bytes, not file handles, for multipart.** A file handle streams **once**; with
two servers the second upload would send an empty body. Map blobs are read into
memory (capped by `MAP_MAX_UPLOAD_MB`) and posted to each.
- **Per-server map state.** `uploaded.json` is `{server: {path: fingerprint}}`.
With one shared key, enabling a second server later would find every map
already "uploaded" and that server would never receive them. The old flat
layout is migrated on read.
- **Throttled error reporting.** A persistently failing server would emit an
ERROR every 2 s forever, burying the log *and* filling the ring that gets
shipped to `/{sn}/logs`. Each `(what, server, status)` signature reports in
full at most once per `ERROR_LOG_COOLDOWN` (60 s); suppressed occurrences are
counted and shown on the next line that prints (`[+3 more since last report]`),
and the counter resets on success, so nothing is hidden — only de-duplicated.
---
## Status and faults
`derive_status()` returns the same vocabulary the X2 reports —
`charging | moving | idle | offline` — so one dashboard renders every robot:
```
no state at all + no battery → offline
current_a > 0.05 → charging
max |joint velocity| > 0.15 → moving (MOVING_VEL)
otherwise → idle
```
The *real* motion mode (`pd_sitdown`, `rl_basic`, …) goes in `control.mode`
rather than being squeezed into `status`, because it is a much larger vocabulary
than the dashboard's four states.
`derive_faults()` emits **strings**, not objects — the fleet ingest 500s on
fault objects. Codes: `LOW_BATTERY`, `MOTOR_OVERTEMP`, `MOS_OVERTEMP`,
`POWER_FAULT`, `POWER_DISABLED`, `MOTOR_FAULT`, `MOTOR_OFFLINE`, `COMMS_STALE`.
Alerts dedup on the **code before the first `:`**, never the whole string: every
fault embeds a live number (`battery 21%`, `no robot state for 12s`) that changes
almost every tick, so string-dedup re-fires the same alert every `POLL_INTERVAL`.
A code alerts on its rising edge, then at most once per `ALERT_LOG_COOLDOWN`
while it persists; clearing it makes the next occurrence a rising edge again.
---
## Rate decimation — measured, not assumed
`/hardware/joint_state` publishes at **500 Hz** and `/hardware/motor_debug` at
**100 Hz**. Telemetry resamples every 2 s, so nearly all of that is thrown away.
Measured on the robot, as % of **one** core (the Jetson has 12), by starting the
agent with subscriptions removed:
| configuration | CPU | implies |
|---|---|---|
| all four topics | 26.5% | — |
| without `joint_state` | 6.5% | `joint_state` ≈ **20%** |
| without `joint_state` + `motor_debug` | 2.2% | `motor_debug` ≈ **4.3%** |
| | | everything else ≈ **2.2%** |
So the 500 Hz stream was ~75% of the agent's cost, for one number.
Two mitigations, in order of how much they actually helped:
1. **`ENG_JOINT_MIN_PERIOD` / `ENG_MOTOR_MIN_PERIOD`** gate the callbacks to
20 Hz / 5 Hz. The gate is the first statement in every high-rate callback.
2. **`ENG_RAW_SUBSCRIBE=1`** (default) subscribes those two topics with
`raw=True`, so the gate runs *before* deserialization and a dropped message
is never turned into a Python object. Measured **27.2% → 23.8%**.
That second number is the interesting one: it is a real win but far smaller than
the hypothesis predicted, which means most of the remaining cost is CycloneDDS
delivering 500 msg/s and waking the executor — not building the message object.
The docstring records this so nobody re-derives it.
**Why it is not optimised further.** The remaining ~20% would disappear if the
subscription were created and destroyed around each sample (a ~12% duty cycle).
That would emit DDS endpoint-discovery traffic every 2 s onto `eth1` — the
network this robot's **motion controller** lives on (peer `192.168.0.163`).
Trading ~2% of an application core for periodic discovery churn on a realtime
control network is the wrong trade. The subscription stays stable.
If CPU ever matters more than the `moving` status, `ENG_TOPIC_JOINTS=` (empty)
drops it entirely and the agent costs ~6% of one core. `status` then reports
`idle` instead of `moving`, and `control.mode` still shows the real motion task.
---
## Temperature filtering
`_floats(v, 0, 200)` keeps only readings in `0 < t <= 200 °C`. A `0.0` in
`motor_temperature[]` means "slot not reporting", not "0 degrees": averaging it
in would drag `motor_temp.avg` down and mask a genuinely hot joint. `count` in
the payload is the number of *valid* readings, so a shrinking count is itself a
signal.
---
## Exit path
rclpy's CycloneDDS C++ threads are still running when Python finalises, so the
process dies with `SIGABRT` — *after* completing its work correctly. systemd
records that as a failed exit, which would mask a genuine crash.
`_exit(code)` therefore: shuts rclpy down, sleeps 200 ms so `spin()` unblocks,
flushes the log handlers, then `os._exit()` to skip the static destructors that
abort. `SIGINT`/`SIGTERM` handlers route into it, so `systemctl restart` records
`Deactivated successfully` instead of `Failed with result 'signal'`.
---
## CLI
```
--dry-run build every payload and print it; never POST (safe on a live feed)
--once one map pass + one real telemetry post, exit
--map-only upload discovered maps once; no state source, no telemetry
--simulate synthetic robot state (the map scan stays real)
--force re-upload maps even if unchanged
--list list discovered maps and exit
--interval override POLL_INTERVAL
-v debug logging (includes per-request urllib3 lines)
```
`--dry-run` is the right first move after any `.env` change: it exercises the
full read path and prints the exact JSON without touching the server.

12
agent/requirements.txt Normal file
View File

@ -0,0 +1,12 @@
# sanad_api_eng — EngineAI PM01 fleet agent.
requests>=2.25
# only needed if ENG_POSITION_SOURCE=rosbridge (reads /odom over a websocket):
# websocket-client>=1.6,<2
# NOT installed here:
# rclpy — comes from the robot's ROS 2 Humble install; not on
# PyPI and must match the robot's distro.
# interface_protocol — EngineAI's own message package, already built on the
# robot at /app/applications/install/interface_protocol.
# Both become importable once the unit sources ros_env.sh. The agent degrades to
# heartbeat mode when no state source is reachable, so a missing backend never
# stops telemetry.

37
agent/sanad-api-eng.service Normal file
View File

@ -0,0 +1,37 @@
[Unit]
Description=Sanad fleet agent (EngineAI PM01) — telemetry + map + logs + alerts
Documentation=file:/opt/sanad_api_eng/README.md
After=network-online.target docker.service
Wants=network-online.target
[Service]
Type=simple
# Runs as root, deliberately:
# * survives reboot with no `loginctl enable-linger` (the X2 deployment's one
# open item — a user unit there does NOT come back after a power cycle)
# * can read /var/lib/docker/containers/*/\*-json.log, which is how the Sanad
# app's logs reach the fleet server. A non-root user gets EACCES there and
# project_logs stays null forever.
User=root
WorkingDirectory=/opt/sanad_api_eng
# set +u is REQUIRED: ROS's setup.bash reads unbound variables and would abort
# the unit under set -u. ros_env.sh is the robot's OWN environment file — it
# pins ROS_DOMAIN_ID=69, RMW_IMPLEMENTATION=rmw_cyclonedds_cpp and
# CYCLONEDDS_URI (eth1). Sourcing it rather than restating those values means
# the agent follows the robot if EngineAI ever changes them.
ExecStart=/bin/bash -c 'set +u; \
. /opt/ros/humble/setup.bash >/dev/null 2>&1 || true; \
. /app/applications/install/bringup/ros_env.sh >/dev/null 2>&1 || true; \
exec /usr/bin/python3 -u /opt/sanad_api_eng/sanad_api_eng.py'
Restart=always
RestartSec=5
KillSignal=SIGINT
TimeoutStopSec=15
StandardOutput=journal
StandardError=journal
SyslogIdentifier=sanad-api-eng
[Install]
WantedBy=multi-user.target

2678
agent/sanad_api_eng.py Normal file
View File

File diff suppressed because it is too large Load Diff

262
docs/PM01_INTERFACE.md Normal file
View File

@ -0,0 +1,262 @@
# What the EngineAI PM01 actually exposes
Everything here was read off the live robot at `10.210.136.150` on 2026-08-27,
not from a datasheet. Each section names the `.env` variable it feeds.
---
## 1. The ROS 2 environment — the part that silently breaks
The PM01's stack does **not** run on the default ROS domain, and it does not use
the default DDS transport:
```bash
# /app/applications/install/bringup/ros_env.sh (the robot's own file)
export ROS_DOMAIN_ID=69
export RMW_IMPLEMENTATION=rmw_cyclonedds_cpp
export CYCLONEDDS_URI=file:///app/applications/install/bringup/cyclonedds.xml
source /opt/ros/humble/setup.bash
source /app/applications/install/setup.bash --extend
source /app/applications/install/bringup/product.env # PRODUCT=t800
```
and the CycloneDDS config pins discovery to one NIC:
```xml
<NetworkInterface name="eth1" /> <!-- 192.168.0.162 -->
<Peer address="192.168.0.163" /> <!-- the motion controller -->
```
Source only `/opt/ros/humble/setup.bash` and `ros2 topic list` returns **two**
topics (`/parameter_events`, `/rosout`) — no error, just an empty robot. Source
`ros_env.sh` and the same command returns **44**.
That is why the systemd unit sources `ros_env.sh` rather than restating
`ROS_DOMAIN_ID=69` itself: if EngineAI ever changes the domain, the interface
pinning or the peer address, the agent follows automatically.
`set +u` is required before sourcing — ROS's `setup.bash` reads unbound
variables and aborts the unit under `set -u`.
---
## 2. Topics the agent reads
All four are published continuously by the robot's own stack, verified with
`ros2 topic hz`:
| topic | type | rate | feeds |
|---|---|---|---|
| `/hardware/power_info` | `interface_protocol/msg/PowerInfo` | 20 Hz | `battery`, `charging`, `battery_detail` |
| `/hardware/motor_debug` | `interface_protocol/msg/MotorDebug` | 100 Hz | `motor_temp`, motor faults |
| `/hardware/joint_state` | `interface_protocol/msg/JointState` | 500 Hz | `status: moving` |
| `/motion/motion_state` | `interface_protocol/msg/MotionState` | 5 Hz | `control.mode`, `control.switchable_modes` |
The agent **subscribes only**. It never publishes, never calls a service, and
never requests a motion transition.
### The message definitions
```
# PowerInfo # MotorDebug
bool enable float32[] mos_temperature
float32 percentage float32[] motor_temperature
float32 voltage float32[] voltage
float32 current float32[] current
float32 current_limit int32[] error_code
int32 error_code uint8[] offline
uint8[] enable
# JointState # MotionState
std_msgs/Header header string current_motion_task
float64[] position string[] available_transition_motions
float64[] velocity
float64[] torque
```
### Rate decimation is not optional
`joint_state` at 500 Hz and `motor_debug` at 100 Hz mean 600 Python callbacks
per second on the robot's own Jetson — for values the agent resamples once every
2 s. `ENG_JOINT_MIN_PERIOD=0.05` and `ENG_MOTOR_MIN_PERIOD=0.2` decimate them to
20 Hz and 5 Hz. A monitoring agent must not tax the machine it monitors.
### QoS must stay `best_effort`
A `RELIABLE` subscriber receives **nothing** from a `BEST_EFFORT` publisher —
the subscription is created, no error is raised, and the field stays null
forever. A `BEST_EFFORT` subscriber reads from either kind, so
`ENG_ROS_QOS=best_effort` is the only setting that cannot silently fail.
---
## 3. Battery — the sign convention is inverted
`PowerInfo.percentage` is already `0..100`, so `ENG_SOC_SCALE=percent`.
The current sign is the trap. Measured over 15 minutes with the robot sitting
idle and **not** on a charger:
```
19:35:52 percentage: 27.0 voltage: 54.744 current: 1.865
19:36:18 percentage: 26.0 voltage: 54.693 current: 1.766
19:37:38 percentage: 26.0 voltage: 54.582 current: 1.703
19:52:16 percentage: 22.0 voltage: 53.3 current: 2.01
```
Percentage falls, voltage falls, **current stays positive**. So on the PM01
positive current means *discharging* — the opposite of the ROS `BatteryState`
convention the agent's default assumes.
Hence **`ENG_CURRENT_SIGN=-1`**. With the default `+1` the robot would report
`charging: true` and `status: "charging"` permanently while its battery drained
to zero — the failure would look like healthy telemetry, which is exactly the
kind of bug that survives review.
`PowerInfo` carries **no** pack temperature, state-of-health or cycle count, so
`battery_detail.temp_c` is null and `soh`/`cycles` are 0. Those fields are left
unmapped rather than pointed at a plausible-looking wrong field.
---
## 4. Motor temperature — real here, unlike the X2
The X2 agent reports `motor_temp: null` because that robot publishes no
per-motor temperature at all. The PM01 publishes **25 motor temperatures and 25
driver MOSFET temperatures at 100 Hz**, so this field carries real data:
```json
"motor_temp": { "max": 55.1, "avg": 29.5, "min": 24.0,
"count": 25, "mos_max": 45.6, "mos_avg": 29.9 }
```
Readings outside `0 < t <= 200 °C` are dropped: `0.0` means "slot not
reporting", and averaging it in would drag the fleet-wide average down and hide
a genuinely hot joint.
`MotorDebug` also carries per-motor `error_code[]` and `offline[]`, which become
the `MOTOR_FAULT` and `MOTOR_OFFLINE` alerts — real hardware fault channels the
X2 had no equivalent for.
---
## 5. Motion mode — the robot names its own state
The X2 had no documented FSM id scheme, so its `control.mode` reported
`"unknown"`. The PM01 publishes the mode **as a string**, together with the
exact set of transitions it will currently accept:
```yaml
current_motion_task: pd_sitdown
available_transition_motions:
- passive
- rl_mimic_sitdown_to_stance
```
which maps directly onto the telemetry `control` block with no lookup table to
guess:
```json
"control": { "mode": "pd_sitdown",
"switchable_modes": ["passive", "rl_mimic_sitdown_to_stance"],
"remote_switch_enabled": false,
"source": "ros2:/motion/motion_state" }
```
Observed task names so far: `passive`, `idle`, `pd_sitdown`,
`rl_mimic_sitdown_to_stance`, `rl_amp`, `rl_basic` (the last two from
`robot_manager`'s `notifier.yaml`).
`remote_switch_enabled` is `false` and `CONTROL_ENABLE=0`: the transitions are
**reported, never requested**. Switching motion mode on a humanoid is a motion
command, and this agent is read-only by design.
---
## 6. Position — genuinely unavailable
There is **no odometry topic on this robot**. `ros2 topic list` shows no
`/odom`, no `/tf`, no `amcl_pose`. The PM01's motion stack is a whole-body
controller, not a navigation stack: `/motion/data_monitor/base/*` carries yaw
and pelvis velocity for gait control, but nothing integrates a world pose.
The Sanad app's nav module is present but not running:
```json
{"bringup_alive": false, "rosbridge_alive": false, "reachable": false,
"mode": null, "active_map": null, "mode_label": "IDLE"}
```
So `ENG_POSITION_SOURCE=none` and `position` reports `null`. **Null means "not
available", never `{x: 0, y: 0}`** — a fabricated origin would put the robot at
the map corner on the fleet dashboard and look like real data.
Three ways to turn it on the day localisation runs, all `.env`-only:
```ini
ENG_POSITION_SOURCE=ros2 # + ENG_TOPIC_ODOM=/odom
ENG_POSITION_SOURCE=http # reads the Sanad /api/nav/status pose
ENG_POSITION_SOURCE=rosbridge # reads /odom over the rosbridge websocket
```
---
## 7. Maps
No saved maps exist on this robot: no `.pgm`/`.yaml` set and no RTAB-Map `.db`
anywhere under the Sanad data dir, and `/api/nav/maps` returns `[]`. The map
field therefore reports:
```json
"map": { "uploaded": false, "state": "no_map", "maps_found": 0,
"error": "no saved map found (maps_dir=…, robot=sanad)" }
```
The scanner is live and unchanged from the X2 agent — the moment a map is saved
under `MAPS_DIR` it is rendered to PNG and uploaded once per server.
---
## 8. Host / platform
| | |
|---|---|
| Board | NVIDIA Jetson AGX Orin Developer Kit |
| Board serial | `1421326045624` |
| L4T | R36.4.3, kernel `5.15.148-6-engine-tegra` |
| OS | Ubuntu 22.04.5 LTS, Python 3.10.12 |
| ROS | Humble, domain 69, CycloneDDS on `eth1` |
| Product tag | `t800` (config dir `pm01`) |
| Disk | 250.6 GB total, ~201 GB free |
| NIC (identity) | `wlP1p1s0` — `10.210.136.150`, MAC `6c:d5:52:cc:73:c4` |
| Other NICs | `eno1` 192.168.100.162, `eth1` 192.168.0.162 (DDS) |
| Timezone | **Asia/Shanghai** — the clock is correct in UTC, but local time reads +8. `TZ_OFFSET_HOURS=4` renders Dubai time in the `time` field. |
### Services on the robot
| port | what |
|---|---|
| 8014 | **Sanad Dashboard** (container `sanad-t8`) — registered as the fleet remote URL |
| 9001 | supervisord web UI |
| 9002 | EngineAI dashboard node |
| 9003 | Foxglove Studio (caddy) |
| 9004 | code-server |
| 8765 | foxglove_bridge |
The robot's own ROS apps run under **supervisord** (`/etc/supervisor/conf.d/ros_apps.conf`),
not systemd. The fleet agent deliberately does not join that group: a crash or
restart of the agent must never be able to take the robot's motion stack with it.
---
## 9. Project logs need root
The Sanad app runs in the docker container `sanad-t8`, and its log is at
`/var/lib/docker/containers/<id>/<id>-json.log` — root-owned, mode 600. The
`ubuntu` user gets `EACCES` and `project_logs` would stay null forever.
This is one of the two reasons the agent runs as a **root system service**
rather than a user service. The other is reboot survival: `Linger=no` on this
image and polkit denies `loginctl enable-linger` to a non-root user, so a
`--user` unit would not come back after a power cycle — the open item still
outstanding on the X2 deployment.

91
install.sh Normal file
View File

@ -0,0 +1,91 @@
#!/usr/bin/env bash
# install.sh — deploy the EngineAI PM01 fleet agent onto a robot.
#
# bash install.sh <ip> [user] [token]
# bash install.sh 10.210.136.150 ubuntu "$ECO_DEV_TOKEN"
#
# Idempotent: re-running updates the code and leaves an existing .env (and its
# tokens) untouched. Pass a token only on the FIRST install, or to rotate it.
#
# Auth: uses FLEET_SSH_KEY if set, else sshpass with FLEET_SSH_PASS
# (default "ubuntu"). Set FLEET_SSH_PASS in the environment — never edit it in.
set -euo pipefail
IP="${1:-}"
USER_="${2:-ubuntu}"
TOKEN="${3:-}"
[ -z "$IP" ] && { echo "usage: bash install.sh <ip> [user] [token]" >&2; exit 2; }
HERE="$(cd "$(dirname "$0")" && pwd)"
SSH_OPTS="-o StrictHostKeyChecking=accept-new -o ConnectTimeout=10"
if [ -n "${FLEET_SSH_KEY:-}" ]; then
SSH() { ssh $SSH_OPTS -i "$FLEET_SSH_KEY" "$USER_@$IP" "$@"; }
PUT() { scp $SSH_OPTS -i "$FLEET_SSH_KEY" "$1" "$USER_@$IP:$2"; }
elif command -v sshpass >/dev/null 2>&1; then
P="${FLEET_SSH_PASS:-ubuntu}"
SSH() { sshpass -p "$P" ssh $SSH_OPTS "$USER_@$IP" "$@"; }
PUT() { sshpass -p "$P" scp $SSH_OPTS "$1" "$USER_@$IP:$2"; }
else
SSH() { ssh $SSH_OPTS "$USER_@$IP" "$@"; }
PUT() { scp $SSH_OPTS "$1" "$USER_@$IP:$2"; }
fi
say() { printf '\n\033[1m== %s\033[0m\n' "$1"; }
say "staging files on $IP"
SSH 'mkdir -p ~/.sanad_stage'
PUT "$HERE/agent/sanad_api_eng.py" '~/.sanad_stage/'
PUT "$HERE/agent/.env.example" '~/.sanad_stage/'
PUT "$HERE/agent/requirements.txt" '~/.sanad_stage/'
PUT "$HERE/agent/sanad-api-eng.service" '~/.sanad_stage/'
PUT "$HERE/tools/probe_eng.sh" '~/.sanad_stage/'
say "installing (needs sudo on the robot)"
# The .env is created from .env.example ONLY if absent, so re-running never
# clobbers a live token. sudo -S reads the password from stdin when the robot
# has no passwordless sudo.
SSH "sudo -S -p '' bash -s" <<REMOTE
set -e
install -d -m 755 /opt/sanad_api_eng
install -d -m 750 /var/lib/sanad_api_eng
install -m 755 ~$USER_/.sanad_stage/sanad_api_eng.py /opt/sanad_api_eng/sanad_api_eng.py
install -m 644 ~$USER_/.sanad_stage/.env.example /opt/sanad_api_eng/.env.example
install -m 644 ~$USER_/.sanad_stage/requirements.txt /opt/sanad_api_eng/requirements.txt
install -m 755 ~$USER_/.sanad_stage/probe_eng.sh /opt/sanad_api_eng/probe_eng.sh
install -m 644 ~$USER_/.sanad_stage/sanad-api-eng.service /etc/systemd/system/sanad-api-eng.service
if [ ! -f /opt/sanad_api_eng/.env ]; then
install -m 600 -o root -g root /opt/sanad_api_eng/.env.example /opt/sanad_api_eng/.env
echo " created /opt/sanad_api_eng/.env from .env.example — SET SN AND TOKEN"
else
echo " kept existing /opt/sanad_api_eng/.env (tokens preserved)"
fi
chmod 600 /opt/sanad_api_eng/.env
if [ -n "$TOKEN" ]; then
sed -i "s#^DEVICE_TOKEN=.*#DEVICE_TOKEN=$TOKEN#" /opt/sanad_api_eng/.env
echo " DEVICE_TOKEN set"
fi
systemctl daemon-reload
systemctl enable sanad-api-eng >/dev/null 2>&1
systemctl restart sanad-api-eng
rm -rf ~$USER_/.sanad_stage
REMOTE
say "waiting for the first telemetry post"
sleep 12
SSH "sudo -S -p '' journalctl -u sanad-api-eng -n 25 --no-pager -o cat" \
| grep -E 'fleet server|subscribed|telemetry ok|ERROR|WARNING' || true
say "status"
SSH "sudo -S -p '' systemctl is-active sanad-api-eng; sudo -S -p '' systemctl is-enabled sanad-api-eng"
cat <<'DONE'
Next:
* set SN (and ROBOT_NAME) in /opt/sanad_api_eng/.env, then restart
* to add the second fleet server, set SERVER_2_TOKEN + SERVER_2_ENABLE=1
* live log: ssh <user>@<ip> 'sudo journalctl -u sanad-api-eng -f'
DONE

118
tools/probe_eng.sh Normal file
View File

@ -0,0 +1,118 @@
#!/usr/bin/env bash
# probe_eng.sh — read-only EngineAI PM01 discovery.
#
# bash tools/probe_eng.sh <ip> [user]
# bash tools/probe_eng.sh 10.210.136.150 ubuntu
#
# Prints what the robot ACTUALLY exposes, each line labelled with the .env
# variable it feeds. Installs nothing, writes nothing, publishes to no topic —
# safe to run against a live robot.
#
# Auth: uses an ssh key if FLEET_SSH_KEY is set, otherwise sshpass with
# FLEET_SSH_PASS (default "ubuntu"). Set FLEET_SSH_PASS in the environment
# rather than editing this file.
set -uo pipefail
IP="${1:-}"
USER_="${2:-ubuntu}"
if [ -z "$IP" ]; then
echo "usage: bash tools/probe_eng.sh <ip> [user]" >&2
exit 2
fi
SSH_OPTS="-o StrictHostKeyChecking=accept-new -o ConnectTimeout=8 -o BatchMode=no"
if [ -n "${FLEET_SSH_KEY:-}" ]; then
RUN() { ssh $SSH_OPTS -i "$FLEET_SSH_KEY" "$USER_@$IP" "$@"; }
elif command -v sshpass >/dev/null 2>&1; then
RUN() { sshpass -p "${FLEET_SSH_PASS:-ubuntu}" ssh $SSH_OPTS "$USER_@$IP" "$@"; }
else
RUN() { ssh $SSH_OPTS "$USER_@$IP" "$@"; }
fi
hdr() { printf '\n\033[1m== %s\033[0m\n' "$1"; }
# The robot's own environment file is the source of truth for ROS_DOMAIN_ID,
# RMW_IMPLEMENTATION and CYCLONEDDS_URI. Sourcing it (rather than guessing 0)
# is the difference between seeing 44 topics and seeing 2.
ROSENV='set +u
. /opt/ros/humble/setup.bash >/dev/null 2>&1
. /app/applications/install/bringup/ros_env.sh >/dev/null 2>&1'
hdr "host (-> SOFTWARE_*, MAC_INTERFACE, STORAGE_PATH)"
RUN "hostname; uname -srm; grep PRETTY_NAME /etc/os-release
echo -n 'board : '; tr -d '\0' < /proc/device-tree/model 2>/dev/null; echo
echo -n 'board_serial: '; tr -d '\0' < /proc/device-tree/serial-number 2>/dev/null; echo
echo -n 'l4t : '; head -1 /etc/nv_tegra_release 2>/dev/null
echo -n 'product : '; cat /app/applications/install/bringup/product.env 2>/dev/null
echo '--- NICs (MAC_INTERFACE = the one carrying the LAN address) ---'
ip -o -4 addr show | awk '{printf \" %-10s %-18s mac=\", \$2, \$4}
{\"cat /sys/class/net/\" \$2 \"/address\" | getline m; print m}'
echo '--- disk (STORAGE_PATH) ---'; df -h / | tail -1"
hdr "ROS env (-> ROS_DOMAIN_ID; the unit must source ros_env.sh)"
RUN "cat /app/applications/install/bringup/ros_env.sh 2>/dev/null
echo '--- CycloneDDS interface pinning ---'
cat /app/applications/install/bringup/cyclonedds.xml 2>/dev/null"
hdr "ROS topics (-> ENG_TOPIC_* / ENG_TYPE_*)"
RUN "$ROSENV; timeout 30 ros2 topic list -t 2>/dev/null"
hdr "message definitions (-> ENG_FIELD_*)"
RUN "for m in PowerInfo JointState MotorDebug MotionState; do
echo \"--- \$m ---\"
find /app/applications/install/interface_protocol -name \"\$m.msg\" \
-exec cat {} \; 2>/dev/null
echo
done"
hdr "live samples (-> verify ENG_SOC_SCALE / ENG_CURRENT_SIGN / units)"
RUN "$ROSENV
echo '--- /hardware/power_info (battery: percentage, voltage, current) ---'
timeout 12 ros2 topic echo --once /hardware/power_info 2>/dev/null
echo '--- /motion/motion_state (control.mode + switchable_modes) ---'
timeout 12 ros2 topic echo --once /motion/motion_state 2>/dev/null
echo '--- /hardware/motor_debug (motor_temp; first lines only) ---'
timeout 12 ros2 topic echo --once /hardware/motor_debug 2>/dev/null | head -12"
hdr "publish rates (-> ENG_JOINT_MIN_PERIOD / ENG_MOTOR_MIN_PERIOD)"
RUN "$ROSENV
for t in /hardware/power_info /hardware/joint_state /hardware/motor_debug /motion/motion_state; do
printf ' %-28s ' \"\$t\"
timeout 8 ros2 topic hz \$t 2>/dev/null | grep -m1 average || echo 'NO DATA'
done"
hdr "CHARGE DIRECTION (-> ENG_CURRENT_SIGN) ~45s"
echo " If percentage FALLS while current is POSITIVE, positive = discharge"
echo " and ENG_CURRENT_SIGN must be -1 (otherwise the robot reports 'charging'"
echo " forever while its battery drains)."
RUN "$ROSENV
for i in 1 2 3; do
printf ' %s ' \"\$(date +%H:%M:%S)\"
timeout 8 ros2 topic echo --once /hardware/power_info 2>/dev/null \
| tr '\n' ' ' | sed 's/---//'
echo
[ \$i -lt 3 ] && sleep 18
done"
hdr "position sources (-> ENG_POSITION_SOURCE / ENG_TOPIC_ODOM)"
RUN "$ROSENV
echo -n ' odometry topics: '
timeout 20 ros2 topic list 2>/dev/null | grep -iE 'odom|/tf$|amcl|pose' || echo 'NONE (position reports null)'
echo -n ' sanad nav status: '
curl -s -m5 http://127.0.0.1:8014/api/nav/status 2>/dev/null | head -c 220; echo"
hdr "dashboards / remote (-> REMOTE_PORTS, SSH_USER)"
RUN "echo '--- listening ports ---'; ss -tlnp 2>/dev/null | awk 'NR>1{print \" \" \$4}' | sort -u
for p in 8014 8001 9002 9003 9004 8765; do
printf ' %-6s ' \$p
curl -s -m3 -o /dev/null -w 'HTTP %{http_code}' http://127.0.0.1:\$p/ 2>/dev/null || printf 'closed'
curl -s -m3 http://127.0.0.1:\$p/ 2>/dev/null | grep -qi 'sanad\|dashboard' && printf ' <- dashboard page'
echo
done"
hdr "project logs (-> PROJECT_LOG_CONTAINER; needs root)"
RUN "sudo -n docker ps --format ' {{.Names}} {{.Status}}' 2>/dev/null \
|| echo ' (needs sudo; the agent runs as root and can read these)'"
hdr "done"
echo "Set the values above in agent/.env, then: systemctl restart sanad-api-eng"