Sidra 5ce59ef443 Update 2026-08-27 16:24

2026-08-27 16:24:12 +04:00

14 KiB

Raw Blame History

EngineAI Fleet — EngineAI PM01 agent

On-robot agent that reports the EngineAI PM01 ("T800")'s live state to the YS Lootah fleet server / Eco system.

Same concept, same payload schema and same endpoints as the AGIBOT X2 agent (agi_fleet/), retargeted to the PM01 — and extended to post the identical payload to more than one fleet server at once.

Status: LIVE. Running on 10.210.136.150 as the system service sanad-api-eng, posting to https://eco-dev.yslootahrobotics.com every 2 s. https://eco.yslootahrobotics.com is configured and one line away from enabled — see §5.

telemetry ok: battery=21 charging=False status=idle mode=pd_sitdown pos=None
              motor_max=55.0 faults=1 map=no_map -> eco-dev=200

1. Layout

engineai_fleet/
├── README.md                  ← this file
├── agent/
│   ├── sanad_api_eng.py       ← THE AGENT (deployed as-is)
│   ├── .env.example           ← every setting, documented
│   ├── requirements.txt       ← requests (rclpy comes from the robot's ROS)
│   ├── sanad-api-eng.service  ← systemd unit (root, survives reboot)
│   └── AGENT_README.md        ← agent internals: backends, field mapping
├── tools/
│   └── probe_eng.sh           ← read-only robot discovery
└── docs/
    └── PM01_INTERFACE.md      ← what the PM01 actually exposes

2. The robot


Host	`10.210.136.150` (NIC `wlP1p1s0`)
SSH	`ubuntu@10.210.136.150`
Display name	`pm01_150`
Hardware	NVIDIA Jetson AGX Orin, arm64, L4T R36.4.3
Board serial	`1421326045624`
OS	Ubuntu 22.04.5 LTS, kernel `5.15.148-6-engine-tegra`
ROS	Humble, `ROS_DOMAIN_ID=69`, CycloneDDS pinned to `eth1`
Product tag	`t800` (config dir `pm01`)
Install dir	`/opt/sanad_api_eng/`
Service	`systemctl sanad-api-eng` (system unit, root)

3. How it gets its data

Four ROS 2 topics from the robot's own stack, all read-only:

what	source
battery, voltage, current, charging	`/hardware/power_info` (`interface_protocol/msg/PowerInfo`)
motor + MOSFET temperatures, motor faults	`/hardware/motor_debug` (`MotorDebug`) — 25 motors
locomotion → `status: moving`	`/hardware/joint_state` (`JointState`) velocities
motion mode + allowed transitions	`/motion/motion_state` (`MotionState`)
OS, kernel, arch, board, L4T, storage, MAC	read directly from the host
project logs	docker json-log of the `sanad-t8` container

The field mapping actually in use

ENG_SOURCE=ros2
ENG_TOPIC_POWER=/hardware/power_info    ENG_TYPE_POWER=interface_protocol/msg/PowerInfo
ENG_TOPIC_MOTORS=/hardware/motor_debug  ENG_TYPE_MOTORS=interface_protocol/msg/MotorDebug
ENG_TOPIC_JOINTS=/hardware/joint_state  ENG_TYPE_JOINTS=interface_protocol/msg/JointState
ENG_TOPIC_MOTION=/motion/motion_state   ENG_TYPE_MOTION=interface_protocol/msg/MotionState

ENG_FIELD_SOC=percentage       ENG_FIELD_VOLTAGE=voltage
ENG_FIELD_CURRENT=current      ENG_FIELD_TEMPS=motor_temperature
ENG_FIELD_MOS_TEMPS=mos_temperature
ENG_FIELD_MOTION=current_motion_task
ENG_SOC_SCALE=percent          ENG_CURRENT_SIGN=-1     # ← measured, see §4
ENG_ROS_QOS=best_effort        ROS_DOMAIN_ID=69

ENG_JOINT_MIN_PERIOD=0.05      # 500 Hz → 20 Hz  (protects the robot's CPU)
ENG_MOTOR_MIN_PERIOD=0.2       # 100 Hz →  5 Hz

MAC_INTERFACE=wlP1p1s0
REMOTE_PORTS=8014,8001,9002,9003,8000,8080

Nothing above is hard-coded in the agent — it is all .env. Retargeting is an .env edit + restart, never a code change.

4. What it sends

One JSON object every 2 s to POST /api/v1/fleet/ingest/telemetry with Authorization: Bearer <token>, plus /{sn}/alert, /{sn}/logs, /{sn}/map and /{sn}/remote — to every enabled server. Verified field-by-field against the robot's own readings:

robot ground truth	agent sends
`power_info percentage: 21.0`	`battery: 21`
`power_info voltage: 53.3`	`voltage_v: 53.3`
`power_info current: +2.47` (discharging)	`current_a: -2.47`, `charging: false`
`power_info current_limit: 90.0`	`current_limit_a: 90.0`
`motor_debug motor_temperature[25]`	`motor_temp: {max:55.1, avg:29.5, min:24.0, count:25}`
`motor_debug mos_temperature[25]`	`motor_temp.mos_max: 45.6, mos_avg: 29.9`
`motion_state current_motion_task`	`control.mode: "pd_sitdown"`
`motion_state available_transition_motions`	`control.switchable_modes: [...]`
`df /` → 250.6 GB, 201 GB free	`storage: {total_gb:250.64, free_gb:201.3, used_percent:14.6}`

Also registers the Sanad Dashboard (http://10.210.136.150:8014) and ssh ubuntu@10.210.136.150 for the fleet UI, and ships the Sanad app's logs as project_logs: "sanad-t8-logs".

The battery sign — the one trap on this robot

PowerInfo.current stays positive while the pack drains. Measured over 15 minutes, idle and off-charger: 27% → 22%, 54.74 V → 53.3 V, current +1.7 … +2.5 A throughout. So positive means discharging here — the opposite of the ROS BatteryState convention.

Hence ENG_CURRENT_SIGN=-1. With the default +1 the robot would report charging: true and status: "charging" forever while its battery went flat — a failure that looks exactly like healthy telemetry.

Fields that are `null` — and why

position — the PM01 publishes no odometry topic at all (no /odom, no /tf, no amcl_pose); its motion stack is a whole-body controller, not a navigation stack, and the Sanad nav bringup is not running. null means "not available", never {x: 0, y: 0} — a fabricated origin would park the robot in the corner of the fleet map and look like real data. One .env line turns it on the day localisation runs (ENG_POSITION_SOURCE).
battery_detail.temp_c / soh / cycles — PowerInfo carries no pack temperature, state-of-health or cycle count. Left unmapped rather than pointed at a plausible-looking wrong field.
map — no_map; there are no saved maps on this robot yet.

What this robot reports that the X2 could not

motor_temp (the X2 publishes no per-motor temperature at all — permanently null there), control.mode by name plus the real transition set (the X2 had no documented FSM id scheme and reported "unknown"), and three hardware fault channels: POWER_FAULT/POWER_DISABLED from PowerInfo.error_code/enable, and MOTOR_FAULT/MOTOR_OFFLINE from MotorDebug.error_code[]/offline[].

5. Two fleet servers, two tokens

The X2 agent posts to one server. This one keeps a list of endpoints and sends the identical payload to each:

SERVER_URL=https://eco-dev.yslootahrobotics.com     # server 1
DEVICE_TOKEN=<eco-dev token>
SERVER_ENABLE=1

SERVER_2_URL=https://eco.yslootahrobotics.com       # server 2
SERVER_2_TOKEN=                                     # ← paste eco's token
SERVER_2_ENABLE=0                                   # ← then flip to 1

The supplied token works on eco-dev and is rejected by eco. Tested from the robot: eco-dev returns 200 {"ok":true,"robot_id":162}; eco returns 401 for that token, for a garbage token and for no token at all — i.e. it is the application's auth layer, not a firewall (the host is reachable; / answers 302). The two deployments maintain independent token stores.

So server 2 ships configured but disabled. To turn it on:

ssh ubuntu@10.210.136.150
sudo sed -i 's#^SERVER_2_TOKEN=.*#SERVER_2_TOKEN=<eco-token>#; \
             s#^SERVER_2_ENABLE=.*#SERVER_2_ENABLE=1#' /opt/sanad_api_eng/.env
sudo systemctl restart sanad-api-eng

Every log line then carries both results:

telemetry ok: battery=21 … -> eco-dev=200 eco=200

A failing server never blocks the other — each POST is independent, and repeated failures are reported once per minute with a suppressed-count rather than once every 2 s:

ERROR telemetry -> eco FAILED: HTTP 401 Unauthorized  <- token rejected by eco;
      each fleet server issues its OWN token …  [+3 more since last report]
INFO  telemetry ok: battery=21 … -> eco-dev=200 eco=401

Maps track upload state per server, so enabling eco later uploads every existing map to it rather than finding them already marked "uploaded".

6. Operating it

# live log
ssh ubuntu@10.210.136.150 'sudo journalctl -u sanad-api-eng -f'

# what it is sending right now
ssh ubuntu@10.210.136.150 \
  "sudo journalctl -u sanad-api-eng -n 20 --no-pager | grep -oE 'telemetry ok:.*'"

# service control
ssh ubuntu@10.210.136.150 'sudo systemctl restart sanad-api-eng'
ssh ubuntu@10.210.136.150 'sudo systemctl status  sanad-api-eng'

# change a setting (then restart)
ssh ubuntu@10.210.136.150 'sudo nano /opt/sanad_api_eng/.env && sudo systemctl restart sanad-api-eng'

Updating the agent code

scp agent/sanad_api_eng.py ubuntu@10.210.136.150:/tmp/
ssh ubuntu@10.210.136.150 \
  'sudo install -m755 /tmp/sanad_api_eng.py /opt/sanad_api_eng/ && sudo systemctl restart sanad-api-eng'

.env is never overwritten by this, so the tokens stay put.

Changing the serial

SN is the primary key for every /{sn}/ route. Changing it creates a new robot entry on the server rather than renaming the existing one:

ssh ubuntu@10.210.136.150 \
  "sudo sed -i 's#^SN=.*#SN=<real-serial>#' /opt/sanad_api_eng/.env && \
   sudo systemctl restart sanad-api-eng"

Re-discovering the robot's interface

bash tools/probe_eng.sh 10.210.136.150 ubuntu

Read-only — no install, no writes, publishes to no topic, safe on a live robot. Every line it prints is labelled with the .env variable it feeds.

Verifying without touching the live feed

sudo -E python3 /opt/sanad_api_eng/sanad_api_eng.py --dry-run   # builds payloads, never POSTs
sudo -E python3 /opt/sanad_api_eng/sanad_api_eng.py --once      # one real post, then exits

(both need the ROS overlay: set +u; . /app/applications/install/bringup/ros_env.sh)

7. Why it runs as a root system service

The X2 agent runs as a --user unit and has one open item: linger is off, so it does not come back after a power cycle. That is fixed here by running a system unit, which also solves a second problem:

Reboot survival — systemctl enabled, no loginctl enable-linger needed. Verified enabled + active.
Project logs — the Sanad app runs in the sanad-t8 container and its log lives at /var/lib/docker/containers/<id>/<id>-json.log, root-owned mode 600. As ubuntu that read is EACCES and project_logs would stay null forever.

The unit sources the robot's own environment file so the agent follows EngineAI's settings instead of restating them:

ExecStart=/bin/bash -c 'set +u; \
  . /opt/ros/humble/setup.bash >/dev/null 2>&1 || true; \
  . /app/applications/install/bringup/ros_env.sh >/dev/null 2>&1 || true; \
  exec /usr/bin/python3 -u /opt/sanad_api_eng/sanad_api_eng.py'

set +u is required — ROS's setup.bash reads unbound variables and would abort the unit under set -u. ros_env.sh is what sets ROS_DOMAIN_ID=69, RMW_IMPLEMENTATION=rmw_cyclonedds_cpp and the CycloneDDS interface pinning; without it ros2 topic list shows 2 topics instead of 44 and every field silently reports null.

The robot's own ROS apps run under supervisord, not systemd. The agent deliberately does not join that group: a restart of the agent must never be able to disturb the robot's motion stack.

Verified behaviour


`kill -9`	auto-restarts in ~5 s (`Restart=always`), `NRestarts: 1`
`systemctl restart`	`received SIGINT — stopping` → `Deactivated successfully` — clean, no abort
`systemctl is-enabled`	`enabled` — starts at boot

The clean stop matters: rclpy's CycloneDDS threads abort the process at interpreter teardown (terminate called without an active exception), which systemd records as a failed exit and which would mask a real crash. The agent shuts ROS down explicitly on SIGINT/SIGTERM.

8. Security

Outbound only. No inbound port is opened on the robot; every call is an HTTPS POST carrying Authorization: Bearer <token>, keyed by sn.
Read-only toward the robot. It subscribes to four topics and reads two local HTTP status pages. It never publishes, never calls a service, and never commands motion. /motion/motion_state's transition list is reported, never requested; CONTROL_ENABLE=0.
Token handling. Tokens live only in /opt/sanad_api_eng/.env on the robot, mode 600 root-owned, and are .gitignored here. Only .env.example (placeholders) is in this repo.
TLS verified (VERIFY_TLS=1).
Rate-limited. The 500 Hz and 100 Hz streams are gated to 20 Hz / 5 Hz and subscribed raw, so a dropped message is never deserialized. Measured cost: ~24% of one core out of 12 (~2% of the machine), on the application processor — the realtime motion controller is a separate board. See agent/AGENT_README.md for the per-subscription breakdown and why it is not optimised further.

9. Relationship to the other fleet agents

sanad_api_eng.py is standalone. It shares no files with agi_fleet/ (the X2 agent) or with the Unitree fleet/ agents (G1 / R1 / Go2), and contains zero AgiBot or Unitree code. The three can be changed independently.

It does deliberately keep the X2 agent's telemetry schema, endpoint paths, env-var discipline and map/logs/alerts/remote loops unchanged, so the same fleet server ingests all of them identically — the only structural difference is the multi-server endpoint list described in §5.

14 KiB Raw Blame History