202 lines
5.8 KiB
Markdown
202 lines
5.8 KiB
Markdown
# Marcus — Control & Startup Guide
|
|
|
|
**Updated**: 2026-04-06
|
|
|
|
---
|
|
|
|
## Quick Start
|
|
|
|
### Prerequisites (Jetson Orin NX)
|
|
|
|
```bash
|
|
# Terminal 1 — Start Holosoma (locomotion policy)
|
|
source ~/.holosoma_deps/miniconda3/bin/activate hsinference
|
|
cd ~/holosoma
|
|
~/.holosoma_deps/miniconda3/envs/hsinference/bin/python3 \
|
|
src/holosoma_inference/holosoma_inference/run_policy.py \
|
|
inference:g1-29dof-loco \
|
|
--task.model-path src/holosoma_inference/holosoma_inference/models/loco/g1_29dof/fastsac_g1_29dof.onnx \
|
|
--task.velocity-input zmq \
|
|
--task.state-input zmq \
|
|
--task.interface eth0
|
|
```
|
|
|
|
### Option A — Terminal Mode (on Jetson)
|
|
|
|
```bash
|
|
# Terminal 2 — Start Marcus Brain
|
|
conda activate Marcus
|
|
ollama serve & sleep 3
|
|
cd ~/Marcus
|
|
python3 run_marcus.py
|
|
```
|
|
|
|
Direct keyboard control. All commands typed locally.
|
|
|
|
### Option B — Server + Client (remote)
|
|
|
|
```bash
|
|
# Terminal 2 (Jetson) — Start Server
|
|
cd ~/Marcus
|
|
python3 -m Server.marcus_server
|
|
|
|
# Terminal 3 (Workstation) — Connect Client
|
|
cd ~/Robotics_workspace/yslootahtech/Project/Marcus
|
|
python3 -m Client.marcus_cli
|
|
```
|
|
|
|
Client prompts for connection:
|
|
```
|
|
Connection options:
|
|
1) eth0 — 192.168.123.164:8765
|
|
2) wlan0 — 10.255.254.86:8765
|
|
3) custom
|
|
Choose [1/2/3] or IP:
|
|
```
|
|
|
|
Or skip prompt: `python3 -m Client.marcus_cli --ip 192.168.123.164 --port 8765`
|
|
|
|
---
|
|
|
|
## Command Reference
|
|
|
|
### Movement
|
|
| Command | Action |
|
|
|---------|--------|
|
|
| `turn left` / `turn right` | Rotate (2s default) |
|
|
| `walk forward` / `move back` | Walk (2s default) |
|
|
| `walk 1 meter` | Precise odometry walk |
|
|
| `walk backward 2 meters` | Precise backward walk |
|
|
| `turn right 90 degrees` | Precise odometry turn |
|
|
| `turn right then walk forward` | Multi-step compound |
|
|
| `come to me` / `come here` | Forward 2s (instant, no AI) |
|
|
| `stop` | Gradual stop |
|
|
|
|
### Vision
|
|
| Command | Action |
|
|
|---------|--------|
|
|
| `what do you see` | LLaVA describes camera view |
|
|
| `describe the room` | LLaVA scene description |
|
|
| `is anyone here` | LLaVA person check |
|
|
| `yolo` | Show YOLO detection status |
|
|
|
|
### Goal Navigation
|
|
| Command | Action |
|
|
|---------|--------|
|
|
| `goal/ stop when you see a person` | YOLO fast search + stop |
|
|
| `goal/ find a laptop` | YOLO + LLaVA search |
|
|
| `goal/ stop when you see a guy holding a phone` | YOLO + LLaVA compound verification |
|
|
| `find a person` | Auto-detected as goal (no prefix needed) |
|
|
| `look for a bottle` | Auto-detected as goal |
|
|
|
|
### Place Memory
|
|
| Command | Action |
|
|
|---------|--------|
|
|
| `remember this as door` | Save current position |
|
|
| `go to door` | Navigate to saved place |
|
|
| `places` | List all saved places |
|
|
| `forget door` | Delete place |
|
|
| `rename door to entrance` | Rename place |
|
|
| `where am I` | Show odometry position |
|
|
| `go home` | Return to start position |
|
|
|
|
### Patrol
|
|
| Command | Action |
|
|
|---------|--------|
|
|
| `patrol` | Autonomous patrol (prompts for duration) |
|
|
| `patrol: door → desk → exit` | Named waypoint patrol |
|
|
|
|
### Image Search
|
|
| Command | Action |
|
|
|---------|--------|
|
|
| `search/ /path/to/photo.jpg` | Find target from reference image |
|
|
| `search/ /path/to/photo.jpg person in blue shirt` | Image + hint |
|
|
| `search/ person in blue shirt` | Text-only search |
|
|
|
|
### Session Memory
|
|
| Command | Action |
|
|
|---------|--------|
|
|
| `last command` | Show last typed command |
|
|
| `do that again` | Repeat last command |
|
|
| `undo` | Reverse last movement |
|
|
| `last session` | Previous session summary |
|
|
| `session summary` | Current session stats |
|
|
|
|
### System
|
|
| Command | Action |
|
|
|---------|--------|
|
|
| `help` | Command reference |
|
|
| `example` | Usage examples |
|
|
| `q` / `quit` | Shutdown |
|
|
|
|
### Client-Only Commands (CLI)
|
|
| Command | Action |
|
|
|---------|--------|
|
|
| `status` | Ping server + LiDAR status |
|
|
| `camera` | Get camera configuration |
|
|
| `profile low/medium/high/full` | Switch camera profile |
|
|
| `capture` | Take a photo |
|
|
|
|
---
|
|
|
|
## Network Configuration
|
|
|
|
| Interface | IP | Use |
|
|
|-----------|-----|------|
|
|
| `eth0` | 192.168.123.164 | Robot internal network (Jetson - G1 - LiDAR) |
|
|
| `wlan0` | 10.255.254.86 | Office WiFi (Jetson - Workstation) |
|
|
|
|
| Service | Port | Protocol |
|
|
|---------|------|----------|
|
|
| Marcus WebSocket | 8765 | ws:// |
|
|
| ZMQ Velocity | 5556 | tcp:// (PUB/SUB) |
|
|
| Ollama API | 11434 | HTTP |
|
|
| LiDAR | 192.168.123.120 | Livox Mid360 |
|
|
|
|
All configurable in `Config/config_Network.json`.
|
|
|
|
---
|
|
|
|
## Subsystem Status
|
|
|
|
On startup, the server/brain shows:
|
|
```
|
|
YOLO : active (19 tracked classes, CPU, yolov8m.pt)
|
|
Odometry : active (dead reckoning, +/-10cm)
|
|
Memory : active (session_016_2026-04-06)
|
|
Camera : 424x240@15 (RealSense D435I)
|
|
LiDAR : ALIVE (Livox Mid360 at 192.168.123.120)
|
|
Arms : pending (GR00T N1.5 not yet integrated)
|
|
```
|
|
|
|
---
|
|
|
|
## Troubleshooting
|
|
|
|
| Issue | Cause | Fix |
|
|
|-------|-------|-----|
|
|
| `ModuleNotFoundError: No module named 'Server'` | Wrong directory | `cd ~/Marcus` then run |
|
|
| Robot doesn't move | Holosoma not running | Start Holosoma first (Terminal 1) |
|
|
| Robot doesn't move | ZMQ port conflict | Only run one of Server or Brain, not both |
|
|
| `Camera: {e} reconnecting` | USB bandwidth | Reduce to `low` profile |
|
|
| LLaVA slow (>10s) | GPU VRAM full | Kill other GPU processes, or use `qwen2.5vl:3b` |
|
|
| `YOLO not available` | ultralytics not installed | `pip install ultralytics` |
|
|
| Client can't connect | Wrong IP or server not running | Check `status` command, verify IP |
|
|
|
|
---
|
|
|
|
## File Locations
|
|
|
|
| What | Path |
|
|
|------|------|
|
|
| Brain code | `~/Marcus/Brain/` |
|
|
| Server | `~/Marcus/Server/marcus_server.py` |
|
|
| Config | `~/Marcus/Config/` |
|
|
| Prompts | `~/Marcus/Config/marcus_prompts.yaml` |
|
|
| YOLO model | `~/Marcus/Models/yolov8m.pt` |
|
|
| Session data | `~/Marcus/Data/Brain/Sessions/` |
|
|
| Places | `~/Marcus/Data/History/Places/places.json` |
|
|
| Logs | `~/Marcus/logs/` |
|
|
|
|
See `Doc/architecture.md` for full project structure and file-by-file documentation.
|