Marcus/Doc/pipeline.md

Marcus — End-to-End Pipeline

Robot persona: Sanad (wake word + self-intro) Updated: 2026-04-21

One map of every data path from sensor to motor, voice to speech. Cross-reference with architecture.md (what each file is), functions.md (exact function signatures — AST-generated), and MARCUS_API.md (usage examples + JSON schemas).

---

Boot sequence

Brain/marcus_brain.py::init_brain() — called once from run_marcus.py or marcus_server.py.

run_marcus.py
      │
      ▼
init_brain()
      │
      ├─ init_zmq()                            PUB bind tcp://127.0.0.1:5556 → Holosoma
      ├─ start_camera()                        RealSense 424×240@15fps → shared _raw_frame
      ├─ init_yolo(raw_frame, raw_lock)        YOLOv8m CUDA FP16, 19 classes — background thread
      ├─ init_odometry()                       ROS2 /dog_odom → dead reckoning fallback
      ├─ init_memory()                         loads Data/Brain/Sessions/session_NNN/
      │
      ├─ if subsystems.lidar:       init_lidar()         multiprocessing spawn SLAM_worker
      ├─ if subsystems.imgsearch:   init_imgsearch()     (off by default)
      ├─ if subsystems.autonomous:  AutonomousMode()     patrol state machine
      │
      ├─ send_cmd("start") + 0.5s + send_cmd("walk") + 0.5s   Holosoma handshake
      │
      ├─ if subsystems.voice:       _init_voice()        ▼ voice pipeline below
      └─ _warmup_llava()                        first Qwen2.5-VL inference
                                                "SANAD AI BRAIN — READY"

Subsystem flags live in config_Brain.json::subsystems. Current defaults:

"subsystems": { "lidar": true, "voice": true, "imgsearch": false, "autonomous": true }

---

Voice pipeline (when subsystems.voice = true)

G1 body mic (array)
  └─ UDP multicast 239.168.123.161:5555 ── int16 mono 16 kHz PCM
        ▼
Voice/builtin_mic.py::BuiltinMic
  ring buffer (64 KB) + read_chunk(n)
        ▼
Voice/wake_detector.py::WakeDetector
  pure-numpy energy state machine  (SILENCE ⇄ SPEAKING)
  adaptive noise floor: eff_threshold = max(speech_threshold, baseline × 3)
  fires on 0.35-1.5 s bursts followed by 0.3 s silence → captures burst audio
        ▼
Voice/marcus_voice.py::VoiceModule._handle_wake()
  ├─ 1. Whisper verify on the burst audio:
  │        text = faster-whisper(burst)
  │        accept if _has_wake_word(text)  OR  startswith(s/sh/z)
  │        reject otherwise (cough, clap, hello, okay) → silent return
  ├─ 2. audio_api.speak("Yes")  → G1 body speaker (~1.5 s)
  ├─ 3. post_tts_settle_sec wait + mic flush
  ├─ 4. _record_command()  — hysteretic VAD
  │        speech_entry_rms / silence_exit_rms (adapt from wake baseline)
  │        trim leading silence (keep 300 ms pre-roll) → tight clip for Whisper
  ├─ 5. _transcribe(audio)
  │        faster-whisper (base.en int8 CPU)
  │        beam_size=5, temperature=0, initial_prompt bias toward Sanad vocab
  │        GARBAGE_PATTERNS + min_transcription_length reject noise hallucinations
  ├─ 6. _normalize_command(text)
  │        difflib fuzzy-match vs stt.command_vocab
  │        "Turn right up" → "turn right"  (canonical form)
  └─ 7. on_command(text, "en")
        ▼
Brain/marcus_brain.py::process_command(text)
  ├─ regex fast-path → Brain/command_parser.py::try_local_command()
  │    places · odometry walk/turn · patrol · session recall · goal_nav
  │    + SIMPLE_DIR ("go back", "right", "forward") · STOP_SIMPLE ("stop", "halt")
  │    + NAT_GOAL_RE (naturalised goals like "the chair") · auto on/off
  │    (~50 ms when matched — NO LLM call)
  └─ else → _handle_llava(text)
        ├─ get_frame()  (10×50 ms poll, no 1 s stall)
        ├─ API/llava_api.py::ask(text, img)
        │    ollama.chat(qwen2.5vl:3b, num_batch=128, num_ctx=2048, num_predict=120)
        │    → parse_json() → {actions, arm, speak, abort}
        └─ Brain/executor.py::execute(d)
                ├─ actions → MOVE_MAP[dir] → API/zmq_api.py::send_vel → Holosoma
                ├─ arm     → API/arm_api.py  (stub for now)
                └─ abort   → gradual_stop()
        ▼
result["speak"]  →  audio_api.speak(reply)
        ▼
API/audio_api.py::speak(text, lang="en")
  ├─ mute mic (flush BuiltinMic buffer)
  ├─ Voice/builtin_tts.py::BuiltinTTS.speak(text)
  │    client.TtsMaker(text, speaker_id=0)   — G1 on-board engine, English only
  │    time.sleep(len(text) * 0.08)
  └─ unmute mic → back to listening

Config knobs (all in config_Voice.json::stt):

• Wake: speech_threshold (floor), min_word_duration, max_word_duration, post_silence, wake_cooldown, wake_adaptive_mult, wake_diag_log_sec
• Verify: wake_verify_enabled
• Record: speech_entry_rms, silence_exit_rms, silence_duration_sec, max_record_sec, min_record_sec, ambient_mult, ambient_cap_rms
• Whisper: whisper_model, whisper_compute_type, whisper_beam_size, whisper_no_speech_threshold, whisper_log_prob_threshold, whisper_initial_prompt, mic_gain
• Vocab: wake_words, command_vocab, garbage_patterns, command_vocab_cutoff, min_transcription_length
• Mode: mode (wake_and_command | always_on | always_on_gated), wake_ack (tts|none)

---

Terminal / WebSocket command pipeline (same brain, skips voice)

run_marcus.py stdin   OR   Server/marcus_server.py WebSocket
        ▼
Brain/marcus_brain.py::process_command(text)
        ▼  (same parser → LLaVA → executor → ZMQ as above)
        ▼
result dict  →  stdout   OR   WebSocket reply frame

---

Vision pipeline (continuous, consumed by brain on demand)

RealSense D435 (USB)
  └─ 424×240 BGR 15 fps
      → API/camera_api.py — shared _raw_frame (thread-safe)
                    │                  │
                    │                  └─ get_frame() → JPEG base64 on demand
                    ▼
       Vision/marcus_yolo.py (daemon thread)
       YOLOv8m @ cuda:0 FP16 imgsz=320
       → _latest_detections (thread-safe list)
         yolo_sees / yolo_closest / yolo_summary / yolo_fps
                    ▼
       Navigation/goal_nav.py  (fast YOLO check → Qwen-VL fallback)
       Autonomous/marcus_autonomous.py  (patrol scan every N steps)
       Brain/marcus_brain.py  (status / alerts)

---

Movement pipeline

Brain/executor.py  OR  Brain/command_parser.py  OR  Navigation/*
        │   uses MOVE_MAP from config_Navigation.json
        ▼
API/zmq_api.py::send_vel(vx, vy, vyaw)  JSON over ZMQ PUB (port 5556)
        ▼
Holosoma RL policy (separate process, hsinference env)
        ▼
G1 low-level joint commands over DDS/eth0
        ▼
29-DOF body motion

---

LiDAR pipeline (when subsystems.lidar = true)

Livox Mid-360 (192.168.123.120, UDP)
        ▼
Lidar/SLAM_worker.py  (multiprocessing.spawn subprocess — CUDA-safe spawn)
    ├─ SLAM_engine, SLAM_Filter, SLAM_LoopClosure, SLAM_Submap, SLAM_NavRuntime
    ├─ publishes pose + obstacle flags back to parent via Queue
    └─ writes occupancy grids to Data/Navigation/Maps/
        ▼
API/lidar_api.py  (reads the queues, exposes:)
        ├─ obstacle_ahead() → bool
        ├─ get_lidar_status() → dict (pose, loc_state, frame age, FPS, ICP ms)
        └─ LIDAR_AVAILABLE
        ▼
Navigation/goal_nav.py rotation thread — pauses motion on obstacle_ahead()
Brain/command_parser.py — responds to "lidar status" queries

---

Knobs that control each stage

Knob	Location	Effect
subsystems.lidar	config_Brain.json	SLAM subprocess on/off
subsystems.voice	config_Brain.json	BuiltinMic + Whisper + TtsMaker loop on/off
subsystems.imgsearch	config_Brain.json	image-guided search init on/off
subsystems.autonomous	config_Brain.json	auto-patrol state machine init on/off
num_batch, num_ctx	config_Brain.json	llama.cpp compute-graph size (128 / 2048 ≈ 1.8 GiB graph — do not raise on 16 GB Jetson)
num_predict_main	config_Brain.json	120 tokens max for the main JSON reply
yolo_device, yolo_half	config_Vision.json	cuda / FP16 (hard-required; CPU not allowed)
mic.backend	config_Voice.json	builtin_udp (G1 array) or pactl_parec (Hollyland fallback)
mic_udp.group/port	config_Voice.json	where to join the G1 audio multicast
mic_udp.read_timeout_sec	config_Voice.json	BuiltinMic.read_chunk budget (default 0.04 s)
tts.backend	config_Voice.json	builtin_ttsmaker (only supported option)
stt.wake_words	config_Voice.json	33 fuzzy variants of "Sanad" for the wake-verify substring match
stt.command_vocab	config_Voice.json	68 canonical command phrases for fuzzy-normalization ("turn right up" → "turn right")
stt.garbage_patterns	config_Voice.json	17 Whisper noise-hallucinations to reject ("thanks for watching", "okay", etc.)
stt.speech_threshold etc.	config_Voice.json	energy wake detector thresholds — see Doc/controlling.md "Voice" for the full tuning matrix
stt.whisper_*	config_Voice.json	faster-whisper model, compute type, beam size, confidence gates, bias prompt
stt.mode	config_Voice.json	wake_and_command (default) / always_on / always_on_gated
timeout_ms, stale_threshold_s, reconnect_delay_s	config_Camera.json	RealSense frame timeout, reconnect trigger, initial backoff
default_max_steps, step_delay_s, rotate_speed, min_steps_warmup	config_ImageSearch.json	image-guided search rotation cadence (wired into Vision/marcus_imgsearch.py)
default_walk_speed, dist_tolerance, angle_tolerance, safety_timeout_mult, dr_update_hz	config_Odometry.json	precise motion control (wired into Navigation/marcus_odometry.py)
MARCUS_LOG_MAX_BYTES, MARCUS_LOG_BACKUP_COUNT, MARCUS_LOG_DIR	env vars	log rotation size, backup count, log directory override

---

Per-command latency (estimated, post-fixes)

Step	Typical	Notes
Wake-word detect	<100 ms	pure-numpy energy detector, 50 ms analysis windows
Wake verify (first wake)	~2000 ms	includes faster-whisper base.en cold load
Wake verify (subsequent)	300–700 ms	Whisper cached, decodes ~0.5-1.5 s burst
"Yes" TTS ack	~1500 ms	G1 firmware TtsMaker minimum
Record until silence	1–5 s	depends on user speech; max_record_sec=5 cap
Pre-silence trim	<1 ms	numpy slice
faster-whisper STT	500–1500 ms	base.en int8 on CPU, beam_size=5
Fuzzy-match canonicalisation	<1 ms	difflib against 68 phrases
Camera frame fetch	<50 ms	poll loop, no 1 s blocking stall
Ollama Qwen2.5-VL	800–1500 ms	num_batch=128 / num_ctx=2048 / num_predict=120
Executor + ZMQ send	<10 ms	fire-and-forget PUB
TtsMaker playback	~len(text) × 80 ms	synthesizes + plays on robot

Total wake → answer-playback: ~2.5–4 s for a short vision question like "what do you see" (vs. 5–8 s with the pre-restructure edge-tts/Gemini overhead).