AI_Photographer/Current_runtime.md

# Current Runtime

Production runtime architecture (as implemented now):

`photo_sanad.sh -> voice_sanad.py -> persistent component loops + WS supervisor -> mode-gated command handling -> unified replay/capture -> server/dashboard/upload`

## Process Model

- `Scripts/photo_sanad.sh` resets `mode.current_mode` to `mode.default_mode` on each launch.
- Default full runtime path:
- startup mode is normally `manual`
- `photo_sanad.sh` resolves active PulseAudio sink/source, launches `Core/direct_camera_service.py` in the `teleimager` env, then starts `Gemini/voice_sanad.py` in the gemini env
- `Core/direct_camera_service.py` is backend-first and serves external UI assets from `Web/direct_camera.html`, `Web/direct_camera.css`, and `Web/direct_camera.js`
- preferred RealSense default is read from `Data/Settings/config.json -> camera.preferred_realsense_serial` and falls back to another detected camera if absent
- in the default full runtime path, `Gemini/voice_sanad.py`, the dashboard server, the direct camera server, and replay/trigger services remain running in both `manual` and `ai`
- in the full runtime path, `AUTONOMOUS_ENABLE` is auto-armed by default so dashboard mode can switch from `manual` to `ai` without restart
- Optional lean manual path:
- `MANUAL_LEAN_RUNTIME=1`
- skips the direct camera server and the heavy manual/AI runtime services
- keeps Gemini + dashboard only
- `Gemini/voice_sanad.py` is the single long-lived orchestrator process.
- Runtime logs are centralized under `Logs/`, with one stable file per component.
- It starts these loops once and keeps them alive:
- `capture_mic`
- `receive_audio`
- `play_audio`
- `keepalive`
- `Modes/Manual/trigger_loop.py` in the full runtime path
- autonomous-mode supervisor in the full runtime path
- `Modes/AI/autonomous_manager.py` only while `AUTONOMOUS_ENABLE=1` and runtime mode is `ai`
- Runtime health writer (`Data/Runtime/runtime_health.json`)
- Mode policy sync
- Gemini WebSocket is managed by a dedicated reconnect supervisor. WS reconnect does not restart other loops.
- In the normal full runtime path, mode switches change gating/state only; they do not tear down `voice_sanad.py`, the dashboard server, the direct camera server, or replay services.

## Mode Control

- Source of truth is `Data/Settings/config.json`:
- `mode.current_mode` in `manual|ai`
- launcher resets `mode.current_mode` to `mode.default_mode` at process start
- API updates mode live via `Server/photo_server.py`:
- `/api/set_mode`
- Voice command gating is enforced on user transcription events in `Gemini/gemini_voice.py`.

## Mode Semantics

| Mode | Voice photo commands (`request_photo/yes_photo/no_photo`) | Manual R2+X | Autonomous flow |
|---|---|---|---|
| `manual` | Off | On | Paused |
| `ai` | On | On | On only if `AUTONOMOUS_ENABLE=1` |

Additional mode rules:
- Gemini conversation can stay active in both modes when `gemini.mic_enabled=true`.
- In full runtime, `manual` still includes the direct camera server, replay/trigger, uploader, and dashboard capture services.
- In full runtime, switching dashboard mode from `manual` to `ai` does not restart the process.
- In full runtime, switching between `manual` and `ai` does not stop `voice_sanad.py`, the dashboard server, the direct camera server, or replay/trigger services.
- If `AUTONOMOUS_ENABLE=1`, autonomous manager is armed and starts live when runtime mode becomes `ai`.
- Switching back to `manual` pauses autonomous flow again.
- In optional `MANUAL_LEAN_RUNTIME=1`, capture/replay/autonomous services are intentionally unavailable.

Removed from this project:
- command-mode functionality was extracted to `G1_Lootah/AI_Command`

## Remote Safety Controls

- `R2+X`: starts replay + photographer talk + unified capture pipeline.
- `R2+L1`: global hard cancel safety combo (active in runtime loops):
- cancels pending capture
- cancels active replay path
- resets autonomous interaction session to `IDLE`

## AI/Autonomous Runtime

State machine in `Modes/AI/autonomous_manager.py`:

`IDLE -> WAIT_CONFIRM -> FRAMING -> COUNTDOWN -> RETAKE_CONFIRM (optional) -> COMPLETE -> IDLE`

Special blocked state:

`IDLE_BLOCKED` when strict YOLO readiness fails.

Behavior:

- Autonomous manager is supervised by runtime mode:
- `manual` -> paused
- `ai` -> active when `AUTONOMOUS_ENABLE=1`
- In full runtime, autonomous services can already be armed while still paused in `manual`, so mode switches are live.
- In full runtime, the direct camera server and replay infrastructure remain started while autonomous manager is paused in `manual`.
- On stable intent, manager opens audio gate, triggers a short greeting-hand replay, and asks whether the visitor wants a photo.
- On stable single-person intent, manager can identify a returning guest or enroll a new guest into `photos/people/`.
- Group-first greeting is used when group is detected.
- Confirmation uses flag commands from voice layer (`request_photo.flag`, `confirm_yes.flag`, `confirm_no.flag`).
- Hard target lock can pin one subject/group through the session.
- Framing checks: center, size, blur, exposure, headroom, eye-line.
- AI greeting replay is controlled by `vision.autonomous_greeting_replay_enabled` and `vision.autonomous_greeting_replay_file`.
- AI photo-time replay is controlled by `vision.autonomous_capture_replay_enabled`.
- When AI photo-time replay is enabled, autonomous capture uses the active replay file from `Data/Settings/config.json -> replay.active_file`, same as manual `R2+X`.
- When a capture succeeds for an identified guest, the saved photo is attached into that guest's folder in `photos/people/`.
- After capture, retake recommendation can move flow to `RETAKE_CONFIRM` (max retakes from config).
- On completion: CTA prompt and cooldown reset.

## Vision Runtime

`Modes/AI/vision_detector.py` provides:

- Backend selection `normal|yolo`.
- YOLO runtime selection `ultralytics|opencv`.
- Person/face detection and group clustering.
- Intent detection using depth-first logic and bbox-area fallback.
- Target lock fields:
- `target_lock_active`, `target_lock_type`, `target_lock_id`, `target_switch_blocked_count`
- Camera/depth health fields:
- `camera_ok`, `depth_ok`, `camera_restarts`, `depth_restarts`

Strict production gate:

- If `vision.yolo_strict_required=true` and YOLO readiness is not valid, AI session is blocked in `IDLE_BLOCKED`.

## Gemini Integration

`Gemini/gemini_voice.py`:

- Uses WS attach/detach model:
- `attach_ws()`
- `detach_ws()`
- `is_ws_connected()`
- Live-safe sends:
- `send_text_prompt_live()`
- `send_vision_context_live()`
- Command matching uses user transcription events, not model text.
- Continuous vision context is streamed from autonomous manager.
- Context can be silent (`vision.gemini_context_silent=true`) and model audio is suppressed for context-only turns.
- Exposes runtime health snapshot for dashboard/API.
- In `manual`, Gemini conversation can remain available while AI photo flags stay disabled.
- Mic state is controlled live through `/api/mic` and `/api/set_mic`.

## Unified Capture Pipeline

All capture paths use `Server/capture_service.py`:

- Replay execution + trigger marker callback capture.
- Timed fallback capture if trigger marker is missing.
- Capture retries using watchdog settings:
- `watchdog.camera_capture_retry_count`
- `watchdog.camera_capture_retry_delay_sec`
- Upload trigger flag is touched after successful capture.

Replay integrity is validated at startup and fallback replay can be selected automatically.

## Component Recovery (Watchdog)

- WS failure: reconnect WS channel only.
- Mic failure: restart mic component only.
- Speaker failure: restart speaker component only.
- Detector frame starvation: recover detector camera/depth inputs only.
- Capture camera failure: retry capture call only.
- Process stays alive unless startup fatal occurs (for example empty Gemini API key).

## Server and Dashboard

`Server/photo_server.py` + `Web/gallery.js` provide:

- Mode APIs:
- `/api/mode`, `/api/set_mode`, `/api/mode_policy`
- Mic APIs:
- `/api/mic`, `/api/set_mic`
- Detector/AI readiness APIs:
- `/api/detector_backend`, `/api/set_detector_backend`
- `/api/ai_readiness`
- AI options APIs:
- `/api/ai_options`
- `/api/set_ai_options?hard_target_lock_enabled=0|1&retake_prompt_enabled=0|1&autonomous_greeting_replay_enabled=0|1&autonomous_greeting_replay_file=...&autonomous_capture_replay_enabled=0|1&face_recognition_enabled=0|1&face_recognition_threshold=...`
- Replay APIs:
- `/api/replays`
- `/api/get_replay`
- `/api/set_replay?name=...`
- `/api/delete_replay?name=...`
- `/api/rename_replay?old=...&new=...`
- `/api/download_replay?name=...`
- `/api/replay_record_status`
- `/api/replay_record_start?name=...&seconds=...`
- `/api/replay_test_status`
- `/api/test_replay?name=...`
- `/api/upload_replay`
- Runtime state APIs:
- `/api/autonomous_state`
- `/api/runtime_health`
- Camera APIs:
- `/api/camera_health`
- `/api/camera_sources`
- `/api/set_camera_source?source=...`
- `/api/set_camera_resolution?width=...&height=...&fps=...`
- `/api/set_preferred_camera?serial=...`
- Photo APIs:
- `/api/capture`, `/api/photos`, `/api/delete`, `/api/reupload`, `/api/upload_now`, `/api/download_zip`
- Live preview:
- `/preview.mjpg`
- preview is off by default and only runs when requested from the dashboard
- preview camera/OpenCV is loaded lazily when preview is requested

Dashboard panels include mode controls, detector backend/readiness, AI options, autonomous state, runtime health, live camera preview, camera source switching, camera resolution changes, preferred RealSense serial persistence, active replay selection, replay inventory management, and replay recording controls.

Replay-management rules:
- replay inventory covers the full `Data/G1` tree
- replay recording is allowed only in `manual`
- replay test/play is allowed only in `manual`
- rename/download/delete/upload remain available from the dashboard inventory tools
- People APIs:
- `/api/people`
- `/api/person_image?id=...&kind=face|scene`
- `/api/download_person?id=...`
- `/api/delete_person?id=...`
- `/api/reset_people`
- `/api/upload_person`
- Audio prompt APIs:
- `/api/audio_prompts`
- `/api/set_audio_prompt_mode?mode=audio|gemini`
- `/api/set_audio_prompt_fallback?enabled=0|1`
- `/api/audio_prompt_record_status`
- `/api/download_audio_prompt?key=...`
- `/api/delete_audio_prompt?key=...`
- `/api/upload_audio_prompt`
- `/api/audio_prompt_record`

Dashboard audio-prompt behavior:
- operators can upload prerecorded WAV clips for each AI situation key
- operators can delete or download existing clips
- operators can record a prompt clip directly from text using the same Gemini replay path as `Project/SanadVoice/gemini_voice/sanad_replay.py`
- operators can switch fixed AI situation speech between:
  - `audio`: recorded prompt clips first
  - `gemini`: Gemini speech for those same fixed situations
- if a prompt clip is missing while `audio_prompts.mode=audio`, runtime falls back to Gemini text when `audio_prompts.fallback_to_gemini=true`

Dashboard people-registry behavior:
- sidebar shows enrolled guests with face + scene thumbnails
- operators can upload a new face image to create or extend a guest profile
- operators can attach additional photos to an existing guest profile
- operators can download or delete one guest, or reset the whole registry

`Core/direct_camera_service.py` serves its own camera UI from external web assets under `Web/` rather than embedding HTML/CSS/JS in Python.

## Runtime State Files

- `Data/Settings/config.json`
- `Data/Runtime/autonomous_state.json`
- `Data/Runtime/runtime_health.json`
- `Data/Runtime/error_counters.json`
- `Data/Runtime/error_events.jsonl`
- `Data/Runtime/upload_db.json`
- `Data/Audio/`
- `Data/Settings/audio_prompt_records.json`
- `photos/people/`
- `photos/Captures/`
- `photos/samples/`

These runtime JSON files are generated lazily. In a clean project tree, some of them will not exist until the corresponding component starts writing state.

## Core Config Blocks

- `mode`: runtime mode.
- `vision`: backend/runtime, strict YOLO, context stream settings, hard lock, retake, framing thresholds, greeting replay, and AI capture replay.
- `vision`: backend/runtime, strict YOLO, context stream settings, hard lock, retake, framing thresholds, greeting replay, AI capture replay, and face-recognition controls.
- `watchdog`: WS backoff, component restart delay, capture retry policy.

## Notes

- Capture choreography is unified across manual trigger, autonomous flow, and dashboard capture.
- Hands/replay behavior during capture remains driven by replay files in `Data/G1`.
- Replay recordings created from the dashboard are stored directly under `Data/G1` and become selectable as active replays without restart.
- Imported AI prompt recordings are stored under `Data/Audio/` and indexed by `Data/Settings/audio_prompt_records.json`.
- In `audio` prompt mode, AI detection, greeting, confirmation, countdown, refusal, retake, and thank-you situations use recorded clips first.
- After a fixed prompt finishes, runtime returns to normal Gemini conversation flow automatically.