335 lines
14 KiB
Plaintext
335 lines
14 KiB
Plaintext
G1 Audio Playback & Recording — Working Configuration
|
|
======================================================
|
|
Date: 2026-04-07 (updated 2026-04-23)
|
|
|
|
UPDATE 2026-04-23
|
|
------------------
|
|
Three scripts previously used the broken chunked-PlayStream path. Fixed:
|
|
|
|
g1_voice_recorder.py — replay path now sends ALL audio in ONE
|
|
_CallRequestWithParamAndBin call with a unique
|
|
stream_id. No more PlayStream.
|
|
g1_play_wav.py — same fix; removed TtsMaker warm-up (not needed).
|
|
g1_interactive_player.py — removed chunked loop (same stream_id across
|
|
chunks was silently dropping after the first).
|
|
|
|
Added three new scripts:
|
|
|
|
g1_audio_devices.py — pactl wrapper: list sources/sinks, find by
|
|
keyword, auto-detect Hollyland mic, unmute,
|
|
set volume, parec quick-test.
|
|
g1_edge_tts.py — edge-tts bilingual (AR+EN) TTS pipeline —
|
|
text → MP3 → 16 kHz WAV → G1 speaker.
|
|
Voices default to ar-AE-HamdanNeural / en-US-GuyNeural.
|
|
template_audio_script.py — copy-paste starter demonstrating the
|
|
reliable primitives (prep mic, record via
|
|
parec, play in ONE call, always STOP).
|
|
|
|
Quick reference — the ONE audio play path that works on G1:
|
|
from unitree_sdk2py.g1.audio.g1_audio_api import (
|
|
ROBOT_API_ID_AUDIO_START_PLAY, ROBOT_API_ID_AUDIO_STOP_PLAY)
|
|
c._Call(ROBOT_API_ID_AUDIO_STOP_PLAY, json.dumps({"app_name": APP}))
|
|
time.sleep(0.3)
|
|
sid = f"s_{int(time.time()*1000)}"
|
|
param = json.dumps({"app_name": APP, "stream_id": sid,
|
|
"sample_rate": 16000, "channels": 1, "bits_per_sample": 16})
|
|
c._CallRequestWithParamAndBin(ROBOT_API_ID_AUDIO_START_PLAY, param, list(pcm))
|
|
time.sleep(duration + 0.5)
|
|
c._Call(ROBOT_API_ID_AUDIO_STOP_PLAY, json.dumps({"app_name": APP}))
|
|
|
|
------------------
|
|
|
|
|
|
|
|
BUILT-IN SPEAKER PLAYBACK (via Unitree Voice Service)
|
|
-----------------------------------------------------
|
|
Method: _CallRequestWithParamAndBin (NOT PlayStream)
|
|
Key discovery: JSON params MUST include sample_rate, channels, bits_per_sample
|
|
Required: Send ALL audio in ONE call (streaming chunks does NOT work)
|
|
Required: Unique stream_id per play (timestamp-based), or audio won't repeat
|
|
Required: Call AUDIO_STOP_PLAY before each new play to reset stream
|
|
|
|
Working code pattern:
|
|
from unitree_sdk2py.g1.audio.g1_audio_api import *
|
|
|
|
c._Call(ROBOT_API_ID_AUDIO_STOP_PLAY, json.dumps({"app_name": "p"}))
|
|
time.sleep(0.3)
|
|
|
|
sid = f"s_{int(time.time() * 1000)}"
|
|
param = json.dumps({
|
|
"app_name": "p",
|
|
"stream_id": sid, # MUST be unique each time
|
|
"sample_rate": 16000, # MUST include these format params
|
|
"channels": 1,
|
|
"bits_per_sample": 16,
|
|
})
|
|
c._CallRequestWithParamAndBin(ROBOT_API_ID_AUDIO_START_PLAY, param, list(pcm))
|
|
time.sleep(duration + 1)
|
|
c._Call(ROBOT_API_ID_AUDIO_STOP_PLAY, json.dumps({"app_name": "p"}))
|
|
|
|
Audio format: 16kHz, mono, 16-bit signed little-endian PCM
|
|
SDK required: Robot's gemini conda env (has _CallRequestWithParamAndBin)
|
|
Workstation SDK (g1_env): Does NOT have _CallRequestWithParamAndBin — must run on robot
|
|
|
|
What does NOT work:
|
|
- PlayStream() method (sends data but no audio output)
|
|
- Streaming chunks in a loop (only first call sometimes plays)
|
|
- _CallBinary / _Call with PCM data (error code 100 from workstation SDK)
|
|
- Running playback from .py script files (only python3 -c inline works reliably)
|
|
- aplay / paplay through PulseAudio (Jetson platform-sound has no physical speaker)
|
|
|
|
TTS (Text-to-Speech):
|
|
- English: works
|
|
- Chinese: works
|
|
- Arabic: does NOT work (falls back to Chinese)
|
|
- speaker_id: 0, 1, 2 all work (different voices)
|
|
|
|
API IDs:
|
|
ROBOT_API_ID_AUDIO_TTS = 1001
|
|
ROBOT_API_ID_AUDIO_ASR = 1002 (returns 3104 — not available)
|
|
ROBOT_API_ID_AUDIO_START_PLAY = 1003
|
|
ROBOT_API_ID_AUDIO_STOP_PLAY = 1004
|
|
ROBOT_API_ID_AUDIO_GET_VOLUME = 1005
|
|
ROBOT_API_ID_AUDIO_SET_VOLUME = 1006
|
|
ROBOT_API_ID_AUDIO_SET_RGB_LED = 1010
|
|
APIs 1007-1009, 1011-1012: not registered (code 3103)
|
|
|
|
WIRELESS MICROPHONE RECORDING
|
|
------------------------------
|
|
Device: Hollyland Wireless Microphone (USB)
|
|
Vendor: Shenzhen Hollyland Technology Co.,Ltd
|
|
Product ID: 0007
|
|
ALSA: card 2, device 0 (hw:2,0)
|
|
PulseAudio source index: 3
|
|
Source name: alsa_input.usb-Shenzhen_Hollyland_Technology_Co._Ltd_Wireless_microphone_C63X223T6MX-01.analog-stereo
|
|
Format: s24le, 2ch, 48000Hz
|
|
|
|
CRITICAL: Mic was MUTED by default. Must unmute before recording.
|
|
|
|
Step-by-step mic setup commands (run on robot):
|
|
# 1. View all PulseAudio sources (find mic index)
|
|
pactl list sources short
|
|
|
|
# 2. Check mic details (mute status, volume, format)
|
|
pactl list sources | grep -A 10 "Wireless"
|
|
|
|
# 3. Set wireless mic as default input source
|
|
pactl set-default-source alsa_input.usb-Shenzhen_Hollyland_Technology_Co._Ltd_Wireless_microphone_C63X223T6MX-01.analog-stereo
|
|
|
|
# 4. Unmute the mic (index 3)
|
|
pactl set-source-mute 3 0
|
|
|
|
# 5. Set mic volume to 100%
|
|
pactl set-source-volume 3 100%
|
|
|
|
# 6. Verify mic is capturing audio
|
|
timeout 2 parec -d 3 --format=s16le --rate=16000 --channels=1 --raw > /tmp/raw_test.pcm
|
|
python3 -c "
|
|
import numpy as np
|
|
a = np.fromfile('/tmp/raw_test.pcm', dtype=np.int16)
|
|
print(f'Samples={len(a)}, min={a.min()}, max={a.max()}, std={a.std():.0f}')
|
|
if a.std() > 50: print('MIC WORKS!')
|
|
else: print('Still silent - check transmitter is ON')
|
|
"
|
|
|
|
Recording method (parec, not PyAudio — PyAudio gives silence through pulse device):
|
|
subprocess.Popen(['parec', '-d', '3', '--format=s16le', '--rate=16000', '--channels=1', '--raw'], stdout=subprocess.PIPE)
|
|
time.sleep(duration)
|
|
proc.terminate()
|
|
raw = proc.stdout.read()
|
|
|
|
PyAudio recording through device index 25 (pulse) gives ALL ZEROS even after unmuting.
|
|
Must use parec with source index 3 directly.
|
|
|
|
FULL RECORD + PLAYBACK EXAMPLE (run on robot in gemini conda env):
|
|
python3 -c "
|
|
import time, wave, json, numpy as np, subprocess
|
|
from unitree_sdk2py.core.channel import ChannelFactoryInitialize
|
|
from unitree_sdk2py.g1.audio.g1_audio_client import AudioClient
|
|
from unitree_sdk2py.g1.audio.g1_audio_api import *
|
|
|
|
# Record 5 seconds via parec (wireless mic, source index 3)
|
|
print('Recording 5 seconds... speak now!')
|
|
proc = subprocess.Popen(
|
|
['parec', '-d', '3', '--format=s16le', '--rate=16000', '--channels=1', '--raw'],
|
|
stdout=subprocess.PIPE)
|
|
time.sleep(5)
|
|
proc.terminate()
|
|
raw = proc.stdout.read()
|
|
audio = np.frombuffer(raw, dtype=np.int16)
|
|
print(f'Recorded {len(audio)} samples, std={audio.std():.0f}')
|
|
|
|
# Save to WAV
|
|
path = '/home/unitree/SanadVoice/recorded voices/my_recording.wav'
|
|
wf = wave.open(path, 'wb')
|
|
wf.setnchannels(1)
|
|
wf.setsampwidth(2)
|
|
wf.setframerate(16000)
|
|
wf.writeframes(audio.tobytes())
|
|
wf.close()
|
|
print(f'Saved: {path}')
|
|
|
|
# Play back on G1 built-in speaker
|
|
ChannelFactoryInitialize(0, 'eth0')
|
|
c = AudioClient()
|
|
c.SetTimeout(10.0)
|
|
c.Init()
|
|
c.SetVolume(100)
|
|
c._Call(ROBOT_API_ID_AUDIO_STOP_PLAY, json.dumps({'app_name':'p'}))
|
|
time.sleep(0.3)
|
|
|
|
pcm = audio.tobytes()
|
|
sid = f's_{int(time.time()*1000)}'
|
|
param = json.dumps({
|
|
'app_name': 'p',
|
|
'stream_id': sid,
|
|
'sample_rate': 16000,
|
|
'channels': 1,
|
|
'bits_per_sample': 16
|
|
})
|
|
c._CallRequestWithParamAndBin(ROBOT_API_ID_AUDIO_START_PLAY, param, list(pcm))
|
|
time.sleep(len(audio)/16000 + 1)
|
|
c._Call(ROBOT_API_ID_AUDIO_STOP_PLAY, json.dumps({'app_name':'p'}))
|
|
print('Done')
|
|
"
|
|
|
|
NETWORK & SSH
|
|
--------------
|
|
Robot IP: 192.168.123.164
|
|
SSH user: unitree
|
|
SSH pass: 123
|
|
SSH key: configured (ssh-copy-id done)
|
|
DDS interface on robot: eth0
|
|
DDS interface on workstation: enp3s0
|
|
|
|
Robot conda envs:
|
|
- base: no unitree_sdk2py
|
|
- gemini: has unitree_sdk2py with PlayStream + _CallRequestWithParamAndBin
|
|
Python: /home/unitree/miniconda3/envs/gemini/bin/python3
|
|
|
|
RUNNING SERVICES ON ROBOT
|
|
---------------------------
|
|
PulseAudio: PID 1752 (holds hw:1,0 speaker and hw:2,0 mic)
|
|
sanad_webserver.py: PID 1815 (gemini_voice_v2)
|
|
sanad_voice.py: PID ~4640+ (gemini_voice)
|
|
Systemd: sanad_voice.service (auto-start)
|
|
|
|
PulseAudio sink (speaker): alsa_output.platform-sound.analog-stereo (card 1 APE device 0)
|
|
- Volume: 100%, not muted
|
|
- Does NOT produce audible output (no physical speaker on this path)
|
|
- Built-in speaker is only accessible through Unitree voice RPC service
|
|
|
|
SanadVoice project speaker: Anker PowerConf USB (external, not always connected)
|
|
Sink: alsa_output.usb-Anker_PowerConf_A3321-DEV-SN1-01.analog-stereo
|
|
Source: alsa_input.usb-Anker_PowerConf_A3321-DEV-SN1-01.mono-fallback
|
|
|
|
RECORDED VOICES LOCATION
|
|
--------------------------
|
|
Robot: /home/unitree/SanadVoice/recorded voices/ (54 WAV files, English)
|
|
Robot: /home/unitree/SanadVoice/Audio/ (additional audio)
|
|
Workstation: ~/Robotics_workspace/yslootahtech/G1_Lootah/Audio_Recorder/DataG1/
|
|
|
|
MARCUS TTS CONFIGURATION (2026-04-08)
|
|
======================================
|
|
Backend: edge-tts (online, Microsoft Edge TTS API, no API key needed)
|
|
Requires: Internet connection on robot
|
|
|
|
Voices:
|
|
Arabic: ar-AE-HamdanNeural (UAE male)
|
|
English: en-US-GuyNeural (US male)
|
|
Note: For same voice both languages, use ar-AE-HamdanNeural for both
|
|
(English will have slight Arabic accent)
|
|
|
|
Config: /home/unitree/Marcus/Config/config_Voice.json
|
|
"en_backend": "edge_tts"
|
|
"ar_backend": "edge_tts"
|
|
"edge_voice_ar": "ar-AE-HamdanNeural"
|
|
"edge_voice_en": "en-US-GuyNeural"
|
|
|
|
Pipeline:
|
|
audio_api.speak(text, lang)
|
|
→ edge-tts generates MP3 (1-2s)
|
|
→ pydub converts MP3 → 16kHz WAV
|
|
→ _CallRequestWithParamAndBin → G1 speaker
|
|
→ mic muted during playback, unmuted after
|
|
|
|
Fallback: Built-in TtsMaker for English if edge-tts fails (no internet)
|
|
|
|
Mic: Auto-detected by scanning PulseAudio for "wireless"/"hollyland"/"usb"
|
|
Auto-unmutes, sets volume 100%, sets as default source
|
|
Uses parec (not PyAudio) for recording
|
|
Muted during TTS playback to prevent self-listening
|
|
|
|
Files updated:
|
|
/home/unitree/Marcus/API/audio_api.py — XTTS + edge-tts + builtin backends
|
|
/home/unitree/Marcus/Voice/marcus_voice.py — Uses audio_api's auto-detected mic
|
|
/home/unitree/Marcus/Voice/xtts_server.py — XTTS-v2 persistent server (for offline use later)
|
|
/home/unitree/Marcus/Config/config_Voice.json — TTS config
|
|
|
|
Conda envs:
|
|
marcus (Python 3.8) — Brain, vision, controller, audio playback SDK
|
|
marcus_tts (Python 3.10) — XTTS-v2, SILMA TTS (for future offline use)
|
|
gemini (Python 3.11) — Has _CallRequestWithParamAndBin
|
|
|
|
TTS OPTIONS TESTED
|
|
-------------------
|
|
| Backend | Arabic | English | Offline | Speed | Status |
|
|
|------------------|--------|---------|---------|--------|---------------|
|
|
| edge-tts | Yes | Yes | No | 1-2s | WORKING ✓ |
|
|
| Built-in TtsMaker| No | Yes | Yes | 0s | WORKING ✓ |
|
|
| XTTS-v2 (CPU) | Yes | Yes | Yes | ~14s | WORKING (slow)|
|
|
| XTTS-v2 (GPU) | Yes | Yes | Yes | ~2s | NO (CUDA 11.4 no cp310 wheel) |
|
|
| SILMA TTS | Yes | Yes | Yes | ? | BROKEN (deps) |
|
|
| Piper TTS | Yes | No | Yes | ~1s | BROKEN (piper_phonemize on ARM64) |
|
|
| gTTS | Yes | Yes | No | 1-2s | WORKING (female only) |
|
|
| SpeechT5 local | Yes | No | Yes | ~2-3s | NOT TESTED (model on robot) |
|
|
|
|
JETSON ORIN NX LIMITATIONS
|
|
---------------------------
|
|
- JetPack 5.1.1, L4T R35.3.1, CUDA 11.4
|
|
- NVIDIA only provides PyTorch cp38 wheels (Python 3.8)
|
|
- No cp310 GPU PyTorch available — must build from source
|
|
- scikit-learn TLS bug on aarch64 (libgomp cannot allocate memory in static TLS block)
|
|
Fix: LD_PRELOAD=$(find env/lib -name "libgomp*.so*" | head -1)
|
|
- apt is broken (unmet dependencies), cannot install espeak-ng
|
|
|
|
SCRIPTS DEPLOYED ON ROBOT
|
|
---------------------------
|
|
/home/unitree/g1_interactive_player.py — Interactive WAV selector + playback
|
|
/home/unitree/g1_play_wav.py — CLI WAV player
|
|
/home/unitree/g1_tts_test.py — TTS language test
|
|
/home/unitree/Marcus/Voice/xtts_server.py — XTTS-v2 persistent server
|
|
|
|
FILE INVENTORY (workstation /G1_Lootah/Audio_Recorder)
|
|
-------------------------------------------------------
|
|
Recording + replay:
|
|
g1_voice_recorder.py — workstation PyAudio record → scp → G1 replay
|
|
g1_play_wav.py — (on-robot) play any WAV on G1 speaker
|
|
g1_interactive_player.py — (on-robot) interactive WAV picker
|
|
g1_voice_deploy.py — deploy recordings to robot
|
|
Device management:
|
|
g1_audio_devices.py — pactl wrapper: list/find/auto-hollyland/quick-test
|
|
TTS:
|
|
g1_edge_tts.py — Microsoft Edge TTS bilingual (AR+EN)
|
|
g1_tts_arabic.py — Piper / tts_arabic offline TTS (ARM64 broken per above)
|
|
g1_tts_test.py — TTS smoke test (English only)
|
|
Templates + tests:
|
|
template_audio_script.py — copy-paste starter with the reliable primitives
|
|
test_g1_speaker.py — speaker init sanity check
|
|
test_g1_pcm.py — raw PCM playback test
|
|
test_marcus_audio.py — Marcus env audio pipeline end-to-end
|
|
Data:
|
|
DataG1/ — local recordings (WAV, 16 kHz mono int16)
|
|
|
|
Run recipes:
|
|
# Workstation side
|
|
python3 g1_voice_recorder.py record --name hello --seconds 5
|
|
python3 g1_voice_recorder.py replay --name hello --ip 192.168.123.164
|
|
python3 g1_edge_tts.py "Good morning" --lang en --save out.wav --no-play
|
|
|
|
# On robot (gemini env)
|
|
python3 g1_audio_devices.py auto-hollyland
|
|
python3 g1_play_wav.py /path/to/file.wav
|
|
python3 g1_edge_tts.py "مرحبا بكم" --lang ar
|
|
python3 template_audio_script.py echo --seconds 3
|