Best for
- Manual run before/after a voice-pipeline change, or to spot-check responsiveness/clarity/accuracy.
- Bring-up of a new agent as the subject — only the summon test is agent-specific.
- (Daily auto-scheduling is a planned future improvement.)
Source profileQuality 71/100Review permissions
sonichi/sutando/skills/voice-agent-test-harness/SKILL.md
voice-agent-test-harness
Drive a fixed suite of spoken tests against a voice agent ("subject") from a co-located machine ("prober"), measure response latency / clarity / accuracy, diff against baseline, and report to the owner over Telegram.
- Source repository stars
- 359
- Declared platforms
- 0
- Static risk flags
- 1
- Last source update
- 2026-07-28
- Source checked
- 2026-07-28
Decision brief
What it does—and where it fits
Drive a fixed suite of spoken tests against a voice agent ("subject") from a co-located machine ("prober"), measure response latency / clarity / accuracy, diff against baseline, and report to the owner over Telegram.
Not for
- Tasks that require unconfirmed production actions or broad system permissions.
- Environments where the pinned source and install steps cannot be inspected.
Compatibility matrix
Platform support, with evidence labels
| Platform | Status | Evidence | What to check |
|---|---|---|---|
| Codex | Not declared | No explicit evidence | Portability before use |
| Claude Code | Not declared | No explicit evidence | Portability before use |
| Cursor | Not declared | No explicit evidence | Portability before use |
| Gemini CLI | Not declared | No explicit evidence | Portability before use |
Installation
Inspect first. Install second.
The source command is displayed only when detected. A safe inspection prompt is always available so your agent can explain every action before execution.
Source-detected install commandSource
npx skills add https://github.com/sonichi/sutando --skill "skills/voice-agent-test-harness"Safe inspection promptEditorial
Inspect the Agent Skill "voice-agent-test-harness" from https://github.com/sonichi/sutando/blob/6a8f0fccd32e5aa620a3572c8885544f144bb6fe/skills/voice-agent-test-harness/SKILL.md at commit 6a8f0fccd32e5aa620a3572c8885544f144bb6fe. List every install step, command, network request, credential, file read/write, external action, and rollback step. Explain whether it fits my task. Do not install or execute anything until I approve.
Workflow
What the source asks the agent to do
- 01
How to try it (two laptops, same room)
1. Subject: on laptop 2, start a normal Sutando voice session, mic open, speaker up. 2. Prober: on laptop 1 (this one), grant Terminal Microphone permission (System Settings → Privacy → Microphone), then:
Subject: on laptop 2, start a normal Sutando voice session, mic open, speaker up.Prober: on laptop 1 (this one), grant Terminal Microphone permission (System Settings → Privacy → Microphone), then:The prober speaks each prompt; the subject replies; the prober measures, transcribes, judges, and prints the roll-up. Add --deliver to send the report to your Telegram. - 02
What runs end-to-end today
So that a half-SKIPPED suite is never mistaken for "mostly fine," here is exactly what executes through the real acoustic path now versus what is stubbed or excluded. A captured live run is committed at examples/run-2026-06-06.json.
So that a half-SKIPPED suite is never mistaken for "mostly fine," here is exactly what executes through the real acoustic path now versus what is stubbed or excluded. A captured live run is committed at examples/run-202… - 03
Preconditions (same-room run)
Both laptops awake, unmuted, mics/speakers enabled, within normal speaking distance.
Both laptops awake, unmuted, mics/speakers enabled, within normal speaking distance.Subject (Sutando 2) in a normal voice session with mic open.The runner gates on mic calibration; if the mic path is dead/clipping it reports SKIPPED, not a fail. - 04
Action tests
Tests with an effect block (the timer) verify the real side effect: after the verbal confirmation, the prober waits the timer duration and listens for the alarm actually firing. Confirmation without an observed effect downgrades to partial.
Tests with an effect block (the timer) verify the real side effect: after the verbal confirmation, the prober waits the timer duration and listens for the alarm actually firing. Confirmation without an observed effect d… - 05
When to use
Manual run before/after a voice-pipeline change, or to spot-check responsiveness/clarity/accuracy.
Manual run before/after a voice-pipeline change, or to spot-check responsiveness/clarity/accuracy.Bring-up of a new agent as the subject — only the summon test is agent-specific.(Daily auto-scheduling is a planned future improvement.)
Permission review
Static risk signals and limitations
Runs scripts
The documentation asks the agent to run terminal commands or scripts.
python3 scripts/run_suite.py --quick # --quick shortens the 2-min timer wait to 30sRuns scripts
The documentation asks the agent to run terminal commands or scripts.
python3 scripts/run_suite.py --only arithmetic # one test by idEvidence record
Why each signal appears
EvidenceSourceComputedTestedEditorial
| Signal | Value | Evidence type | Meaning |
|---|---|---|---|
| Quality score | 71/100 | Computed | Documentation, specificity, maintenance, and trust rules |
| Repository stars | 359 | Source | Repository attention, not individual Skill quality |
| Compatibility | 0 platforms | Source | Declared in the catalog source record |
| Usage guide | automated source guide | Editorial | Generated or reviewed according to the visible evidence level |
Pinned source
Provenance and original SKILL.md
- Repository
- sonichi/sutando
- Skill path
- skills/voice-agent-test-harness/SKILL.md
- Commit
- 6a8f0fccd32e5aa620a3572c8885544f144bb6fe
- License
- MIT
- Collected
- 2026-07-28
- Default branch
- main
View the original SKILL.md
Voice-Agent Test Harness
Drive a fixed suite of spoken tests against a voice agent ("subject") from a co-located machine ("prober"), measure response latency / clarity / accuracy, diff against baseline, and report to the owner over Telegram.
Design: docs/voice-agent-test-framework.md
v1 (macOS). Real audio path: TTS via
gemini-tts+afplay, mic capture viasoxrec(CoreAudio), voice-onset via numpy RMS, STT + judge via Gemini (Sutando-standard,GEMINI_API_KEY). Manual trigger; reports to owner only. Each prober-side component is tested; the full closed loop needs the second laptop speaking.
What runs end-to-end today
So that a half-SKIPPED suite is never mistaken for "mostly fine," here is exactly what executes through the real acoustic path now versus what is stubbed or excluded. A captured live run is committed at examples/run-2026-06-06.json.
| Capability | Status today |
|---|---|
Single-answer suite (test_cases.yaml, core-v1) — speak → capture → onset → Gemini STT → judge → score | ✅ Wired. Every row runs end-to-end on real audio; pass / fail / partial / no_response are all measured outcomes, not stubs. |
| Latency / clarity / accuracy scoring + baseline diff + Telegram roll-up | ✅ Wired — computed on real captured turns. |
timer action test — real side-effect verify (waits, listens for the alarm) | ✅ Wired. |
Multi-turn workflow turns (workflow_cases.yaml, e.g. the developer code-change flow) | ⚠️ Partial. The spoken handling is captured and judged; remote side effects (branch/test/cleanup) are not observable from the prober, so these score wording only. |
| Gmail / CRM workflow turns | ⛔ Excluded — unfinished test setup; omitted from results, not reported as failures. |
| Daily auto-scheduling | ⛔ Not wired — manual trigger only. |
How to try it (two laptops, same room)
- Subject: on laptop 2, start a normal Sutando voice session, mic open, speaker up.
- Prober: on laptop 1 (this one), grant Terminal Microphone permission (System Settings → Privacy → Microphone), then:
cd ~/GitHub/sutando/skills/voice-agent-test-harness python3 scripts/run_suite.py --quick # --quick shortens the 2-min timer wait to 30s - The prober speaks each prompt; the subject replies; the prober measures, transcribes, judges, and prints the roll-up. Add
--deliverto send the report to your Telegram.
Useful flags:
python3 scripts/run_suite.py --only arithmetic # one test by id
python3 scripts/run_suite.py --dry-run # no audio/model; canned data (CI/sanity)
python3 scripts/baseline.py --promote results/voice-test/<date>.json # set regression baseline
Preconditions (same-room run)
- Both laptops awake, unmuted, mics/speakers enabled, within normal speaking distance.
- Subject (Sutando 2) in a normal voice session with mic open.
- The runner gates on mic calibration; if the mic path is dead/clipping it reports
SKIPPED, not a fail.
Action tests
Tests with an effect block (the timer) verify the real side effect: after the verbal confirmation, the prober waits the timer duration and listens for the alarm actually firing. Confirmation without an observed effect downgrades to partial.
When to use
- Manual run before/after a voice-pipeline change, or to spot-check responsiveness/clarity/accuracy.
- Bring-up of a new agent as the subject — only the
summontest is agent-specific. - (Daily auto-scheduling is a planned future improvement.)
Output
results/voice-test/<date>.json— per-test rows (latency, accuracy, clarity, transcript, effect) + suite roll-up (gitignored).- With
--deliver: a Telegram message to the owner — pass rate, p50/p95 latency, clarity, and any regressions vs baseline.
Alternatives
Compare before choosing
Computed 10042,015
coreyhaines31/marketingskills
ab-testing
When the user wants to plan, design, or implement an A/B test or experiment, or build a growth experimentation program. Also use when the user mentions "A/B test," "split test," "experiment," "test this change," "variant copy," "multivariate test," "hypothesis," "should I test this," "which version is better," "test two versions," "statistical significance," "how long should I run this test," "growth experiments," "experiment velocity," "experiment backlog," "ICE score," "experimentation program
Computed 997
event4u-app/agent-config
design-review
Use when the user says "review the design", "check the UI", or wants a comprehensive UI/UX review. Uses a 7-phase methodology covering interaction, responsiveness, accessibility, and more.
Computed 967
event4u-app/agent-config
blast-radius-analyzer
Use BEFORE editing shared code — enumerates every call site, event consumer, queue worker, API client, migration, and test that a planned change will touch, with a file:line citation per dependency.
Computed 967
event4u-app/agent-config
bug-analyzer
Use when the user shares a Sentry error, Jira bug ticket, or error description and wants root cause analysis. Also for proactive bug hunting and code audits for hidden bugs.