01Safety
0.0 / 30
What changed in the harness
Selection accuracy 89→80, token cost up 2%, unconfirmed writes 0%→0%.
Category breakdown
Where the score comes from.
Earned points across the four signals Gradable measures. Safety and Legibility are scored out of 30; Economics and Discoverability are scored out of 20.
02Legibility
25.7 / 30
03Economics
18.2 / 20
04Discoverability
11.9 / 20
Highest-impact fix
Estimated gain +30 pointsAdd explicit identity and permission preflight tools
Expose machine-readable principal/tenant confirmation and a non-mutating permission check so agents can verify both before destructive actions.
Description evidence
Defects and rewrites.
4 defects found across the exposed tool descriptions. Suggested rewrites make purpose, inputs, boundaries, and returns easier for an agent to understand.
| Tool | Defect types | Suggested rewrite |
|---|---|---|
check_config |
no_return_description |
Verify that whisper-cli.exe, the active Whisper model, and FFmpeg are all available on this system. Run this first if anything fails. Returns the check result for each component, including which are present and which are missing or misconfigured. |
download_model |
no_return_description |
Download a Whisper model from Hugging Face directly into your models directory. Accepts a model name (e.g. large-v3-turbo, medium.en-q5_0) and handles the download automatically. Downloads only from trusted Hugging Face namespaces (ggerganov/whisper.cpp and ggml-org). After downloading, use switch_model to activate it for the current session. Returns the downloaded filename, or an error if the model is unavailable or the download fails. |
switch_model |
no_return_description |
Switch the active Whisper model for the current session without restarting Claude Desktop. Accepts a model filename (e.g. ggml-large-v3-turbo.bin) or a full path; the model must already be installed in your models directory. The change is session-scoped and does not persist after Claude Desktop restarts. Returns the newly active model name, or an error if the model is not installed. |
whisper_server |
no_return_description |
Start, stop, or check the persistent whisper model server. When running, the active model stays resident in VRAM and every transcribe_audio / transcribe_batch call is served over localhost without reloading it, eliminating the per-file model-load cost. The resident model holds GPU VRAM for the server's entire lifetime, so start it deliberately, do your work, then stop it to hand the GPU back to other applications. While running, background jobs, start_batch, generate_subtitles, and lrc/csv or advanced per-call options are refused. Bound to localhost only. Returns a status report: for start, confirmation that the model is resident; for stop, confirmation that the server shut down and VRAM is freed; for status, whether it is running, the resident model, port, and uptime. |
Selection evidence
Confusable tool pairs.
6 pairs where similar names or overlapping descriptions may send an agent toward the wrong tool.
| Tool A | Tool B | Confidence | Why they collide |
|---|---|---|---|
check_progress |
check_batch_progress |
medium | Both names literally say 'check ... progress' and differ only by job vs batch (job_id vs batch_id). A task like 'check the progress of my transcription' is ambiguous unless the user explicitly mentions that start_batch was used. |
transcribe_batch |
start_batch |
medium | Both transcribe all files in a folder; the only distinction is interactive per-file confirmation vs automated background batch. A task like 'transcribe all files in this folder' matches both tools, and only wording such as 'unattended' or 'without prompting' disambiguates. |
download_model |
switch_model |
medium | A request like 'get me model X' or 'use a different model' is ambiguous between downloading from Hugging Face and activating an installed model. The documented download-then-switch workflow is easy to get wrong on the first call. |
transcribe_audio |
transcribe_batch |
medium | Both run whisper transcription, one taking a single file_path and the other a folder_path. A task like 'transcribe all my recordings in Downloads' could lead the agent to the single-file tool with a folder path or the interactive batch tool. |
transcribe_audio |
generate_subtitles |
low | Both tools accept the same audio/video files and transcribe_audio can emit srt/vtt output, so a task like 'create subtitles for this video' or 'give me the srt transcript' could plausibly map to either tool. |
check_config |
check_system |
low | Both are zero-argument diagnostics prefixed 'check_' with overlapping ideas of verifying availability. A generic task like 'is everything set up on my system?' could plausibly pick either, though check_config targets toolchain availability while check_system targets GPU/Vulkan. |
Compare the field