01Safety
0.0 / 30
What changed in the harness
Selection accuracy 95→93, token cost up 8%, unconfirmed writes 0%→0%.
Category breakdown
Where the score comes from.
Earned points across the four signals Gradable measures. Safety and Legibility are scored out of 30; Economics and Discoverability are scored out of 20.
02Legibility
24.9 / 30
03Economics
19.7 / 20
04Discoverability
12.4 / 20
Highest-impact fix
Estimated gain +30 pointsAdd explicit identity and permission preflight tools
Expose machine-readable principal/tenant confirmation and a non-mutating permission check so agents can verify both before destructive actions.
Description evidence
Defects and rewrites.
8 defects found across the exposed tool descriptions. Suggested rewrites make purpose, inputs, boundaries, and returns easier for an agent to understand.
| Tool | Defect types | Suggested rewrite |
|---|---|---|
transcribe |
params_unexplained |
Transcribe hours-long audio or video into an accurate, speaker-labeled (diarized), timestamped transcript with correctly-timed SRT/VTT captions. Pass a file_ref from upload_file or a url (YouTube video, podcast episode, RSS feed, Google Drive/Dropbox share). quality selects the speed/accuracy tradeoff: 'fast' for quicker results or the default 'accurate' for maximum accuracy. Audio is never used to train models. Returns { job_id, status }; fetch the result with get_transcription. |
get_transcription |
params_unexplained no_return_description |
Fetch a transcription by its job_id (the id returned by transcribe); the call returns the full text, per-speaker timestamped segments, summary, duration_seconds, billed_minutes, and language of that job. Set wait_ms (up to 25000) to long-poll so short clips come back in one call; otherwise poll until status is 'done'. |
upload_file |
no_return_description |
Upload local audio/video the agent is holding - as base64 bytes (bytes_base64) or a file path (path) - and returns a file_ref string to pass to transcribe. Use this when the media has no public URL. Requires the Pepys R2 storage backend. |
list_transcriptions |
params_unexplained |
List this account's recent transcription jobs with their job_id, status, title, and duration, so you can resume, fetch, or export an earlier result instead of re-transcribing. limit caps how many jobs are returned (default 20, max 100); status filters to only jobs with that status (queued, processing, done, failed, or canceled). |
list_podcast_episodes |
params_unexplained no_return_description |
Given a podcast RSS feed or Apple Podcasts show URL in feed_url, list its episodes; the call returns each episode's title, publish date, episode_guid, and audio_url so you can pick exactly which one to transcribe. limit caps how many episodes are returned (default 50, max 200). |
transcribe_podcast_feed |
params_unexplained |
Batch-transcribe a whole podcast feed: pass the feed's RSS URL in feed_url and the tool fans out every episode (or only the latest N via latest) to individual jobs, returning a set of job_ids. Set diarize to true to label who said what in each transcript. Paid capability (throughput/abuse gate). |
export_transcript |
params_unexplained no_return_description |
Export the finished transcript identified by job_id (returned by transcribe) in the given format - srt, vtt, txt, md, or json - and returns the transcript in that format with correct caption timings. Segment-level export is free; word-level-timed export (word_level: true) is a paid unlock. (DOCX/PDF are available in the Pepys web app.) |
search_transcript |
params_unexplained no_return_description |
Search inside a long transcript, identified by job_id, for a query phrase and returns only the matching timestamped segments, so you can locate a quote or topic without loading the whole transcript. query is the phrase to find; case_sensitive (default false) makes matching respect letter case; whole_word (default false) matches full words only; max_results caps the number of hits returned (default 20, max 100); context_segments also returns that many neighbor segments around each hit. |
Selection evidence
Confusable tool pairs.
5 pairs where similar names or overlapping descriptions may send an agent toward the wrong tool.
| Tool A | Tool B | Confidence | Why they collide |
|---|---|---|---|
transcribe |
transcribe_podcast_feed |
high | Both accept podcast RSS feeds, and transcribe explicitly documents RSS feeds as a valid url, so a task like 'transcribe this podcast feed' is genuinely ambiguous between single-episode (transcribe with episode_guid) and batch whole-feed (transcribe_podcast_feed). |
get_transcription |
list_transcriptions |
medium | A vague request like 'fetch my transcription' or 'get my earlier transcript' could route to get_transcription (which needs a job_id) or list_transcriptions, whose description explicitly mentions fetching/resuming earlier results; only knowing a specific job_id disambiguates. Shared 'fetch'/'duration' vocabulary reinforces the ambiguity. |
list_podcast_episodes |
transcribe_podcast_feed |
medium | Both target podcast feeds, and list_podcast_episodes even says it exists so you can 'pick exactly which one to transcribe'. A task like 'transcribe one episode from this feed' may trigger list_podcast_episodes, while 'transcribe the feed' may wrongly invoke the batch tool when only one episode or a listing was wanted. |
export_transcript |
search_transcript |
low | Both act on a finished transcript by job_id; a task like 'pull the relevant sections out of the transcript' could be read as search (returning matching segments) or export (producing a full transcript file). Descriptions mostly disambiguate via the action verbs. |
get_transcription |
get_credit_balance |
low | Shared 'minutes' vocabulary: a task like 'how many transcription minutes do I have' could lead to get_transcription (which reports billed_minutes) instead of get_credit_balance (remaining credits), though the descriptions frame remaining-credit checking clearly. |
Compare the field