01Safety
0.0 / 30
What changed in the harness
Selection accuracy 100→100, token cost up 7%, unconfirmed writes 0%→0%.
Category breakdown
Where the score comes from.
Earned points across the four signals Gradable measures. Safety and Legibility are scored out of 30; Economics and Discoverability are scored out of 20.
02Legibility
27.2 / 30
03Economics
19.6 / 20
04Discoverability
11.7 / 20
Highest-impact fix
Estimated gain +30 pointsAdd explicit identity and permission preflight tools
Expose machine-readable principal/tenant confirmation and a non-mutating permission check so agents can verify both before destructive actions.
Description evidence
Defects and rewrites.
7 defects found across the exposed tool descriptions. Suggested rewrites make purpose, inputs, boundaries, and returns easier for an agent to understand.
| Tool | Defect types | Suggested rewrite |
|---|---|---|
transcribe |
params_unexplained |
Transcribe hours-long audio or video into an accurate, speaker-labeled (diarized), timestamped transcript with correctly-timed SRT/VTT captions – work a general model can't do on a raw file. Accepts a file_ref from upload_file or a url (YouTube, podcast episode, RSS feed, Google Drive/Dropbox share). quality selects the transcription engine tier: 'fast' trades accuracy for speed, 'accurate' (default) is slower but more precise. Audio is never used to train models. Returns { job_id, status }; fetch the result with get_transcription. |
get_transcription |
params_unexplained |
Fetch a transcription job by the job_id returned from transcribe: returns full text, per-speaker timestamped segments, summary, duration_seconds, billed_minutes, and language. Set wait_ms (up to 25000) to long-poll so short clips come back in one call; otherwise poll until status is 'done'. |
list_transcriptions |
params_unexplained |
List this account's recent transcription jobs with their job_id, status, title, and duration, so you can resume, fetch, or export an earlier result instead of re-transcribing. limit caps how many jobs are returned (default 20, max 100); status filters the list to jobs in a specific state (queued, processing, done, failed, or canceled). |
list_podcast_episodes |
params_unexplained |
Given a podcast RSS feed or Apple Podcasts show URL (feed_url), list its episodes (title, publish date, episode_guid, audio_url) so you can pick exactly which one to transcribe. limit caps how many episodes are returned (default 50, max 200). |
transcribe_podcast_feed |
params_unexplained |
Batch-transcribe a whole podcast feed (feed_url) in one call – fan out every episode, or the latest N (latest), to individual jobs. diarize labels who said what in each resulting transcript (paid capability, default false). Returns a set of job_ids. Paid capability (throughput/abuse gate). |
export_transcript |
params_unexplained |
Export a finished transcript, identified by the job_id returned from transcribe, as SRT, VTT, TXT, Markdown, or JSON, with correct caption timings. Segment-level export is free; word-level-timed export (word_level:true) is a paid unlock. (DOCX/PDF are available in the Pepys web app.) |
search_transcript |
params_unexplained |
Search inside a transcript, identified by the job_id returned from transcribe, for a phrase (query) and get back only the matching timestamped segments – locate a quote or topic in an hours-long recording without loading the whole transcript into context. case_sensitive toggles case-sensitive matching (default false); whole_word restricts matches to whole-word boundaries (default false); context_segments also returns that many neighboring segments around each hit (0-3, default 0); max_results caps how many matches are returned (default 20, max 100). |
Selection evidence
Confusable tool pairs.
5 pairs where similar names or overlapping descriptions may send an agent toward the wrong tool.
| Tool A | Tool B | Confidence | Why they collide |
|---|---|---|---|
transcribe |
transcribe_podcast_feed |
high | A task like 'transcribe this podcast feed' or 'transcribe episode 3 of this feed' is ambiguous: transcribe accepts a feed url with episode_index/episode_guid for a single episode, while transcribe_podcast_feed batch-transcribes the whole feed or latest N — an agent could easily pick the batch tool when only one episode was wanted, or vice versa. |
get_transcription |
list_transcriptions |
medium | A request like 'get my recent transcription' could trigger either: get_transcription needs a specific job_id to fetch full content, while list_transcriptions returns recent jobs without needing an id — an agent unsure of the job_id might wrongly call get_transcription without one, or use list when it actually needs full transcript content. |
list_transcriptions |
list_podcast_episodes |
low | Both are 'list' tools with title metadata, but one lists account transcription jobs and the other lists episodes from an external feed URL — the required inputs (none vs feed_url) are different enough that confusion is unlikely except in a vague 'list my episodes' phrasing. |
get_transcription |
get_credit_balance |
low | Both mention 'minutes' (billed_minutes vs credit balance in minutes), so a task like 'how many minutes do I have left' could theoretically be confused with checking a job's billed_minutes, but the tools serve clearly distinct purposes and get_transcription requires a job_id making misselection unlikely. |
export_transcript |
search_transcript |
low | Both operate on a finished transcript via job_id, but one exports full formatted output and the other searches for a phrase — a request like 'get me the part about X from the transcript' could plausibly be routed to export instead of search, though the intents are fairly distinguishable. |
Compare the field