01Safety
0.0 / 30
What changed in the harness
Selection accuracy 100→100, token cost up 1%, unconfirmed writes 0%→0%.
Category breakdown
Where the score comes from.
Earned points across the four signals Gradable measures. Safety and Legibility are scored out of 30; Economics and Discoverability are scored out of 20.
02Legibility
28.3 / 30
03Economics
16.7 / 20
04Discoverability
15.3 / 20
Highest-impact fix
Estimated gain +30 pointsAdd explicit identity and permission preflight tools
Expose machine-readable principal/tenant confirmation and a non-mutating permission check so agents can verify both before destructive actions.
Description evidence
Defects and rewrites.
1 defect found across the exposed tool descriptions. Suggested rewrites make purpose, inputs, boundaries, and returns easier for an agent to understand.
| Tool | Defect types | Suggested rewrite |
|---|---|---|
cancel_workflow |
no_return_description |
Cancel an in-progress workflow by ID. All queued and dispatched steps are marked CANCELLED; completed steps are not reversed, and you are only billed for steps that completed before cancellation. Returns a workflow_id echoing your input, a boolean cancelled indicating whether cancellation succeeded, and the resulting workflow status (e.g. CANCELLED or COMPLETED). |
Selection evidence
Confusable tool pairs.
7 pairs where similar names or overlapping descriptions may send an agent toward the wrong tool.
| Tool A | Tool B | Confidence | Why they collide |
|---|---|---|---|
get_job_status |
get_workflow_status |
medium | A user asking 'what's the status of my job?' is ambiguous when the thing submitted was a workflow — get_workflow_status takes a workflow_id while get_job_status takes a job_id, and the two 'status' tools are easily conflated. |
transcribe_from_url |
transcribe_media |
medium | Both run the identical one-call transcription pipeline; the only difference is source. A task like 'transcribe this recording' is ambiguous about whether the user has a public URL (from_url) or a local/large file needing upload first (transcribe_media). |
transcode_from_url |
transcode_video |
medium | Both perform the same transcode, differing only in source: transcode_from_url takes a public URL, transcode_video takes an object_key from get_upload_url. A user saying 'transcode this video' without specifying whether it's reachable by URL can send the agent down the wrong path. |
transcode_from_url |
convert_from_url |
medium | Both accept a public URL and both say 'offload to Botverse — returns a job_id immediately'; a task like 'convert the file at this URL' is ambiguous, and the 'convert' keyword plus URL input can push an agent to convert_from_url even when the source is a video/audio file that needs transcoding. |
convert_file |
convert_content |
low | Real but mild ambiguity: a user who already has file bytes/text says 'convert this document', and convert_content (inline string/base64) is described as preferred when content is in hand, while convert_file wrongly assumes an object_key from get_upload_url. |
convert_from_url |
convert_content |
low | Both convert documents inline-style; only the source differs (public URL vs content string). Descriptions explicitly disambiguate them, so confusion is plausible mostly when the user's task does not state whether the document is remote or already held. |
get_upload_url |
get_download_url |
low | Both return presigned URLs and share nearly all tokens; a vague request like 'give me the URL for my file' is genuinely ambiguous about direction, though most tasks explicitly say upload or download. |
Compare the field