01Safety
0.0 / 30
What changed in the harness
Selection accuracy 92→94, token cost up 3%, unconfirmed writes 0%→0%.
Category breakdown
Where the score comes from.
Earned points across the four signals Gradable measures. Safety and Legibility are scored out of 30; Economics and Discoverability are scored out of 20.
02Legibility
26.1 / 30
03Economics
19.9 / 20
04Discoverability
15.5 / 20
Highest-impact fix
Estimated gain +30 pointsAdd explicit identity and permission preflight tools
Expose machine-readable principal/tenant confirmation and a non-mutating permission check so agents can verify both before destructive actions.
Description evidence
Defects and rewrites.
2 defects found across the exposed tool descriptions. Suggested rewrites make purpose, inputs, boundaries, and returns easier for an agent to understand.
| Tool | Defect types | Suggested rewrite |
|---|---|---|
list_conversions |
no_return_description |
List all supported conversion routes as source/target format pairs. Optionally filter by source_format to see only the routes available for converting FROM that format. Returns an array of supported {source_format, target_format} conversion pairs. |
do_file_job |
no_return_description |
Describe what you want done to a file in plain language — e.g. "translate this contract to German", "pull every table out of this PDF into Excel", "shrink this video to under 25MB", "convert this to PDF". The instruction is routed to the right underlying job (translate_file, extract_tables, compress_file, convert_file, etc.) automatically. Returns that job's result — typically a job_id plus a download URL or a free preview and checkout link, depending on the matched job type — or, if the request isn't supported yet, an honest explanation of what is. Provide EITHER source_url OR base64_content, plus a filename with extension. |
Selection evidence
Confusable tool pairs.
7 pairs where similar names or overlapping descriptions may send an agent toward the wrong tool.
| Tool A | Tool B | Confidence | Why they collide |
|---|---|---|---|
convert_file |
do_file_job |
high | do_file_job's own description gives 'convert this to PDF' as a canonical example, so a plain-language convert request plausibly routes to either the dedicated convert_file tool or the generic do_file_job dispatcher. |
translate_file |
do_file_job |
high | do_file_job explicitly lists 'translate this contract to German' as an example use case, directly overlapping with translate_file's purpose, so an agent could pick either for a translation request. |
compress_file |
do_file_job |
high | do_file_job's description includes 'shrink this video to under 25MB' as a sample instruction, which is exactly compress_file's use case, creating real selection ambiguity. |
convert_file |
compress_file |
medium | Requests like 'make this PDF smaller' or 'compress this video to mp4' blur the line between changing format (convert_file's target_format) and reducing size (compress_file's target_size_kb), especially if the user conflates format change with size reduction. |
do_file_job |
check_job |
low | Both concern 'file jobs' but do_file_job initiates work while check_job only checks status of an existing job_id; the verbs (do vs check) and required params (instruction vs job_id) make confusion unlikely. |
convert_file |
translate_file |
low | Both operate on documents and share 'file' terminology, but converting file format and translating language are semantically distinct enough that confusion is unlikely except in poorly worded requests. |
translate_file |
compress_file |
low | Both accept PDFs and file uploads but serve unrelated purposes (language translation vs size reduction), making genuine mis-selection unlikely. |
Compare the field