01Safety
0.0 / 30
What changed in the harness
Selection accuracy 93→100, token cost up 5%, unconfirmed writes 0%→0%.
Category breakdown
Where the score comes from.
Earned points across the four signals Gradable measures. Safety and Legibility are scored out of 30; Economics and Discoverability are scored out of 20.
02Legibility
26.9 / 30
03Economics
20.0 / 20
04Discoverability
13.1 / 20
Highest-impact fix
Estimated gain +30 pointsAdd explicit identity and permission preflight tools
Expose machine-readable principal/tenant confirmation and a non-mutating permission check so agents can verify both before destructive actions.
Description evidence
Defects and rewrites.
2 defects found across the exposed tool descriptions. Suggested rewrites make purpose, inputs, boundaries, and returns easier for an agent to understand.
| Tool | Defect types | Suggested rewrite |
|---|---|---|
scanner_get_scan_status |
name_restates_behavior params_unexplained no_return_description |
Poll the progress of a scan previously queued via scanner_start_scan. Requires the scan_id returned by that call. Returns the scan's current status (queued, running, completed, or failed) along with the number of findings discovered so far. |
scanner_get_findings |
params_unexplained no_return_description |
Retrieve the security findings produced by a completed scan. Requires the scan_id returned by scanner_start_scan. Returns a list of findings, each with severity, CVE identifier, detection template, title, affected URL, and priority_score, sorted from highest to lowest priority. |
Selection evidence
Confusable tool pairs.
3 pairs where similar names or overlapping descriptions may send an agent toward the wrong tool.
| Tool A | Tool B | Confidence | Why they collide |
|---|---|---|---|
scanner_start_scan |
scanner_get_scan_status |
medium | A vague instruction like 'check on the scan' or 'run the scan' could lead an agent to call get_scan_status when it means to kick off scanning, or vice versa, especially since start_scan's description references polling status immediately after. |
scanner_get_scan_status |
scanner_get_findings |
low | Both relate to inspecting scan progress/results and take the same scan_id parameter, so a request like 'what did the scan find' could momentarily be routed to status instead of findings, but the descriptions (status counts vs listing findings) are distinct enough to keep confusion low. |
scanner_register_target |
scanner_verify_target |
low | A task like 'verify my domain for scanning' could be misrouted to register_target since both are part of the onboarding flow, but register_target requires a URL and verify_target requires a target_id, making the correct choice fairly clear from context already established (a target_id would exist after registration). |
Compare the field