01Safety
0.0 / 30
What changed in the harness
Selection accuracy 98→98, token cost down 0%, unconfirmed writes 0%→0%.
Category breakdown
Where the score comes from.
Earned points across the four signals Gradable measures. Safety and Legibility are scored out of 30; Economics and Discoverability are scored out of 20.
02Legibility
25.9 / 30
03Economics
10.2 / 20
04Discoverability
15.4 / 20
Highest-impact fix
Estimated gain +30 pointsAdd explicit identity and permission preflight tools
Expose machine-readable principal/tenant confirmation and a non-mutating permission check so agents can verify both before destructive actions.
Description evidence
Defects and rewrites.
1 defect found across the exposed tool descriptions. Suggested rewrites make purpose, inputs, boundaries, and returns easier for an agent to understand.
| Tool | Defect types | Suggested rewrite |
|---|---|---|
raw_connection_reuse |
no_return_description |
Send multiple raw HTTP request strings sequentially over a single TLS connection to a target host, useful for testing connection-level attacks like request smuggling, pipeline confusion, and socket poisoning. Returns the raw response bytes received for each request sent on the connection, in order. |
Selection evidence
Confusable tool pairs.
12 pairs where similar names or overlapping descriptions may send an agent toward the wrong tool.
| Tool A | Tool B | Confidence | Why they collide |
|---|---|---|---|
sqli_blind_boolean |
sqli_blind_time |
high | A task like 'test this parameter for blind SQL injection' doesn't specify boolean vs time-based technique — both tools share identical url/parameter inputs and detect the same vulnerability class via different mechanisms, so the agent must guess which extraction method the user wants. |
race_single_packet |
race_last_byte_sync |
high | Both exploit TOCTOU race conditions with the identical target_url+requests schema; a generic task like 'test this coupon redemption endpoint for race conditions' gives no signal about HTTP/2 single-packet vs HTTP/1.1 last-byte-sync technique, so the agent could pick either. |
auth_cookie_tamper |
role_escalation_test |
high | Both manipulate cookies to test for privilege escalation on a protected URL; a task like 'see if I can escalate privileges by changing my session cookie' plausibly maps to either the generic cookie-tamper tool or the role-specific escalation tool, since role_escalation_test is really a specialized case of cookie tampering. |
cloudtrail_analyze |
cloudtrail_find_anomalies |
medium | Both operate on the same log_dir input; a vague request like 'check these CloudTrail logs for suspicious activity' could trigger the general analyze tool or the dedicated anomaly-finder, since 'analyze for anomalies' blurs the line between the two. |
volatility_linux |
volatility_windows |
medium | If the user says 'run volatility against this memory dump' without stating the OS, the agent must infer Linux vs Windows from context; both tools share the 'run a plugin against a memory dump' framing and only differ in plugin enum and profile requirement. |
nosqli_auth_bypass |
nosqli_detect |
medium | A task like 'test this login endpoint for NoSQL injection' could go to either the auth-bypass-specific tool or the general detection tool, since nosqli_detect also works on POST/JSON parameters and login forms are a common injection target for both. |
graphql_introspect |
graphql_find_hidden |
medium | A task like 'find hidden/undocumented fields in this GraphQL API' matches graphql_find_hidden's description almost verbatim but introspection is the standard first approach to enumerating all fields, so the agent may reach for graphql_introspect instead. |
maldoc_analyze |
maldoc_extract_macros |
medium | A task like 'extract the macros from this suspicious document' could invoke the dedicated extract_macros tool or the full maldoc_analyze pipeline (which also extracts VBA macros as part of its output), since both accept the same file_path and overlap in macro extraction. |
recon_quick |
recon_directory_bruteforce |
medium | recon_quick already returns 'accessible_directories' as part of its quick scan, so a task like 'find hidden directories on this site' could plausibly trigger either the lightweight recon_quick or the dedicated recon_directory_bruteforce. |
raw_http_send |
raw_connection_reuse |
medium | Both send raw HTTP bytes for low-level attacks like smuggling; a task such as 'test this server for request smuggling' doesn't indicate whether a single crafted request or multiple requests over one connection is needed, and both tools target the same use case in their descriptions. |
recon_dns |
recon_vhost |
low | A task like 'enumerate subdomains of this target' could be routed to DNS record enumeration (recon_dns, via AXFR/CNAME) or virtual-host Host-header fuzzing (recon_vhost), since both discover subdomain-like names though via very different mechanisms. |
auth_bruteforce |
nosqli_auth_bypass |
low | A generic instruction like 'try to bypass this login' could map to credential brute-forcing or NoSQL operator injection, since both target authentication bypass on a login URL, though the tool descriptions make the technique distinct enough to usually disambiguate. |
Compare the field