01Safety
0.0 / 30
What changed in the harness
Selection accuracy 96→98, token cost down 2%, unconfirmed writes 0%→0%.
Category breakdown
Where the score comes from.
Earned points across the four signals Gradable measures. Safety and Legibility are scored out of 30; Economics and Discoverability are scored out of 20.
02Legibility
24.5 / 30
03Economics
10.6 / 20
04Discoverability
13.4 / 20
Highest-impact fix
Estimated gain +30 pointsAdd explicit identity and permission preflight tools
Expose machine-readable principal/tenant confirmation and a non-mutating permission check so agents can verify both before destructive actions.
Description evidence
Defects and rewrites.
4 defects found across the exposed tool descriptions. Suggested rewrites make purpose, inputs, boundaries, and returns easier for an agent to understand.
| Tool | Defect types | Suggested rewrite |
|---|---|---|
pcap_follow_stream |
name_restates_behavior |
Reassemble a single TCP, UDP, or HTTP conversation from a PCAP into its full stream contents, useful for inspecting raw request/response exchanges such as an individual HTTP login or protocol handshake. Pick the conversation with stream_num and the protocol type; returns the reassembled stream_content along with the stream_num and protocol used. Read-only file analysis. |
recon_tls_sans |
name_restates_behavior |
Perform a one-shot TLS handshake against a target to retrieve its certificate, used during reconnaissance to discover related hostnames and audit certificate details. Returns common_name, subject_alternative_names, issuer, validity, and san_count. Read-only TLS handshake. |
ssrf_cloud_metadata |
name_restates_behavior |
Verify whether an SSRF-vulnerable parameter can reach cloud instance metadata endpoints (AWS/GCP/Azure); successful access can leak IAM credentials. Choose a cloud_provider (or 'all') and the tool sends requests to each provider's well-known metadata service endpoint through the given parameter, reporting provider, endpoint, status, length, and response_snippet. Side effects: May cause the target to request cloud metadata and could expose IAM credentials if successful. |
raw_connection_reuse |
no_return_description |
Send multiple raw HTTP requests sequentially over a single TLS connection. Useful for testing connection-level attacks like request smuggling, pipeline confusion, and socket poisoning. Returns an array of the responses received for each request. |
Selection evidence
Confusable tool pairs.
16 pairs where similar names or overlapping descriptions may send an agent toward the wrong tool.
| Tool A | Tool B | Confidence | Why they collide |
|---|---|---|---|
sqli_blind_boolean |
sqli_blind_time |
medium | Both are 'blind SQLi' tools with the same url+parameter schema; a task like 'test this parameter for blind SQL injection' does not say boolean vs time-based, so the agent may pick either detection strategy. |
pcap_dns_analysis |
pcap_tls_analysis |
medium | Identical input shape (pcap_path) and both are pcap protocol 'analysis' tools; a task like 'analyze the traffic in this pcap' leaves the protocol unspecified, so the agent may guess between DNS and TLS analysis. |
volatility_linux |
volatility_windows |
medium | Both run Volatility plugins against a memory dump; a task like 'analyze this memory dump / run pslist on it' often omits the OS, and the OS alone determines which tool is correct, so the agent may pick the wrong one. |
nosqli_auth_bypass |
nosqli_detect |
medium | Both are NoSQL injection tools; a task like 'test NoSQL injection on the login endpoint' fits both the auth-bypass tool and the generic detection tool, so the agent may pick the wrong scope. |
cloudtrail_analyze |
cloudtrail_find_anomalies |
medium | Same input (cloudtrail logs) and overlapping goal; a task like 'analyze/investigate these CloudTrail logs for suspicious activity' could be routed to the parse-overview tool instead of the anomaly finder, or vice versa. |
graphql_introspect |
graphql_find_hidden |
medium | Both descriptions advertise finding 'hidden/undocumented fields', so a task like 'find hidden/undocumented GraphQL fields' matches both and the agent may pick introspection instead of field-suggestion probing. |
maldoc_analyze |
maldoc_extract_macros |
medium | maldoc_analyze's description explicitly includes 'extract VBA macro code', so a task like 'extract the macros from this malicious document' could route to the heavier full-pipeline tool instead of the targeted extractor. |
race_single_packet |
race_last_byte_sync |
medium | Both are race-condition attacks with nearly identical request-array schemas; a task like 'test this endpoint for race conditions / double-spend' does not specify HTTP/2 single-packet vs HTTP/1.1 last-byte delivery, so either may be chosen. |
raw_http_send |
raw_connection_reuse |
medium | Both descriptions mention request smuggling/raw requests over TLS; a task like 'send raw requests to test for request smuggling or socket poisoning' could route to the single-request sender instead of the connection-reuse tool, or vice versa. |
auth_cookie_tamper |
role_escalation_test |
medium | Both test privilege escalation by sending tampered role-style cookie values; a task like 'test if I can escalate to admin by modifying cookies' matches both descriptions almost identically. |
recon_directory_bruteforce |
auth_bruteforce |
medium | Both are brute-force tools sharing 'brute/bruteforce/force'; a task like 'brute-force the target server' does not say whether to brute-force directories or credentials, so the agent may pick the wrong one. |
recon_vhost |
recon_directory_bruteforce |
medium | Both brute-force a web server using wordlists (Host-header fuzzing vs path fuzzing) and take a target; a generic task like 'brute force/fuzz example.com' is ambiguous between the two. |
recon_dns |
recon_vhost |
low | Subdomain/hostname tasks like 'enumerate subdomains of example.com' could route to DNS record enumeration (recon_dns) or Host-header vhost brute force (recon_vhost) depending on which approach the agent guesses is intended. |
auth_bruteforce |
auth_cookie_tamper |
low | A task like 'gain admin access / break into the account' could be interpreted as credential guessing (bruteforce) or session/role cookie manipulation (cookie_tamper); both are auth access-gaining tools. |
auth_bruteforce |
nosqli_auth_bypass |
low | A task like 'bypass the login / get into the account' could lead to either guessing passwords (bruteforce) or injecting NoSQL operators (nosqli_auth_bypass) when the agent does not know the backend. |
xss_payload_generate |
oob_generate_payload |
low | Both generate attacker payloads against a callback/OAST domain; a task like 'generate a payload that calls back to my OAST domain' could route to the generic OOB payload generator or the XSS generator (which has a callback_url for exfiltration). |
Compare the field