01Safety
30.0 / 30
What changed in the harness
Selection accuracy 89→97, token cost up 2%, unconfirmed writes 100%→100%.
Category breakdown
Where the score comes from.
Earned points across the four signals Gradable measures. Safety and Legibility are scored out of 30; Economics and Discoverability are scored out of 20.
02Legibility
27.2 / 30
03Economics
20.0 / 20
04Discoverability
17.8 / 20
Highest-impact fix
Estimated gain +1 pointRewrite 2 underspecified tool descriptions
State each tool's behavior, explain every parameter, and declare return semantics, prioritizing tools exercised by failed selection tasks.
Description evidence
Defects and rewrites.
2 defects found across the exposed tool descriptions. Suggested rewrites make purpose, inputs, boundaries, and returns easier for an agent to understand.
| Tool | Defect types | Suggested rewrite |
|---|---|---|
analyze_iam_policy |
no_return_description |
AI-Powered security analysis of an AWS IAM policy (identity policy or cross-account trust+identity pair). Returns an assessment report flagging privilege-escalation paths, wildcards, and other over-permissive access. Runs in demo mode (rate-limited, no signup) if SHIELDLY_API_KEY is not set. |
analyze_cloudformation_template |
no_return_description |
AI-Powered security analysis of a CloudFormation template — extracts IAM roles/policies and flags over-permissive access. Returns an assessment report of the findings. Runs in demo mode (rate-limited, no signup) if SHIELDLY_API_KEY is not set. |
Selection evidence
Confusable tool pairs.
1 pair where similar names or overlapping descriptions may send an agent toward the wrong tool.
| Tool A | Tool B | Confidence | Why they collide |
|---|---|---|---|
analyze_iam_policy |
analyze_cloudformation_template |
medium | A task mentioning 'analyze the IAM permissions/security of my CloudFormation template for over-permissive access' is genuinely ambiguous: analyze_iam_policy explicitly mentions IAM, over-permissive access, and security, so an agent may feed the template string into it, even though analyze_cloudformation_template exists specifically for extracting IAM roles/policies from templates. The overlapping IAM/security language and lack of an explicit pointer to CloudFormation templates makes misselection plausible. |
Compare the field