01Safety
30.0 / 30
What changed in the harness
Selection accuracy 100→100, token cost up 4%, unconfirmed writes 100%→100%.
Category breakdown
Where the score comes from.
Earned points across the four signals Gradable measures. Safety and Legibility are scored out of 30; Economics and Discoverability are scored out of 20.
02Legibility
29.5 / 30
03Economics
20.0 / 20
04Discoverability
19.1 / 20
Highest-impact fix
Estimated gain +1 pointMake target tools discoverable on the first call
Clarify tool names, decision boundaries, and required argument schemas so an agent can choose and construct the target call without exploratory steps.
Description evidence
Defects and rewrites.
1 defect found across the exposed tool descriptions. Suggested rewrites make purpose, inputs, boundaries, and returns easier for an agent to understand.
| Tool | Defect types | Suggested rewrite |
|---|---|---|
obs_investigate_incident |
no_return_description |
Meta-tool that performs parallel root cause analysis (RCA) queries across all enabled backends. Automatically checks Grafana for firing alerts, Prometheus for offline endpoints (up==0), and Kafka clusters for offline brokers. Returns a consolidated summary of the findings per backend, including firing alerts, offline endpoints, and offline brokers. |
Selection evidence
Confusable tool pairs.
1 pair where similar names or overlapping descriptions may send an agent toward the wrong tool.
| Tool A | Tool B | Confidence | Why they collide |
|---|---|---|---|
obs_health_check |
obs_investigate_incident |
medium | Both tools run parallel checks 'across all configured observability backends' and surface backend health (up==0, offline brokers, firing alerts). A vague task like 'check the status of everything across all backends' or 'what's down across our observability stack?' matches both descriptions, and an agent might pick the health summary when RCA was implied, or the meta-tool when a simple status table was wanted. The sharing of 'across/all/backend/parallel' in both descriptions makes lexical retrieval ambiguous despite distinct intents. |
Compare the field