01Safety
0.0 / 30
What changed in the harness
Selection accuracy 100%, destructive-action safety rate 0% (baseline only -- no rewrite pass applied).
Category breakdown
Where the score comes from.
Earned points across the four signals Gradable measures. Safety and Legibility are scored out of 30; Economics and Discoverability are scored out of 20.
02Legibility
29.1 / 30
03Economics
20.0 / 20
04Discoverability
13.1 / 20
Highest-impact fix
Estimated gain +30 pointsAdd explicit identity and permission preflight tools
Expose machine-readable principal/tenant confirmation and a non-mutating permission check so agents can verify both before destructive actions.
Description evidence
Defects and rewrites.
0 defects found across the exposed tool descriptions. Suggested rewrites make purpose, inputs, boundaries, and returns easier for an agent to understand.
| Tool | Defect types | Suggested rewrite |
|---|---|---|
| No description defects were flagged in this assessment. | ||
Selection evidence
Confusable tool pairs.
5 pairs where similar names or overlapping descriptions may send an agent toward the wrong tool.
| Tool A | Tool B | Confidence | Why they collide |
|---|---|---|---|
orcarouter_models_list |
orcarouter_model_card |
medium | A task like 'give me the pricing and context length for openai/gpt-4o-mini' is ambiguous because the catalog listing also returns pricing, context_length, and description per model. The descriptions try to split browsing (models_list) vs known-id details (model_card), but 'tell me about / details on model X' phrasing can pull an agent to either tool, especially since models_list needs no pre-known id. |
orcarouter_models_list |
orcarouter_providers_list |
medium | Both are read-only catalog listers, and models_list's entries include owned_by (provider attribution) while providers_list includes model_count. A task like 'list everything OrcaRouter has / what providers and models are available' is ambiguous — the agent may call models_list to enumerate providers instead of providers_list, or pick the wrong domain for vague 'show me what's available' phrasing. |
orcarouter_chat |
orcarouter_model_card |
low | A task such as 'ask gpt-4o-mini about its capabilities and pricing' could plausibly route to chat (the only interaction tool) instead of model_card, since the user says 'ask the model'; the model_card description is the intended lookup but the natural wording invites chat. Weak but possible. |
orcarouter_chat |
orcarouter_models_list |
low | With a request like 'which models can I chat with / find a chat-capable model after testing one', the agent could pick models_list (capability='chat' filter) when the user actually wants to send a turn via chat, or vice versa for 'test this model'. The capability enum overlaps with the chatter intent, but the descriptions keep send-vs-enumerate mostly distinct. |
orcarouter_model_card |
orcarouter_providers_list |
low | A task phrased as 'show me the details/card for anthropic providers' is ambiguous: providers_list returns provider display_name and model_count, while model_card expects a provider/slug model ref. An agent given only a provider name may wrongly call model_card with an incomplete ref (hitting its not-found error) instead of providers_list. |
Compare the field