01Safety
0.0 / 30
What changed in the harness
Selection accuracy 100→100, token cost up 7%, unconfirmed writes 0%→0%.
Category breakdown
Where the score comes from.
Earned points across the four signals Gradable measures. Safety and Legibility are scored out of 30; Economics and Discoverability are scored out of 20.
02Legibility
27.4 / 30
03Economics
20.0 / 20
04Discoverability
14.0 / 20
Highest-impact fix
Estimated gain +30 pointsAdd explicit identity and permission preflight tools
Expose machine-readable principal/tenant confirmation and a non-mutating permission check so agents can verify both before destructive actions.
Description evidence
Defects and rewrites.
4 defects found across the exposed tool descriptions. Suggested rewrites make purpose, inputs, boundaries, and returns easier for an agent to understand.
| Tool | Defect types | Suggested rewrite |
|---|---|---|
volt_recommend_route |
no_return_description |
Finds the optimal provider for a model based on cost, latency, reliability, or balanced optimization. Returns the recommended provider with estimated cost, latency, and quality metrics, plus estimated savings compared to your current cost per million tokens. Filters respect the min_quality, max_latency_ms, and blocked_providers constraints. |
volt_get_spend |
no_return_description |
Returns a spending summary for today, 7 days, or 30 days, broken down by provider and model, so you can see total spend and where it is concentrated. |
volt_get_savings |
no_return_description |
Returns a report comparing actual spend against optimal routing for today, 7 days, or 30 days, including total savings achieved, savings missed, and breakdowns by provider and model. |
volt_set_budget_alert |
name_restates_behavior no_return_description |
Configures a spend threshold in USD for a daily, weekly, or monthly period. When spend for the period exceeds the threshold, an alert is triggered. Returns the created alert configuration with its threshold and period. |
Selection evidence
Confusable tool pairs.
6 pairs where similar names or overlapping descriptions may send an agent toward the wrong tool.
| Tool A | Tool B | Confidence | Why they collide |
|---|---|---|---|
volt_get_spend |
volt_get_savings |
medium | Both take identical time_range args and report dollar figures over the same periods. A task like 'show me my savings for 30d' or 'get my spend/savings summary' is ambiguous about whether the user wants raw spend totals or routing savings achieved. |
volt_check_price |
volt_get_spend |
medium | A task like 'what am I paying for gpt-4o?' could mean lookup the provider price for that model (check_price) or the user's actual billed spend broken down by provider/model (get_spend); the shared model/provider vocabulary makes the wrong pick plausible. |
volt_check_price |
volt_recommend_route |
medium | Both accept a model and deal with provider choice and cost. 'What's the cheapest provider for llama-70b?' could be answered by check_price (sorted by price) or recommend_route with optimize=cost; the recommendation description explicitly mentions savings, further blurring the line. |
volt_recommend_route |
volt_get_savings |
medium | Both feature the word 'savings' prominently. 'How much am I saving?' is ambiguous between achieved/actual savings (get_savings) and potential savings versus current cost that recommend_route advertises, especially without explicit tense or routing context. |
volt_check_price |
volt_get_savings |
low | Shared token is only 'compare', and each compares different things (provider pricing vs actual spend vs optimal routing). A task like 'compare what I'd save across providers' is contrived; selection confusion is unlikely but not impossible. |
volt_recommend_route |
volt_get_spend |
low | get_spend has no model/provider arguments and describes a raw spend report, while recommend_route is about choosing a provider for a model. A task mixing 'what I'm spending on gpt-4o and what I should switch to' could touch both, but wrong-member selection on the core intent is unlikely. |
Compare the field