01Safety
0.0 / 30
What changed in the harness
Selection accuracy 97→95, token cost up 1%, unconfirmed writes 0%→0%.
Category breakdown
Where the score comes from.
Earned points across the four signals Gradable measures. Safety and Legibility are scored out of 30; Economics and Discoverability are scored out of 20.
02Legibility
26.7 / 30
03Economics
19.0 / 20
04Discoverability
13.3 / 20
Highest-impact fix
Estimated gain +30 pointsAdd explicit identity and permission preflight tools
Expose machine-readable principal/tenant confirmation and a non-mutating permission check so agents can verify both before destructive actions.
Description evidence
Defects and rewrites.
5 defects found across the exposed tool descriptions. Suggested rewrites make purpose, inputs, boundaries, and returns easier for an agent to understand.
| Tool | Defect types | Suggested rewrite |
|---|---|---|
roamzy_status |
no_return_description |
Check Roamzy API status and agent pause state; returns the overall service status plus a purchases_paused boolean flag. Agents MUST call this before purchase attempts and back off if purchases_paused=true. |
roamzy_list_countries |
no_return_description |
Returns the reference rate card: all 193 countries with their Roamzy per-MB rates in USDT. The user gets one universal eSIM; this list is the rate card, not a product catalog. Call this only if the user asks «what countries are supported» or «show me rates» — don't volunteer it during a purchase flow. |
roamzy_country_detail |
no_return_description |
Returns the per-MB rate in USDT for the given country_slug, as reference info. Call ONLY if the user explicitly asks about price for their destination (e.g. «how much in Japan?»). DO NOT call this during purchase — the user does not need to pick a country to buy. The eSIM works in all 193; rates are reference info, not a purchase gate. |
roamzy_estimate |
no_return_description |
Returns a reference estimate: how many USDT the given MB volume (mb_estimated) would cost at the given country's rate (country_slug). ONLY for informational «how much would 2GB in Japan cost» queries. DON'T use this to gate a purchase — the user doesn't pre-buy data volumes. They fund a USDT balance and pay per actual MB. |
roamzy_list_esims |
no_return_description |
Returns the authenticated user's list of eSIMs, each with status, balance, and msisdn. IMPORTANT: when referring to an eSIM in user-visible text, use msisdn (the eSIM phone number, e.g. '2040XXXXXX') — that is the only user-facing identifier. The id field is an internal ULID for follow-up API calls; do NOT surface it to the user. |
Selection evidence
Confusable tool pairs.
7 pairs where similar names or overlapping descriptions may send an agent toward the wrong tool.
| Tool A | Tool B | Confidence | Why they collide |
|---|---|---|---|
roamzy_status |
roamzy_order_status |
medium | A bare task like 'check the status' or 'what's the current Roamzy status' is ambiguous: it could mean API/agent-pause health (roamzy_status) or the progress of a pending purchase (roamzy_order_status). Both names and descriptions center on 'status', so the agent must rely on context it may not have. |
roamzy_list_esims |
roamzy_get_esim |
medium | A task like 'show me my eSIM' or 'get my eSIM status/balance' could route to either: list_esims returns status/balance/msisdn for all eSIMs with no args, while get_esim returns the same fields plus the activation QR for one specific eSIM and requires an internal esim_id the agent may not have. |
roamzy_get_esim |
roamzy_create_order |
medium | A user saying 'I want to get an eSIM for my trip' intends to purchase, but the phrasing 'get an eSIM' matches the tool name get_esim, which is for retrieving an already-created eSIM by id. A new account with no eSIM makes this misselection plausible. |
roamzy_list_countries |
roamzy_country_detail |
medium | A query like 'what's the data rate in France' is genuinely ambiguous: country_detail is the targeted reference, but list_countries also returns every per-MB rate including France, so an agent could wrongly choose the full rate card instead of the single-country lookup. |
roamzy_country_detail |
roamzy_create_order |
medium | A task mixing purchase and price, e.g. 'how much is an eSIM/data in Japan', straddles reference (country_detail) and purchase (create_order). The user may intend to buy while the agent answers the rate question instead, or pick create_order when only a price lookup was wanted. |
roamzy_status |
roamzy_me |
low | A task like 'what's the status of my account' could be read as API/agent status (roamzy_status) or account info (roamzy_me). Plausible only when the object is missing, since the descriptions are otherwise quite distinct. |
roamzy_list_countries |
roamzy_list_esims |
low | A vague task like 'list what Roamzy has' or 'show me the options' could map to either the supported-country rate card or the user's own eSIMs. The shared 'list' token plus an underspecified object makes this possible, though the semantic difference usually disambiguates. |
Compare the field