01Safety
0.0 / 30
What changed in the harness
Selection accuracy 98→96, token cost up 8%, unconfirmed writes 0%→0%.
Category breakdown
Where the score comes from.
Earned points across the four signals Gradable measures. Safety and Legibility are scored out of 30; Economics and Discoverability are scored out of 20.
02Legibility
26.5 / 30
03Economics
19.9 / 20
04Discoverability
11.2 / 20
Highest-impact fix
Estimated gain +30 pointsAdd explicit identity and permission preflight tools
Expose machine-readable principal/tenant confirmation and a non-mutating permission check so agents can verify both before destructive actions.
Description evidence
Defects and rewrites.
6 defects found across the exposed tool descriptions. Suggested rewrites make purpose, inputs, boundaries, and returns easier for an agent to understand.
| Tool | Defect types | Suggested rewrite |
|---|---|---|
list_models |
name_restates_behavior params_unexplained no_return_description |
Filter the embedded RunAPI catalog and return the models matching optional criteria: modality restricts to image, video, audio, utility, or llm; service narrows by service; action narrows by endpoint name. With no filters, returns the full catalog of available models. |
get_model_info |
no_return_description |
Resolve the endpoint, current runtime pricing, and input constraints for a RunAPI model slug. Add service and action when the model supports multiple endpoints to disambiguate the endpoint. Returns the resolved endpoint details, current pricing, and input constraints for the specified model. |
search_prompts |
params_unexplained no_return_description |
Search RunAPI prompt examples using optional filters: modality restricts to image, image_edit, video, audio, or music; category filters by prompt category; tags matches examples containing all provided tags; q matches prompt title and prompt text; model restricts to a model slug (e.g. flux-kontext-pro or suno-v5); featured selects only featured examples; page and per_page (max 100) control pagination. Free, no API key required. Returns the matching prompt examples. |
create_task |
no_return_description |
Run a RunAPI operation with a caller-generated idempotency key. Asynchronous operations can optionally poll until completion. Returns the created task and its current status; when wait is true, it polls until the task reaches a terminal status before returning. |
get_task |
params_unexplained no_return_description |
Retrieve the current status and latest payload for an existing RunAPI task. service is the RunAPI service slug returned by list_models; task_id identifies the task to fetch; provide action when using media task routes. Returns the current status and latest payload of the task. |
login |
no_return_description |
Authenticate RunAPI by opening a browser PKCE login flow and saving the API key to ~/.config/runapi/config.json. force re-runs the browser login when the current credential comes from the local config file. Completes login and stores the API key, returning once authentication succeeds. |
Selection evidence
Confusable tool pairs.
3 pairs where similar names or overlapping descriptions may send an agent toward the wrong tool.
| Tool A | Tool B | Confidence | Why they collide |
|---|---|---|---|
list_models |
list_actions |
low | Both return catalog-style lists with no required inputs; a task like 'list what I can run' or 'show available endpoints/models' is genuinely ambiguous, though most phrasing (models vs actions) disambiguates. |
list_models |
get_model_info |
medium | A request like 'get info about the available models' could map to either the catalog list or per-model detail, especially since get_model_info also returns pricing and constraints and users rarely phrase 'list vs detail' precisely. |
get_model_info |
check_pricing |
medium | get_model_info explicitly returns 'current runtime pricing', so a task like 'what does model X cost' or 'check pricing for this model' is plausibly routed to get_model_info when given only a model slug, even though check_pricing is the pricing-specific tool. |
Compare the field