01Safety
30.0 / 30
What changed in the harness
Selection accuracy 98→100, token cost up 20%, unconfirmed writes 100%→100%.
Category breakdown
Where the score comes from.
Earned points across the four signals Gradable measures. Safety and Legibility are scored out of 30; Economics and Discoverability are scored out of 20.
02Legibility
27.6 / 30
03Economics
20.0 / 20
04Discoverability
10.4 / 20
Highest-impact fix
Estimated gain +10 pointsMake target tools discoverable on the first call
Clarify tool names, decision boundaries, and required argument schemas so an agent can choose and construct the target call without exploratory steps.
Description evidence
Defects and rewrites.
3 defects found across the exposed tool descriptions. Suggested rewrites make purpose, inputs, boundaries, and returns easier for an agent to understand.
| Tool | Defect types | Suggested rewrite |
|---|---|---|
codewiki_search_repos |
params_unexplained no_return_description |
Search repositories indexed by codewiki.google. 'query' is the search text to match against repository names/descriptions; 'limit' caps the number of results returned (default 10, max 50). Returns a list of matching repositories with identifying metadata (e.g. repo name/path) for use with other codewiki tools. |
codewiki_fetch_repo |
params_unexplained no_return_description |
Fetch generated wiki content for a repository from codewiki.google. 'repo' identifies the repository (e.g. name or path as returned by codewiki_search_repos); 'mode' selects the output shape — 'aggregate' (default) returns the wiki content combined into a single document, while 'pages' returns it split into individual pages. Returns the repository's generated wiki text in the selected format. |
codewiki_ask_repo |
params_unexplained no_return_description |
Ask a natural-language question about a specific repository indexed in codewiki.google, using its generated wiki as context. 'repo' identifies the repository to query; 'question' is the natural-language question to ask; 'history' is an optional list of prior {role, content} turns (up to 20) to provide conversational context for follow-up questions. Returns a natural-language answer grounded in the repository's indexed content. |
Selection evidence
Confusable tool pairs.
3 pairs where similar names or overlapping descriptions may send an agent toward the wrong tool.
| Tool A | Tool B | Confidence | Why they collide |
|---|---|---|---|
codewiki_fetch_repo |
codewiki_ask_repo |
medium | A task like 'tell me about the auth module in repo X' could be satisfied either by fetching the aggregate wiki content or by asking a targeted question; both take a repo identifier and return descriptive text, so an agent might pick fetch_repo when ask_repo would better answer a specific question, or vice versa. |
codewiki_search_repos |
codewiki_ask_repo |
low | Both accept free-text query-like input, but search_repos finds repositories by name/topic while ask_repo requires a specific repo already identified plus a question; confusion is unlikely since ask_repo requires a 'repo' argument the agent must already know, making the intent distinction clear in most phrasings. |
codewiki_search_repos |
codewiki_fetch_repo |
low | A vague request like 'find info on repo X' could tempt an agent to call search_repos instead of fetch_repo, but search_repos is clearly for discovering unknown repos via a query while fetch_repo requires a known repo identifier, so mixing them up is only plausible when the user hasn't specified an exact repo name. |
Compare the field