01Safety
0.0 / 30
What changed in the harness
Selection accuracy 100→97, token cost down 0%, unconfirmed writes 0%→0%.
Category breakdown
Where the score comes from.
Earned points across the four signals Gradable measures. Safety and Legibility are scored out of 30; Economics and Discoverability are scored out of 20.
02Legibility
26.9 / 30
03Economics
17.6 / 20
04Discoverability
11.7 / 20
Highest-impact fix
Estimated gain +30 pointsAdd explicit identity and permission preflight tools
Expose machine-readable principal/tenant confirmation and a non-mutating permission check so agents can verify both before destructive actions.
Description evidence
Defects and rewrites.
2 defects found across the exposed tool descriptions. Suggested rewrites make purpose, inputs, boundaries, and returns easier for an agent to understand.
| Tool | Defect types | Suggested rewrite |
|---|---|---|
wopee_send_chat_message |
no_return_description |
Send a message to the current project's chat room. Use this to post status updates (e.g., 'Test run started...', 'Analysis complete') or informational messages to the chat. The message will appear as a SYSTEM message in the chat room. Requires WOPEE_PROJECT_UUID to be configured. Returns a confirmation once the message has been posted to the chat room. |
wopee_create_github_issue |
no_return_description |
Create a new GitHub issue in the project's connected repository. Use this to report bugs found during testing, track test failures, or create action items from chat discussions. The issue will be created in the GitHub repository linked to the current project. Requires the project to have GitHub integration configured and WOPEE_PROJECT_UUID to be set. Returns the created issue's number and URL on success. |
Selection evidence
Confusable tool pairs.
12 pairs where similar names or overlapping descriptions may send an agent toward the wrong tool.
| Tool A | Tool B | Confidence | Why they collide |
|---|---|---|---|
wopee_fetch_variables |
wopee_update_variables |
medium | Both share the same level/suiteUuid schema and cover the same variable set; a task like 'set the BASE_URL variable' could be misrouted to fetch if the agent conflates 'check/set variables' phrasing, though the verbs read/write are fairly distinct. |
wopee_dispatch_analysis |
wopee_dispatch_agent |
medium | Both 'dispatch' an AI agent against a web app; a vague request like 'run the agent on this suite' could be sent to dispatch_analysis (crawl) instead of dispatch_agent (execute test cases), especially since dispatch_analysis's description explicitly discusses agent crawling. |
wopee_fetch_artifact |
wopee_generate_artifact |
medium | A request like 'get me the user stories for this suite' could ambiguously mean either retrieve existing (fetch) or produce new ones via AI (generate) if the user doesn't know whether artifacts already exist. |
wopee_fetch_artifact |
wopee_update_artifact |
low | Both operate on the same artifact types/suite but fetch is clearly read-only and update requires content, so confusion is unlikely except in vague 'edit the app context' requests where the agent might fetch first mistakenly thinking it's the whole task. |
wopee_fetch_executed_test_cases |
wopee_fetch_test_inventory |
high | Both return test case status lists; a question like 'how many tests do I have' or 'show me my test cases' could be routed to fetch_executed_test_cases instead of the authoritative fetch_test_inventory, especially since one only returns already-run cases while the other covers all including NOT_RUN — the distinction is subtle and easy to miss. |
wopee_fetch_variables |
wopee_fetch_artifact |
low | Both are read-only fetch tools scoped by suiteUuid, but 'variables' vs 'artifact' are semantically distinct enough that confusion is unlikely except in very generic 'get suite data' requests. |
wopee_update_artifact |
wopee_generate_artifact |
medium | A task like 'create the app context for this suite' is ambiguous between AI-generation (generate_artifact) and manual content upload (update_artifact), and the descriptions explicitly cross-reference each other suggesting real overlap risk. |
wopee_update_variables |
wopee_update_artifact |
low | Both are 'update' write operations on suite-scoped data, but variables vs artifact content are distinct enough concepts (config values vs test content) that confusion is unlikely for a reasonably specific task. |
wopee_send_chat_message |
wopee_read_chat_history |
low | Send vs read are clearly opposite operations; confusion is unlikely except in a poorly specified 'update the chat' request, but the verbs are unambiguous. |
wopee_fetch_analysis_suites |
wopee_dispatch_analysis |
low | Both relate to 'analysis suites' but one lists existing suites (read-only) and the other creates+crawls a new one; a request like 'start an analysis' is unlikely to be confused with 'list analyses'. |
wopee_fetch_recent_executions |
wopee_fetch_test_inventory |
medium | A status question like 'what tests have run and what's left' could plausibly be routed to fetch_recent_executions (only ran tests, execution-focused) instead of fetch_test_inventory (full authoritative list including NOT_RUN), since both describe test status reporting. |
wopee_fetch_executed_test_cases |
wopee_fetch_recent_executions |
medium | Both return execution results with status/agent reports for test cases; 'check how the tests went' could go to either since one is project-wide recent executions and the other is suite-scoped executed cases, and the boundary isn't obvious without a suite UUID in hand. |
Compare the field