01Safety
0.0 / 30
What changed in the harness
Selection accuracy 100→100, token cost down 0%, unconfirmed writes 0%→0%.
Category breakdown
Where the score comes from.
Earned points across the four signals Gradable measures. Safety and Legibility are scored out of 30; Economics and Discoverability are scored out of 20.
02Legibility
27.7 / 30
03Economics
18.6 / 20
04Discoverability
11.8 / 20
Highest-impact fix
Estimated gain +30 pointsAdd explicit identity and permission preflight tools
Expose machine-readable principal/tenant confirmation and a non-mutating permission check so agents can verify both before destructive actions.
Description evidence
Defects and rewrites.
3 defects found across the exposed tool descriptions. Suggested rewrites make purpose, inputs, boundaries, and returns easier for an agent to understand.
| Tool | Defect types | Suggested rewrite |
|---|---|---|
fetch_swagger |
no_return_description |
Fetch and parse an OpenAPI/Swagger specification and returns the parsed spec object for use by other tools in this server. Supports: Public URLs (https://api.example.com/swagger.json), Internal URLs accessible from your machine/VPN (https://internal-api.company.com/docs/swagger.json), Local files (/path/to/swagger.json, ./api-spec.yaml), and Auth-protected specs via the auth_header parameter for Bearer/Basic/API-Key authentication. |
generate_test_cases_batch |
name_restates_behavior no_return_description |
Generate QA test cases for multiple endpoints in a single call, then returns the generated test cases (or a confirmation for later export). Restrict scope with the optional filters: tag (filter endpoints by OpenAPI tag), methods (only listed HTTP methods, e.g. ['POST', 'PUT']), and path_prefix (only paths starting with a prefix, e.g. '/api/v2'). When no filters are given, all endpoints in the spec are processed. |
export_test_cases |
no_return_description |
Export previously generated test cases in various formats and returns the export result (generated artifact, typically saved to output_path or to the working directory). Supported formats: markdown, json, csv, allure_csv, gherkin, postman, k6, pytest, testrail_csv (Steps template, one row per step), and testrail_csv_text (Text template, all steps in one field). |
Selection evidence
Confusable tool pairs.
6 pairs where similar names or overlapping descriptions may send an agent toward the wrong tool.
| Tool A | Tool B | Confidence | Why they collide |
|---|---|---|---|
generate_test_cases |
generate_test_cases_batch |
high | Near-identical names and descriptions (both generate QA test cases from the same OpenAPI spec). A task like 'generate test cases for my API' or 'create tests for the endpoints' does not clarify single-endpoint (path+method) vs multiple (tag/path_prefix/methods), so an agent can plausibly pick the wrong one. |
generate_test_cases |
export_test_cases |
medium | Both surface 'test cases'; a task like 'get me the test cases for POST /api/orders' or 'produce test cases in markdown' could route to export (returns previously generated cases in a format) instead of generate (creates fresh cases), or vice versa, when 'generate now' vs 'retrieve/format existing' is not stated. |
generate_test_cases_batch |
export_test_cases |
low | A task like 'get the test cases for all my endpoints' could point to batch generation or to exporting already-generated cases ('_last_batch'); an agent may call the wrong one since export cannot create cases and batch cannot format them, though 'generate' vs 'export' wording usually disambiguates. |
compare_specs |
validate_spec |
low | Both analyze an OpenAPI spec; a vague task such as 'check my API spec' or 'review spec.yaml' could be mapped to either quality/completeness validation or a two-version diff, though two sources clearly signal compare and quality/completeness language signals validate. |
generate_test_cases |
generate_mock_data |
medium | Both take source/path/method and 'generate' endpoint-specific output; a task like 'generate data to test POST /api/orders' or 'create dummy payloads and tests for the endpoint' is ambiguous between QA test cases and mock/sample data, especially when 'test' is used loosely. |
generate_test_cases |
suggest_missing_tests |
low | A request like 'what test cases should I have for my endpoint' or 'add the missing tests' could be routed to generate (creates cases) or suggest_missing_tests (analyzes coverage and lists suggestions without generating), so an agent may pick the analysis tool when generation was implied. |
Compare the field