01Safety
0.0 / 30
What changed in the harness
Selection accuracy 100→100, token cost up 2%, unconfirmed writes 0%→0%.
Category breakdown
Where the score comes from.
Earned points across the four signals Gradable measures. Safety and Legibility are scored out of 30; Economics and Discoverability are scored out of 20.
02Legibility
27.1 / 30
03Economics
18.4 / 20
04Discoverability
12.2 / 20
Highest-impact fix
Estimated gain +30 pointsAdd explicit identity and permission preflight tools
Expose machine-readable principal/tenant confirmation and a non-mutating permission check so agents can verify both before destructive actions.
Description evidence
Defects and rewrites.
5 defects found across the exposed tool descriptions. Suggested rewrites make purpose, inputs, boundaries, and returns easier for an agent to understand.
| Tool | Defect types | Suggested rewrite |
|---|---|---|
fetch_swagger |
no_return_description |
Fetch and parse an OpenAPI/Swagger specification from a public URL, internal/VPN-accessible URL, or local file, optionally using auth_header or extra headers for protected specs. Caches the parsed spec and returns a summary (title, version, endpoints, tags) for use by other tools in this server. |
generate_test_cases_batch |
name_restates_behavior no_return_description |
Run test-case generation across many endpoints of a loaded Swagger spec in one call, optionally narrowed by tag, HTTP methods, or path prefix, using the same category/priority/locale config as generate_test_cases. Returns the generated test cases grouped by endpoint, and caches the result as the 'last batch' for export. |
export_test_cases |
no_return_description |
Export previously generated test cases for a given endpoint key (or "_last_batch" for the most recent batch generation) into markdown, json, csv, allure_csv, gherkin, postman, k6, pytest, testrail_csv, or testrail_csv_text format, saving to output_path or an auto-generated filename. Returns the path of the written export file. |
compare_specs |
no_return_description |
Compare two OpenAPI spec versions, fetched from old_source and new_source (with optional auth_header/headers), to detect added, removed, and modified endpoints, schema changes, and breaking changes. Returns a structured diff report listing each detected change. |
clear_cache |
no_return_description |
Clear in-memory cached data identified by target ('specs', 'test_cases', or 'all'), forcing subsequent calls to re-fetch the spec or regenerate test cases. Returns a confirmation of what was cleared. |
Selection evidence
Confusable tool pairs.
8 pairs where similar names or overlapping descriptions may send an agent toward the wrong tool.
| Tool A | Tool B | Confidence | Why they collide |
|---|---|---|---|
generate_test_cases |
generate_test_cases_batch |
high | Both generate QA test cases from a spec; a request like 'generate test cases for the orders endpoints' is ambiguous between a single endpoint (with path/method) and a batch call filtered by path_prefix/tag, especially since generate_test_cases_batch can also target one endpoint via path_prefix. |
generate_test_cases |
export_test_cases |
low | Names overlap on 'test_cases' but purposes are distinct (create vs. export existing); confusion is unlikely unless the user vaguely says 'get me test cases as CSV' without realizing generation must happen first, which a careful agent would sequence rather than pick the wrong tool. |
generate_test_cases_batch |
export_test_cases |
low | export_test_cases requires an already-generated endpoint key or '_last_batch', so an agent could momentarily consider it for a batch export request, but the clear required-field differences (source vs endpoint/format) make actual tool confusion unlikely. |
compare_specs |
validate_spec |
low | Both operate on OpenAPI specs and produce quality/diff assessments, but compare_specs needs two sources (old/new) while validate_spec needs one; a task like 'check my spec for issues' is unambiguous, so overlap is mainly lexical. |
suggest_missing_tests |
clear_cache |
low | Functionally unrelated (coverage analysis vs cache clearing); despite shared tokens in descriptions, no plausible natural-language request would cause selection confusion between them. |
export_test_cases |
suggest_missing_tests |
low | Both reference generated test cases but serve different purposes (export format vs coverage gap analysis); a phrase like 'show me what's missing in my tests' clearly maps to suggest_missing_tests, not export. |
generate_test_cases |
generate_mock_data |
medium | Both take source/path/method for a single endpoint and 'generate' something related to testing; a request like 'generate sample test data for this endpoint' could be misrouted to generate_test_cases (which makes QA test cases) instead of generate_mock_data (which makes sample request/response payloads). |
generate_test_cases |
suggest_missing_tests |
low | suggest_missing_tests analyzes coverage of already-generated cases rather than creating new ones; a vague 'what tests do I need' could momentarily suggest either, but required params and descriptions clearly differentiate creation vs. gap-analysis. |
Compare the field