01Safety
0.0 / 30
What changed in the harness
Selection accuracy 100→98, token cost up 5%, unconfirmed writes 0%→0%.
Category breakdown
Where the score comes from.
Earned points across the four signals Gradable measures. Safety and Legibility are scored out of 30; Economics and Discoverability are scored out of 20.
02Legibility
27.1 / 30
03Economics
19.9 / 20
04Discoverability
10.4 / 20
Highest-impact fix
Estimated gain +30 pointsAdd explicit identity and permission preflight tools
Expose machine-readable principal/tenant confirmation and a non-mutating permission check so agents can verify both before destructive actions.
Description evidence
Defects and rewrites.
7 defects found across the exposed tool descriptions. Suggested rewrites make purpose, inputs, boundaries, and returns easier for an agent to understand.
| Tool | Defect types | Suggested rewrite |
|---|---|---|
create_diff |
no_return_description |
Modify an existing file by showing a diff and getting user approval before applying the changes. Returns the diff result and whether the user approved and the changes were applied. Use only on existing files; do not use for new files. |
open_file |
no_return_description |
Open a file in the VS Code editor so the user can see it. Use this after creating a brand new file or after applying changes with create_diff. Returns confirmation that the file was opened and revealed in the editor. |
open_project |
no_return_description |
Open a project folder in VS Code so the AI agent's current active working directory is set and visible. Call this as soon as a new session begins. Returns confirmation that the project was opened. |
check_extension_status |
no_return_description |
Check whether the VS Code MCP Extension is installed and responding. Returns the extension's status, indicating whether it is installed and reachable, so you can decide whether retries or setup are needed. |
get_extension_port |
name_restates_behavior no_return_description |
Return the TCP port number that the VS Code MCP Extension is running on, so the AI agent can connect to it directly. Use after verifying the extension is installed and responding. |
list_available_projects |
no_return_description |
List all available projects from the port registry file so the user can choose which project to work with. Returns the set of registered projects, including their identifiers and port information. |
get_context_tabs |
no_return_description |
Retrieve information about tabs that have been specifically marked for AI context inclusion using the VS Code UI toggle. Returns the marked tabs, optionally including their file content or the specified line ranges, so the agent can use only the relevant context. |
Selection evidence
Confusable tool pairs.
4 pairs where similar names or overlapping descriptions may send an agent toward the wrong tool.
| Tool A | Tool B | Confidence | Why they collide |
|---|---|---|---|
get_active_tabs |
get_context_tabs |
high | Both descriptions frame themselves as retrieving tab info 'to provide context for the AI agent'; a task like 'get the tabs for my context' or 'which tabs are active for AI context' could select get_active_tabs even when the user means specifically marked context tabs. |
check_extension_status |
get_extension_port |
medium | A task like 'check the MCP extension' or 'make sure the extension is running/responding' overlaps both; diagnosing a connection issue could plausibly route to get_extension_port (port info) instead of check_extension_status, since both only share the notion of checking the extension. |
open_file |
open_project |
low | Descriptions clearly separate opening a file vs. a project folder, but a task like 'open the project file in VS Code' or 'open up what we're working on' could land on the wrong one since both start with the same purpose verb. |
open_project |
list_available_projects |
low | A task like 'open a project' without a specific path might cause selection of open_project over listing options first, or 'show me projects to open' could pick open_project; the overlap is mostly the word 'project' and the shared setup/selection workflow. |
Compare the field