01Safety
0.0 / 30
What changed in the harness
Selection accuracy 100→100, token cost up 17%, unconfirmed writes 0%→0%.
Category breakdown
Where the score comes from.
Earned points across the four signals Gradable measures. Safety and Legibility are scored out of 30; Economics and Discoverability are scored out of 20.
02Legibility
21.7 / 30
03Economics
19.7 / 20
04Discoverability
11.1 / 20
Highest-impact fix
Estimated gain +30 pointsAdd explicit identity and permission preflight tools
Expose machine-readable principal/tenant confirmation and a non-mutating permission check so agents can verify both before destructive actions.
Description evidence
Defects and rewrites.
18 defects found across the exposed tool descriptions. Suggested rewrites make purpose, inputs, boundaries, and returns easier for an agent to understand.
| Tool | Defect types | Suggested rewrite |
|---|---|---|
set_projects_base_dir |
name_restates_behavior no_return_description |
Configures the root directory under which the server stores and looks up Xcode projects, affecting where later project operations search. Returns a confirmation that the base directory was set, or an error if the path is invalid. |
set_project_path |
name_restates_behavior no_return_description |
Switches the active Xcode project used for subsequent operations to the project at the given .xcodeproj path. Returns a confirmation that the project was activated, or an error if the path is invalid. |
get_active_project |
no_return_description |
Returns detailed information about the currently active Xcode project so callers can confirm which project later operations will target. |
read_file |
name_restates_behavior no_return_description |
Returns the contents of the file at the given path within the active Xcode project, allowing callers to inspect project sources before editing. Accepts a path relative to the project or an absolute path. |
write_file |
no_return_description |
Writes the provided content to a file in the active Xcode project, creating it first when createIfMissing is true and the file does not yet exist. Returns a confirmation that the file was written, or an error if the write failed. |
list_project_files |
name_restates_behavior no_return_description |
Returns the list of files contained in the Xcode project at the given .xcodeproj path, optionally filtered to those matching the specified fileType extension, so callers can discover the project's source files. |
analyze_file |
name_restates_behavior no_return_description |
Runs Xcode's static analyzer on the source file at the given path to detect potential bugs and issues before building. Returns the analysis results, including any warnings or errors found. |
build_project |
name_restates_behavior no_return_description |
Compiles the active Xcode project with the given configuration and scheme, surfacing any build errors. Returns the build result, including warnings and errors, so callers know whether the build succeeded. |
run_tests |
no_return_description |
Runs the tests of the active Xcode project, optionally limited to the named test plan. Returns the test results, including passed, failed, and skipped tests. |
run_xcrun |
no_return_description |
Invokes the named Xcode tool via xcrun, forwarding the optional arguments. Returns the tool's standard output and exit status so callers can act on the result. |
list_simulators |
name_restates_behavior no_return_description |
Returns all iOS simulators available on this machine with their details, including UDIDs and states, by invoking xcrun simctl list --json. Use the returned UDIDs with the simulator boot and shutdown operations. |
boot_simulator |
name_restates_behavior no_return_description |
Starts the iOS simulator with the given UDID so it is running and available for app installs and launches. Returns a confirmation that the simulator booted, or an error if it could not be booted. |
shutdown_simulator |
no_return_description |
Stops the active iOS simulator with the given UDID. Returns a confirmation that the simulator was shut down, or an error if the shutdown failed. |
compile_asset_catalog |
name_restates_behavior no_return_description |
Compiles the asset catalog at catalogPath into compiled assets saved to outputDir, ready to be bundled into an app. Returns the result of the compilation, including any errors, so callers know whether it succeeded. |
run_lldb |
params_unexplained no_return_description |
Starts the LLDB debugger, passing the optional lldbArgs, which are command-line flags and options that control the debugging session (such as attaching to a process or executing commands). Returns the debugger's output and session result. |
trace_app |
no_return_description |
Records a performance trace of the application binary at appPath for the specified duration in seconds using xctrace, so callers can analyze runtime behavior. Returns the trace data or the location of the generated trace. |
swift_package_update |
name_restates_behavior no_return_description |
Refreshes the dependencies of the current Swift package to their latest allowed versions by invoking swift package update. Returns the update output, indicating which packages were updated and any errors. |
list_directory |
name_restates_behavior no_return_description |
Returns the files and subdirectories contained in the directory at the given path, so callers can explore the filesystem. |
Selection evidence
Confusable tool pairs.
3 pairs where similar names or overlapping descriptions may send an agent toward the wrong tool.
| Tool A | Tool B | Confidence | Why they collide |
|---|---|---|---|
set_projects_base_dir |
set_project_path |
medium | Both tools configure a project location and take only a path/directory argument. A natural task like "set the directory for my Xcode projects" or "update the project path settings" is ambiguous about whether the user means the container base directory for all projects or the specific .xcodeproj path, so the agent could pick the wrong one. |
read_file |
analyze_file |
medium | Both operate on a single file inside the project. Vague task verbs like "inspect", "review", or "check this file" do not reveal whether the user wants the file contents (read_file) or a static-analysis pass for issues (analyze_file), so an agent reading only the descriptions could select the wrong tool. |
run_xcrun |
run_lldb |
medium | Both are generic executors with overlapping "run" semantics. A task like "run LLDB" or "execute the debugger" could be satisfied either by run_lldb directly or by run_xcrun with tool="lldb", since lldb is an Xcode command-line tool; the agent cannot reliably disambiguate which wrapper the user intends without more context. |
Compare the field