01Safety
0.0 / 30
What changed in the harness
Selection accuracy 98→98, token cost up 11%, unconfirmed writes 0%→0%.
Category breakdown
Where the score comes from.
Earned points across the four signals Gradable measures. Safety and Legibility are scored out of 30; Economics and Discoverability are scored out of 20.
02Legibility
21.1 / 30
03Economics
19.6 / 20
04Discoverability
9.6 / 20
Highest-impact fix
Estimated gain +30 pointsAdd explicit identity and permission preflight tools
Expose machine-readable principal/tenant confirmation and a non-mutating permission check so agents can verify both before destructive actions.
Description evidence
Defects and rewrites.
18 defects found across the exposed tool descriptions. Suggested rewrites make purpose, inputs, boundaries, and returns easier for an agent to understand.
| Tool | Defect types | Suggested rewrite |
|---|---|---|
set_projects_base_dir |
name_restates_behavior no_return_description |
Configures the workspace by setting the absolute base directory that will be searched for Xcode projects in subsequent tool calls; returns confirmation of the directory that was set. |
set_project_path |
name_restates_behavior no_return_description |
Sets which Xcode project (.xcodeproj) subsequent tool calls will operate on; returns confirmation of the active project path once set. |
get_active_project |
no_return_description |
Returns metadata about the currently active Xcode project, such as its path, name, and available schemes/configuration details. |
read_file |
name_restates_behavior no_return_description |
Opens the specified file within the active Xcode project and returns its full text contents. |
write_file |
no_return_description |
Writes the given content to the specified file in the active Xcode project, optionally creating the file if it doesn't exist (via createIfMissing), and returns confirmation that the write succeeded. |
list_project_files |
name_restates_behavior no_return_description |
Returns the list of file paths contained in the specified Xcode project, optionally filtered to files matching the given extension. |
analyze_file |
name_restates_behavior no_return_description |
Runs Xcode's static analyzer against the specified source file and returns the list of issues or warnings it detects. |
build_project |
name_restates_behavior no_return_description |
Runs an Xcode build for the given scheme and configuration, returning the build result (success or failure) along with any compiler or linker output. |
run_tests |
no_return_description |
Runs the test suite for the active Xcode project, optionally scoped to a named test plan, and returns the test results, including pass/fail counts and failure details. |
run_xcrun |
no_return_description |
Executes the named Xcode command-line tool via 'xcrun', passing any optional argument string, and returns the tool's stdout/stderr output. |
list_simulators |
name_restates_behavior no_return_description |
Returns the full list of available iOS simulators and their details (name, UDID, state, runtime) by invoking 'xcrun simctl list --json'. |
boot_simulator |
name_restates_behavior no_return_description |
Boots the iOS simulator identified by the given UDID and returns confirmation once the simulator has started. |
shutdown_simulator |
no_return_description |
Shuts down the iOS simulator identified by the given UDID and returns confirmation once it has stopped. |
compile_asset_catalog |
name_restates_behavior no_return_description |
Compiles the asset catalog at the given path into the specified output directory using 'actool', and returns the compilation result and any tool output. |
run_lldb |
params_unexplained no_return_description |
Launches the LLDB debugger, optionally passing a raw command-line argument string (e.g. target executable, breakpoints, or LLDB commands) via lldbArgs to configure the session, and returns the debugger's output. |
trace_app |
no_return_description |
Runs 'xctrace' to capture a performance trace of the application at the given path for an optional duration in seconds, and returns the resulting trace file's location or output. |
swift_package_update |
name_restates_behavior no_return_description |
Runs 'swift package update' to resolve and update Swift Package Manager dependencies to their latest allowed versions, and returns the command's output. |
list_directory |
name_restates_behavior no_return_description |
Returns the names of files and subdirectories contained in the specified directory path. |
Selection evidence
Confusable tool pairs.
6 pairs where similar names or overlapping descriptions may send an agent toward the wrong tool.
| Tool A | Tool B | Confidence | Why they collide |
|---|---|---|---|
boot_simulator |
shutdown_simulator |
medium | Both take only a udid and act on simulator power state; a vague instruction like 'toggle the simulator' or misremembering which verb corresponds to starting vs stopping could lead the agent to pick the wrong one. |
read_file |
write_file |
low | Clearly opposite operations (read vs write) with distinct required params (content for write); genuine confusion is unlikely except in trivial slip-ups, not selection ambiguity. |
set_projects_base_dir |
set_project_path |
medium | Both 'set' project-related paths; a task like 'set the project directory to X' is ambiguous between setting the base dir for all projects vs setting the active .xcodeproj path, especially if the agent doesn't parse the distinction between a folder and a .xcodeproj path. |
read_file |
analyze_file |
low | A request like 'check this file for issues' could momentarily be confused with 'read the file', but analyze_file's static-analyzer framing is fairly distinct from a generic read. |
list_project_files |
analyze_file |
low | Both operate on project files but serve very different purposes (listing vs static analysis); unlikely to be genuinely confused. |
set_project_path |
get_active_project |
low | A task like 'what's the active project' vs 'set the active project' are distinguishable by verb intent; low but nonzero risk if phrasing is ambiguous like 'use this project'. |
Compare the field