01Safety
0.0 / 30
What changed in the harness
Selection accuracy 100→100, token cost up 2%, unconfirmed writes 0%→0%.
Category breakdown
Where the score comes from.
Earned points across the four signals Gradable measures. Safety and Legibility are scored out of 30; Economics and Discoverability are scored out of 20.
02Legibility
26.3 / 30
03Economics
14.8 / 20
04Discoverability
11.2 / 20
Highest-impact fix
Estimated gain +30 pointsAdd explicit identity and permission preflight tools
Expose machine-readable principal/tenant confirmation and a non-mutating permission check so agents can verify both before destructive actions.
Description evidence
Defects and rewrites.
9 defects found across the exposed tool descriptions. Suggested rewrites make purpose, inputs, boundaries, and returns easier for an agent to understand.
| Tool | Defect types | Suggested rewrite |
|---|---|---|
sandbox_kill |
no_return_description |
Permanently terminate and destroy a sandbox, including all files, processes, and state. This is irreversible — all data in the sandbox is lost. Use sandbox_pause instead if you want to preserve state for later. To save files before killing, use sandbox_download_url or sandbox_file_read first. Fails if the sandbox ID is not found; on success returns a confirmation that the sandbox was terminated. After killing, the sandboxId is no longer valid for any operations. |
sandbox_pause |
no_return_description |
Pause an active sandbox to persist its full state (filesystem, installed packages, configuration) for later resumption. The sandbox is removed from the active registry and its timeout stops counting. Resume it later with sandbox_resume using the same sandboxId — even across different sessions. Unlike sandbox_kill (which destroys everything), pause preserves state. Running processes may not survive pause/resume. Use this for long-running development sessions where you want to continue later. Fails if the sandbox ID is not found; on success returns a confirmation with the sandboxId so you can resume it later with sandbox_resume. |
sandbox_resume |
no_return_description |
Resume a previously paused sandbox, restoring its filesystem and state. The sandbox becomes active again with a new timeout and is re-added to the active registry. If GITHUB_TOKEN is configured, git authentication is automatically re-established. Only works on sandboxes paused with sandbox_pause — not on killed or expired sandboxes. After resuming, use all other sandbox_* tools normally. Fails if the sandboxId does not correspond to a paused sandbox; on success returns a confirmation with the sandboxId and its new timeout duration. |
sandbox_file_mkdir |
no_return_description |
Create a directory (and any missing parent directories) in a sandbox. Succeeds silently if the directory already exists. Use this to set up directory structure before writing files with sandbox_file_write or before uploading via sandbox_upload_url. Unlike sandbox_file_write (which creates files), this creates empty directories. Unlike sandbox_file_remove (which deletes), this only creates. Returns a confirmation that the directory exists at the requested path. |
sandbox_git_clone |
no_return_description |
Clone a git repository into a sandbox, downloading its files and history. This is typically the first step after sandbox_create — clone a repo, then use sandbox_exec to build/test. Creates a new directory at the destination path containing the repository. Supports both public and private repos (private repos require GITHUB_TOKEN to be configured). Default shallow clone (depth=1) downloads only the latest commit for faster cloning. Modifies the sandbox filesystem by creating the clone directory and writing all repo files. Fails if the destination path already exists or the repo URL is invalid. Unlike sandbox_git_pull (which updates an existing repo), this performs the initial download. Returns the path to the cloned repository for use with later sandbox_exec and sandbox_git_* calls. |
sandbox_git_commit |
no_return_description |
Stage files and create a git commit in a sandbox repository. This is a two-step operation: first stages the specified files (or all changes if none specified), then creates a commit with the given message. Modifies the git history — the commit is added to the current branch. Use sandbox_git_status first to see what changes are available to commit. After committing, use sandbox_git_push to upload to the remote. Fails if there are no changes to commit or if the path is not a git repository. Unlike sandbox_git_push (which uploads commits), this only creates a local commit. Returns the commit hash of the newly created commit. |
sandbox_git_push |
no_return_description |
Push local commits to the remote repository. This is a destructive external operation — commits become visible to others and cannot easily be undone. Requires GITHUB_TOKEN to be configured at server startup; fails immediately with a clear error if not set. Use sandbox_git_commit first to create local commits, then push. For new branches, set setUpstream to true. Unlike sandbox_git_pull (which downloads remote changes) or sandbox_git_commit (which only creates local commits), this uploads commits to the remote. Fails if there are no commits to push or the remote is unreachable; on success returns a confirmation with the remote and branch that were pushed. |
sandbox_git_branch |
no_return_description |
Manage git branches in a sandbox repository: list existing branches, create a new branch, or switch to a different branch. The "list" action is read-only. The "create" action creates a new branch from the current HEAD but does not switch to it. The "checkout" action switches the working directory to the specified branch, modifying files in place — any uncommitted changes may conflict. Requires branchName for "create" and "checkout" actions. Unlike sandbox_git_status (which shows current branch and file changes), this manages branch lifecycle. Unlike sandbox_git_commit or sandbox_git_push, this does not affect commit history. Returns the list of local branches for "list", or a confirmation of the resulting branch for "create" and "checkout". |
sandbox_git_init |
no_return_description |
Initialize a new empty git repository in a sandbox directory. Creates a .git directory at the specified path, enabling git operations (commit, branch, etc.) on files in that directory. Use this when starting a new project from scratch — not needed if you used sandbox_git_clone (which already initializes git). The directory must already exist; use sandbox_file_mkdir first if needed. Does not create any initial commit. Unlike sandbox_git_clone (which downloads an existing repo), this creates a fresh, empty repository. Returns a confirmation with the path where the repository was initialized. |
Selection evidence
Confusable tool pairs.
6 pairs where similar names or overlapping descriptions may send an agent toward the wrong tool.
| Tool A | Tool B | Confidence | Why they collide |
|---|---|---|---|
sandbox_exec |
sandbox_exec_background |
medium | Natural tasks like 'run this command' or 'start the application' don't specify blocking vs background; an agent could pick sandbox_exec for a long-running dev server (hanging until timeout) or sandbox_exec_background for a one-shot build (never seeing output). Only the descriptions' blocking/streaming contrast disambiguates. |
sandbox_kill |
sandbox_process_kill |
medium | A task worded as 'stop/terminate the running job in my sandbox' is ambiguous: sandbox_kill destroys the entire sandbox while sandbox_process_kill stops one process, and both names read identically as 'kill sandbox process'. The destructive-cost difference makes real confusion plausible. |
sandbox_upload_url |
sandbox_download_url |
medium | Tasks phrased as 'generate a URL to transfer this file to/from the sandbox' are directionally ambiguous; only the descriptions' upload-vs-download contrast (and the 'unlike' references) reveal which member is correct, so an agent skimming names could pick the wrong direction. |
sandbox_get_url |
sandbox_download_url |
medium | Both tools 'get a URL for the sandbox,' one for a live service port and one for downloading a file. A task like 'give me a link to access the app/file' could route to either; the distinction (running HTTP service vs presigned file download) is only clear in the descriptions. |
sandbox_keep_alive |
sandbox_pause |
medium | Both are viable responses to 'make sure my sandbox doesn't expire/destroy my work': keep_alive extends the timeout while pause persists state indefinitely. A task about preserving a session for later could plausibly select either, since both prevent auto-termination. |
sandbox_git_clone |
sandbox_git_init |
low | For a task like 'set up a git repository for this new project,' an agent could pick sandbox_git_clone (needs a remote URL) instead of sandbox_git_init (fresh empty repo). The descriptions contrast cleanly (download existing vs init empty), but the natural task leaves the choice open. |
Compare the field