01Safety
0.0 / 30
What changed in the harness
Selection accuracy 100→100, token cost down 1%, unconfirmed writes 0%→0%.
Category breakdown
Where the score comes from.
Earned points across the four signals Gradable measures. Safety and Legibility are scored out of 30; Economics and Discoverability are scored out of 20.
02Legibility
25.5 / 30
03Economics
16.1 / 20
04Discoverability
8.6 / 20
Highest-impact fix
Estimated gain +30 pointsAdd explicit identity and permission preflight tools
Expose machine-readable principal/tenant confirmation and a non-mutating permission check so agents can verify both before destructive actions.
Description evidence
Defects and rewrites.
12 defects found across the exposed tool descriptions. Suggested rewrites make purpose, inputs, boundaries, and returns easier for an agent to understand.
| Tool | Defect types | Suggested rewrite |
|---|---|---|
read_file |
name_restates_behavior params_unexplained |
Reads a file's full contents as UTF-8 text, given a file path within an allowed directory; optionally returns only the first N lines via 'head' or the last N lines via 'tail'. Returns the file content as a string. Deprecated — use read_text_file instead. |
read_text_file |
params_unexplained |
Reads the complete contents of the text file at the given file path (must be within an allowed directory), handling various text encodings and returning detailed error messages if the file cannot be read. Optionally use 'head' to return only the first N lines or 'tail' for the last N lines. Returns the file content as a string. |
read_media_file |
params_unexplained |
Reads the file at the given path (must be within an allowed directory) and returns it as base64-encoded content with its MIME type: image and audio files are returned as image/audio content blocks, and any other file type is returned as an embedded resource. |
write_file |
params_unexplained |
Creates a new file or completely overwrites an existing file at the given path (within an allowed directory) with the given text 'content', using proper encoding. Use with caution as it overwrites existing files without warning. Returns a confirmation message. |
edit_file |
params_unexplained |
Applies line-based edits to the text file at the given path (within an allowed directory): each entry in 'edits' replaces an exact 'oldText' line sequence with 'newText'; set 'dryRun' to preview without writing. Returns a git-style diff of the changes made. |
create_directory |
params_unexplained |
Creates a new directory, including any nested parent directories, at the given path within an allowed directory; succeeds silently if the directory already exists. Returns a confirmation message. |
list_directory |
params_unexplained |
Lists the files and directories directly within the given path (an allowed directory), prefixing each entry with [FILE] or [DIR]. Returns the listing as text. |
list_directory_with_sizes |
params_unexplained |
Lists the files and directories directly within the given path (an allowed directory), including file sizes and [FILE]/[DIR] prefixes; use 'sortBy' to order entries by name or size. Returns the listing as text. |
directory_tree |
params_unexplained |
Recursively lists files and directories starting at the given path (an allowed directory), optionally skipping entries matching any glob in 'excludePatterns'. Returns a JSON string where each entry has 'name', 'type' (file/directory), and, for directories, a 'children' array. |
move_file |
params_unexplained |
Moves or renames a file or directory from 'source' to 'destination' (both paths must be within allowed directories); fails if the destination already exists. Returns a confirmation message. |
search_files |
params_unexplained |
Recursively searches under the given 'path' (an allowed directory) for files and directories whose paths match the glob-style 'pattern' (e.g. '*.ext' or '**/*.ext'), optionally skipping paths matching any glob in 'excludePatterns'. Returns the full matching paths as text. |
get_file_info |
params_unexplained |
Retrieves metadata — size, creation time, last modified time, permissions, and type — for the file or directory at the given path within an allowed directory. Returns this metadata as text. |
Selection evidence
Confusable tool pairs.
7 pairs where similar names or overlapping descriptions may send an agent toward the wrong tool.
| Tool A | Tool B | Confidence | Why they collide |
|---|---|---|---|
read_file |
read_text_file |
high | read_file's description explicitly says it is deprecated in favor of read_text_file and both have identical schemas and purpose, so an agent could easily pick either for a simple 'read this file' task. |
list_directory |
list_directory_with_sizes |
medium | Both list files/dirs with identical [FILE]/[DIR] formatting; a task like 'show me what's in this folder' doesn't clearly indicate whether size info is needed, so the agent may pick either. |
read_file |
read_multiple_files |
low | A task naming several files could plausibly be handled by looping read_file or using read_multiple_files, but the plural naming and array schema make the correct choice fairly clear. |
read_text_file |
read_multiple_files |
low | Same reasoning as read_file/read_multiple_files: a request to read several files could ambiguously map to either repeated single-file reads or the batch tool, though schema differences reduce real confusion. |
read_file |
read_media_file |
low | For binary/image files an agent might mistakenly try read_file (or read_text_file) instead of read_media_file if the task doesn't clearly signal the file is non-text, though descriptions differentiate text vs base64/media fairly well. |
list_directory |
directory_tree |
low | A vague request like 'show me the project structure' could be served by either a flat listing or a recursive tree, causing some ambiguity about depth expectations. |
edit_file |
move_file |
low | Both can be invoked for 'update this file's location/content' style tasks if the request is vague about renaming vs modifying content, though their schemas (source/destination vs path/edits) are quite distinct. |
Compare the field