Public assessment

jiezeng2004-design/PatchWarden (patchwarden)

patchwarden · v1.7.2 · scanned Aug 3, 2026

What changed in the harness

Selection accuracy 96→96, token cost up 7%, unconfirmed writes 0%→0%.

Category breakdown

Where the score comes from.

Earned points across the four signals Gradable measures. Safety and Legibility are scored out of 30; Economics and Discoverability are scored out of 20.

01Safety

0.0 / 30

0.0 out of 30

02Legibility

17.3 / 30

17.3 out of 30

03Economics

10.8 / 20

10.8 out of 20

04Discoverability

14.6 / 20

14.6 out of 20

Highest-impact fix

Estimated gain +30 points

Add explicit identity and permission preflight tools

Expose machine-readable principal/tenant confirmation and a non-mutating permission check so agents can verify both before destructive actions.

Description evidence

Defects and rewrites.

49 defects found across the exposed tool descriptions. Suggested rewrites make purpose, inputs, boundaries, and returns easier for an agent to understand.

Tool	Defect types	Suggested rewrite
`save_plan`	no_return_description	Store an execution plan PatchWarden will keep for local agent execution: supply the plan via content (Markdown, required unless plan_ref is given) or via plan_ref pointing to a file already inside .patchwarden/plans. Title defaults to 'Inline plan' or 'Plan from file' when omitted. Returns a plan_id usable with get_plan and create_task.
`get_plan`	no_return_description	Retrieve a previously saved plan by plan_id (the ID returned by save_plan). Returns the stored plan content so it can be reviewed or reused.
`health_check`	no_return_description	Verify PatchWarden readiness before task execution: check MCP catalog consistency, watcher freshness/supervisor state, workspace readiness, and configured agents (agent availability is executable-only unless an explicit provider probe is requested). Returns a per-area health report; detail=self_diagnostic instead returns expanded read-only evidence covering catalog, watcher, agent, allowlist, workspace, and recent failures.
`list_agents`	no_return_description	Report the configured local agents (from the active runtime config) along with executable and model-argument wiring verification and the config path this process used. Returns provider_status as not_checked because this read-only check never contacts a model provider or tests an account balance.
`export_task_evidence_pack`	no_return_description	Create a bounded evidence pack for a run_task_loop lineage by writing evidence.json and EVIDENCE.md, excluding stdout, stderr, full logs, full diffs, and sensitive file contents. Returns the location (paths) of the written evidence files so clients can find them.
`get_task_status`	no_return_description	Check the current state of a task created by create_task: returns task status, execution phase, watcher health, pending reason, current command, timeout, and change evidence.
`get_result`	params_unexplained	Read result.md for the task identified by task_id (the ID returned by create_task). Returns the task's result content, or structured availability and watcher evidence while the task is not terminal.
`get_result_json`	params_unexplained	Read the structured result.json for the task identified by task_id (the ID returned by create_task), returning the JSON content used for deterministic task acceptance.
`get_diff`	params_unexplained	Read task diff evidence for the task identified by task_id (the ID returned by create_task). Returns the task diff, or structured availability and watcher evidence while it is not ready.
`get_test_log`	params_unexplained	Read test.log for the task identified by task_id (the ID returned by create_task). Returns the test log, or structured availability and watcher evidence while it is not ready.
`list_workspace`	name_restates_behavior no_return_description	Discover which files PatchWarden can read: returns the files and directories under the given path (workspace root when path is omitted), excluding sensitive files such as secrets, keys, and tokens, so callers know what read_workspace_file can access.
`list_tasks`	no_return_description	Return recent tasks matching the optional status, repo_path, active_only, and history_state filters, including each task's status and repo plus watcher state and computed pending reasons. Archived history is excluded by default; pass history_state='archived' or 'all' to include it.
`cancel_task`	no_return_description	Request graceful cancellation of a task by task_id (the ID returned by create_task) and return the outcome of the request; the runner that owns the child process performs termination, and the MCP server never kills a PID read from task files.
`kill_task`	name_restates_behavior no_return_description	Immediately stop a pending or running task that needs hard termination rather than graceful cancellation, and return the outcome of the termination request; the runner validates and kills only the child process it owns, so the server never terminates arbitrary PIDs.
`retry_task`	no_return_description	Create a new task that reuses the existing task's plan, agent, repo_path, and test_command, leaving the original task unchanged. Returns the new task's identifier (as produced by create_task) so it can be polled with wait_for_task.
`get_task_stdout_tail`	params_unexplained	Read the last N lines of agent stdout/stderr for the task identified by task_id (the ID returned by create_task). Returns the tail (default 80, max 200 lines): reads the real-time stdout.log/stderr.log during execution and falls back to result.md after completion; works for pending, running, and completed tasks.
`get_task_log_tail`	params_unexplained	Read the last N lines of a task log file for the task identified by task_id (the ID returned by create_task). Returns only the tail (default 80, max 200 lines) of the chosen file (stdout/stderr/test/verify) with automatic secret redaction; use this instead of read_workspace_file to avoid triggering platform content filters on log output.
`get_task_progress`	params_unexplained no_return_description	Read progress.md for the task identified by task_id (the ID returned by create_task) and return its task phases along with the most recent heartbeat and current command.
`wait_for_task`	params_unexplained no_return_description	Long-poll a task (task_id from create_task) for up to 30 seconds and return the current state: if continuation_required is true, call wait_for_task again immediately without finishing the turn; wait_seconds (default 25, max 30, or its alias timeout_seconds) controls how long to poll. Terminal responses include get_task_summary acceptance evidence.
`get_task_summary`	params_unexplained	Return structured acceptance evidence for the task identified by task_id. Use view=compact first for bounded counts and risk excerpts; use standard only when full changed-file and log-tail detail is required. max_items caps entries per compact evidence group.
`explain_tool`	no_return_description	v0.7.1: Expand a single tool's metadata. Returns the tool's title, summary, risk level, tags, aliases, profiles, modes, schema digest, and, when includeSchema is true, the full inputSchema. Use after discover_tools to understand a specific tool before calling it. Read-only.
`invoke_discovered_tool`	no_return_description	v0.8.1: Invoke a previously discovered tool using a discoveryToken. The token must be obtained from discover_tools first. Enforces 10 security checks: token validity, toolName match, profile allowance, risk ceiling, sensitive path guard, assessment requirement, command whitelist, release confirmation, credential block, and invocation logging. Returns the invoked tool's result, or a security error if any check fails. Cannot call itself recursively.
`create_goal`	no_return_description	v0.8.0: Create a Goal Session for managing a multi-task objective with subgoal dependencies. Generates a structured directory under .patchwarden/goals/{goal_id}/ with GOAL.md, GOALS.md, and goal_status.json. Returns the created goal_id and session location. Use list_goals to enumerate existing goals and read_goal to inspect details.
`read_goal`	no_return_description	v0.8.0: Read full Goal Session details for goal_id. Returns GOAL.md content, goal_status.json, and all subgoals with dependency info. Use after list_goals to inspect a specific goal.
`create_subgoal_task`	params_unexplained	v0.8.0: Create a subgoal within the Goal Session identified by goal_id and immediately launch an associated task, executed by the agent named in agent (the runtime agent that runs the task). Atomically: addSubgoal → create_task → linkTask → mark subgoal running. The subgoal depends_on other subgoals (by id) which must be accepted before suggest_next_subgoal returns it.
`accept_subgoal`	params_unexplained no_return_description	Accept the subgoal identified by subgoal_id within the Goal Session identified by goal_id, after all associated tasks are accepted by audit_task. Supports both legacy status=accepted and done_by_agent with acceptance_status=accepted. Returns confirmation that the subgoal was accepted.
`reject_subgoal`	params_unexplained no_return_description	Reject the subgoal identified by subgoal_id within the Goal Session identified by goal_id, with the given reason. Allowed from any non-terminal status (ready/queued/running/done_by_agent/needs_fix). Records rejected_reason in goal_status.json. Returns confirmation that the rejection was recorded.
`suggest_next_subgoal`	params_unexplained	v0.8.0: Suggest the next executable subgoal for the Goal Session identified by goal_id, based on the dependency graph. Returns a ready subgoal whose dependencies are all accepted. If none ready, returns a blocked_by list. Use to drive goal-directed task sequencing.
`summarize_goal_progress`	params_unexplained no_return_description	Summarize completion of the Goal Session identified by goal_id. Returns counts by status (accepted/rejected/queued/running/ready/needs_fix/done_by_agent), completion_rate, blocked_subgoals, and actionable risks.
`export_handoff`	params_unexplained	v0.8.0: Export a handoff.md document for transferring the Goal Session identified by goal_id to a new conversation. Includes current goal, completed/pending subgoals, recent diff/test results, blockers, next steps, and risks. Writes to .patchwarden/goals/{goal_id}/handoff.md.
`export_goal_report`	name_restates_behavior no_return_description	After a Goal session's tasks are finished, generate a structured final report for the goal identified by goal_id, aggregating subgoal completion, task evidence, and risk summary. Returns the consolidated final report.
`import_speckit_tasks`	no_return_description	Import Spec Kit tasks into the Goal session identified by goal_id as subgoals, mapping task files to scope hints and acceptance criteria. Returns the created subgoals imported from spec_kit_json.
`release_check`	no_return_description	v1.3.0: Run a bounded release readiness check by wrapping the existing release gate for the repo at repo_path. Local stages use existing guarded release-gate commands; remote stages are read-only. Does not publish, push, tag, or create a GitHub Release. Returns the readiness result for target_stage and the stages checked before it.
`release_verify`	no_return_description	v1.3.0: Verify npm/GitHub/CI release facts with read-only HTTPS requests and return a structured pass/warn/fail status per release fact (npm package, GitHub release, CI branch). Does not run local shell commands and does not publish, push, tag, or create a GitHub Release.
`release_cleanup`	no_return_description	v1.3.0: Clean up release artifacts using project-policy auto_cleanup rules and return an audit summary of artifacts removed or that would be removed. Defaults to dry_run=true. Non-dry-run cleanup only removes low-risk ignored/untracked artifacts under repo_path.
`merge_worktree`	no_return_description	v1.0.0: Merge an isolated git worktree's changes back into the main workspace and return the merge status (success/failure) plus the updated worktree_status.json state. Use after a subgoal task (created with isolate_worktree=true) is accepted. On success sets worktree_status.json to status='merged'. Merge failures do NOT delete the worktree (preserved for manual inspection).
`discard_worktree`	no_return_description	v1.0.0: Discard an isolated git worktree safely and return the discard status. Removes the worktree (git worktree remove --force), deletes its branch, and archives status as 'discarded'. Use when a subgoal is rejected or abandoned. All paths pass guardWorkspacePath + sensitiveGuard.
`create_direct_session`	no_return_description	Create a Direct editing session for ChatGPT to apply patches directly and return a session_id to use in subsequent Direct operations. Requires enableDirectProfile: true in config.
`search_workspace`	no_return_description	Search file contents (grep-like) within a Direct session's repo_path and return up to max_results matches with file paths and preview text (bounded by max_preview_chars). Skips .git, node_modules, dist, release, and sensitive files.
`apply_patch`	params_unexplained no_return_description	Apply JSON patch operations to a file within a Direct session's repo_path and return the resulting patch status. session_id identifies the Direct session from create_direct_session; operations is a list of patch steps, each with type (replace_exact, insert_before, insert_after, replace_whole_file), old_text to locate, new_text to insert or replace, and optional occurrence (first, all, exactly_once). Validates expected_sha256 before applying. Pass a fresh review_id when enforce mode is active.
`create_file`	params_unexplained no_return_description	Create a new bounded UTF-8 text file at a repository-relative path inside an active Direct session and return the created file's status. session_id identifies the Direct session from create_direct_session. The target and its parent are revalidated, sensitive content is blocked, and existing targets are never overwritten. Pass a fresh review_id when enforce mode is active.
`mkdir`	params_unexplained	Create one new directory level at the given repository-relative path inside an active Direct session. session_id identifies the Direct session from create_direct_session; path is the new directory whose parent must already exist. Pass a fresh review_id when enforce mode is active. Linked, sensitive, internal, dependency, release, and build-output paths remain blocked.
`move_file`	params_unexplained no_return_description	Move one bounded regular text file within a Direct session and return the move status. session_id identifies the Direct session; source_path and target_path are repository-relative source and destination paths. Requires the current source SHA-256 (expected_source_sha256) and never overwrites the target. Pass a fresh review_id when enforce mode is active.
`delete_file`	params_unexplained no_return_description	Delete one bounded regular text file inside a Direct session and return the deletion status. session_id identifies the Direct session; path is the repository-relative file path. Requires the current file SHA-256 (expected_sha256) and confirm_delete=true; directories and recursive deletion are unsupported. Pass a fresh review_id when enforce mode is active.
`request_direct_review`	params_unexplained	Request a policy-bound review for the exact Direct operation to be performed and return a review_id. In enforce mode, pass the returned review_id unchanged to the matching operation before it expires. session_id identifies the Direct session; operation_type enumerates the operation (patch, create, mkdir, move, delete, verification, verification_bundle); supply the operation-specific parameters (path, source_path, target_path, expected_sha256, expected_source_sha256, operations, content, command, commands, timeout_seconds) matching the operation under review.
`run_verification`	name_restates_behavior params_unexplained no_return_description	Run one Direct allowlisted verification command within a session and return only bounded structured pass/fail status without stdout/stderr tails. session_id identifies the Direct session from create_direct_session; command selects from the allowed allowlist (npm test, npm run test, npm run build, npm run lint, node --check main.js); timeout_seconds caps execution (default 120). Pass a fresh review_id when enforce mode is active.
`run_direct_verification_bundle`	params_unexplained	Run multiple allowlisted Direct verification commands sequentially and return only bounded structured status. Omits stdout/stderr tails and log content. timeout_seconds caps each command's execution (default 120); commands must be in the Direct allowlist; provide a fresh review_id when enforce mode is active.
`finalize_direct_session`	no_return_description	Finalize a Direct session: capture after snapshot, generate diff/summary/change artifacts, mark session as finalized, and return those generated artifacts. Must be called before audit_session.
`sync_file`	no_return_description	Copy a file from source to target within the same Direct session repo and return the copy status. This legacy operation is outside the first Direct review MVP and is disabled when directReview.mode=enforce. Both paths must be inside the session repo_path.

Selection evidence

Confusable tool pairs.

14 pairs where similar names or overlapping descriptions may send an agent toward the wrong tool.

Tool A	Tool B	Confidence	Why they collide
`finalize_direct_session`	`safe_finalize_direct_session`	high	Both finalize a Direct session and differ only in output verbosity (full diff/summary artifacts vs bounded structured evidence). A request like 'finalize the session' or 'finalize and show the diff' matches both descriptions, so the agent must guess which output form the user wants.
`check_release_gate`	`release_check`	high	Near-synonymous names and purposes: both verify release readiness by stage (local_ready..ci_verified). 'Check release readiness' or 'is the release gate green' maps equally to both; release_check even describes itself as wrapping check_release_gate, so a plain readiness request leaves the canonical choice underdetermined.
`create_task`	`create_subgoal_task`	medium	Both create tasks from plans/templates, and create_subgoal_task also launches an associated task with the same inline_plan/template inputs. A request like 'create a task with this plan' could select either; only the goal-binding requirement (goal_id/subgoal_title) distinguishes them.
`get_result`	`get_result_json`	high	Both read a task's result, one from result.md and one from result.json, and neither is flagged as default. 'Get the task result' matches both, so the agent risks returning markdown when structured JSON was wanted or vice versa.
`get_task_stdout_tail`	`get_task_log_tail`	medium	Both return last-N-lines tails of task output with overlapping wording (get stdout/stderr/tail/lines). 'Show the tail of the task log or output' fits both; get_task_log_tail additionally requires picking a file enum while get_task_stdout_tail auto-handles stdout/stderr with result.md fallback, so a generic tail request is underdetermined.
`audit_session`	`safe_audit_direct_session`	medium	Both independently audit a finalized Direct session, differing only in pass/warn/fail decision vs bounded structured evidence. 'Audit the session' or 'audit the session changes' maps to both, so the agent chooses based on an output preference the user never stated.
`run_verification`	`run_direct_verification_bundle`	medium	Both run the same allowlisted Direct verification commands; the difference is single command vs sequential list and bounded output. 'Run verification' or 'run the tests' on a session fits either, and when the user names several commands the agent must decide whether to batch them.
`audit_task`	`safe_audit`	high	safe_audit is literally defined as running audit_task, so 'audit this task' maps to both. The only difference is full independent-review.md vs bounded structured evidence; a user asking to 'audit the task' gives no signal about which output form is wanted.
`safe_audit`	`safe_audit_direct_session`	medium	Identical 'safe audit' structure differing only in task vs direct-session domain. 'Run the safe audit' or 'give me bounded audit evidence' could pick the task variant for a session (or vice versa) when the user omits the domain word.
`safe_diff_summary`	`safe_direct_summary`	medium	Both return guarded 'what changed' summaries that omit diff content; one is task-based, the other session-based. 'Give me a safe summary of the changes' fits both descriptions, and the agent must infer the domain from context the user may not provide.
`safe_test_summary`	`safe_diff_summary`	medium	Both are compact 'safe' summaries of a single task (verification status vs changed-file counts), each omitting log content. A request like 'summarize this task without the logs' or 'quick safe check on the task' could return test evidence when change scope was wanted or vice versa.
`safe_test_summary`	`safe_direct_summary`	medium	Both return verification-aware summaries that omit stdout/stderr and log content, one for a task and one for a Direct session. 'Give me a safe verification summary' fits both, and the task-vs-session distinction is not surfaced by common phrasing.
`get_task_status`	`get_task_progress`	medium	Both report execution phase, heartbeat, and current command (shared tokens: get/current/phase/task). 'What phase is the task in' or 'how is the task going' is satisfied under both definitions, so the agent's choice of status vs progress.md detail is not dictated by the request.
`get_task_progress`	`get_task_summary`	medium	Shared 'get task' prefix with overlapping state descriptions (phases/heartbeat vs acceptance evidence). A request like 'summarize the task's progress' maps to both, and the agent may return acceptance evidence when progress detail was wanted or vice versa.

Compare the field

One score is useful.
The evidence makes it actionable.

Back to the leaderboard