Public assessment

jiezeng2004-design/PatchWarden (patchwarden)

patchwarden · v1.7.2 · scanned Aug 2, 2026

What changed in the harness

Selection accuracy 90→96, token cost up 5%, unconfirmed writes 0%→0%.

Category breakdown

Where the score comes from.

Earned points across the four signals Gradable measures. Safety and Legibility are scored out of 30; Economics and Discoverability are scored out of 20.

01Safety

0.0 / 30

0.0 out of 30

02Legibility

16.2 / 30

16.2 out of 30

03Economics

10.3 / 20

10.3 out of 20

04Discoverability

9.4 / 20

9.4 out of 20

Highest-impact fix

Estimated gain +30 points

Add explicit identity and permission preflight tools

Expose machine-readable principal/tenant confirmation and a non-mutating permission check so agents can verify both before destructive actions.

Description evidence

Defects and rewrites.

49 defects found across the exposed tool descriptions. Suggested rewrites make purpose, inputs, boundaries, and returns easier for an agent to understand.

Tool	Defect types	Suggested rewrite
`save_plan`	no_return_description	Save an execution plan — ChatGPT writes the plan, PatchWarden stores it for local agent execution. Supports plan_ref to load a plan file already placed inside .patchwarden/plans. Returns the stored plan_id and resolved title for use with get_plan or create_task.
`get_plan`	no_return_description	Read a saved plan by its plan_id. Returns the plan's title and full Markdown content, or an error if no plan matches the ID.
`health_check`	no_return_description	Check MCP catalog consistency, watcher freshness/supervisor state, workspace readiness, and configured agents. Agent availability is executable-only unless an explicit provider probe is requested. Use detail=self_diagnostic for expanded read-only evidence. Returns a structured status report covering each checked area (catalog, watcher, agents, workspace, and optionally allowlist/failure evidence) with pass/fail/warn indicators.
`list_agents`	no_return_description	List agents from the active runtime config, verify executable and model-argument wiring, and report the config path used by this process. provider_status remains not_checked because this read-only check does not contact a model provider or test account balance. Returns an array of agent entries (name, executable path, wiring status, provider_status) plus the resolved config path.
`export_task_evidence_pack`	no_return_description	Export a bounded evidence pack for a run_task_loop lineage. Writes evidence.json and EVIDENCE.md without stdout, stderr, full logs, full diff, or sensitive file contents. Returns the written file paths and a bounded summary of included rounds/tasks/warnings.
`get_task_status`	no_return_description	Check task status, execution phase, watcher health, pending reason, current command, timeout, and change evidence. Returns a structured status object covering each of these fields for the given task_id.
`get_result`	params_unexplained	Read result.md for a task, or return structured availability and watcher evidence while the task is not yet terminal. task_id identifies the task whose result should be read (obtained from create_task).
`get_result_json`	params_unexplained	Read the structured result.json for deterministic task acceptance. task_id identifies the task whose result.json should be read (obtained from create_task).
`get_diff`	params_unexplained	Read task diff evidence, or return structured availability and watcher evidence while it is not ready. task_id identifies the task whose diff should be read (obtained from create_task).
`get_test_log`	params_unexplained	Read test.log, or return structured availability and watcher evidence while it is not ready. task_id identifies the task whose test log should be read (obtained from create_task).
`list_workspace`	name_restates_behavior no_return_description	List files and directories within the workspace, or under an optional relative subpath, excluding sensitive files. Returns an array of entries (name, type, and path) found at that location.
`list_tasks`	no_return_description	List recent tasks with status/repo/active/history filters plus watcher state and computed pending reasons. Archived history is excluded by default. Returns an array of matching task summaries (task_id, status, repo, pending reason, watcher state).
`cancel_task`	no_return_description	Request graceful cancellation of a task; the runner that owns the child process performs termination, and the MCP server never kills a PID read from task files. Returns the updated cancellation request status for the given task_id.
`kill_task`	name_restates_behavior no_return_description	Request immediate termination of a pending or running task, validated and executed only by the runner that owns the child process. Returns the resulting termination request status for the given task_id.
`retry_task`	no_return_description	Create a new task with the same plan, agent, repo_path, and test_command as an existing task, leaving the original task unchanged. Returns the newly created task's task_id.
`get_task_stdout_tail`	params_unexplained	Read the last N lines of agent stdout/stderr for a task. Reads from real-time stdout.log/stderr.log during execution, falls back to result.md after completion. Works on pending, running, and completed tasks. task_id identifies the task (default 80 lines, override with lines).
`get_task_log_tail`	params_unexplained	Read the last N lines of a task log file (stdout/stderr/test/verify) with automatic secret redaction. Default 80 lines, max 200. Always returns tail only — never the full file. Use this instead of read_workspace_file to avoid triggering platform content filters on log output. task_id identifies the task whose log is read.
`get_task_progress`	params_unexplained no_return_description	Read progress.md for a task, identified by task_id, and return its phases and the most recent heartbeat/current command.
`wait_for_task`	params_unexplained no_return_description	Long-poll a task (task_id) for up to wait_seconds (or its alias timeout_seconds), max 30 seconds. Returns the task's current status, or continuation_required=true if it has not reached a terminal state (call again immediately without ending the assistant turn); terminal responses include get_task_summary acceptance evidence.
`get_task_summary`	params_unexplained	Return structured acceptance evidence for a task_id. Use view=compact first for bounded counts and risk excerpts (limited by max_items per group); use view=standard only when full changed-file and log-tail detail is required.
`explain_tool`	no_return_description	Expand a single tool's metadata — title, summary, risk level, tags, aliases, profiles, modes, schema digest, and optionally the full inputSchema. Use after discover_tools to understand a specific tool before calling it. Returns the tool's metadata object described above.
`invoke_discovered_tool`	no_return_description	Invoke a previously discovered tool using a discoveryToken obtained from discover_tools. Enforces 10 security checks: token validity, toolName match, profile allowance, risk ceiling, sensitive path guard, assessment requirement, command whitelist, release confirmation, credential block, and invocation logging. Cannot call itself recursively. Returns the invoked tool's own result, or a security-check failure reason if a check blocks the call.
`create_goal`	no_return_description	Create a Goal Session for managing a multi-task objective with subgoal dependencies. Generates a structured directory under .patchwarden/goals/{goal_id}/ with GOAL.md, GOALS.md, and goal_status.json. Use list_goals to enumerate existing goals and read_goal to inspect details. Returns the new goal_id and initial goal_status.json contents.
`read_goal`	no_return_description	Read full Goal Session details including GOAL.md content, goal_status.json, and all subgoals with dependency info, for the given goal_id. Use after list_goals to inspect a specific goal. Returns the goal's title, description, status, and subgoal list with dependencies.
`create_subgoal_task`	params_unexplained	Create a subgoal within a Goal Session and immediately launch an associated task. Atomically: addSubgoal → create_task → linkTask → mark subgoal running. The subgoal depends_on other subgoals (by id) which must be accepted before suggest_next_subgoal returns it. agent optionally names a configured local agent to run the task; no agents are currently configured.
`accept_subgoal`	params_unexplained no_return_description	Accept a subgoal (subgoal_id) within a Goal Session (goal_id) after all its associated tasks are accepted by audit_task. Supports both legacy status=accepted and done_by_agent with acceptance_status=accepted. Returns the updated subgoal status.
`reject_subgoal`	params_unexplained no_return_description	Reject a subgoal (subgoal_id) within a Goal Session (goal_id) with a required reason. Allowed from any non-terminal status (ready/queued/running/done_by_agent/needs_fix). Returns the updated subgoal status after recording rejected_reason in goal_status.json.
`suggest_next_subgoal`	params_unexplained	Suggest the next executable subgoal for a Goal Session (goal_id) based on the dependency graph. Returns a ready subgoal whose dependencies are all accepted, or a blocked_by list if none are ready.
`summarize_goal_progress`	params_unexplained no_return_description	Summarize completion of a Goal Session (goal_id): returns counts by status (accepted/rejected/queued/running/ready/needs_fix/done_by_agent), completion_rate, blocked_subgoals, and actionable risks.
`export_handoff`	params_unexplained	Export a handoff.md document for transferring a Goal Session (goal_id) to a new conversation. Includes current goal, completed/pending subgoals, recent diff/test results, blockers, next steps, and risks. Returns the path to the written .patchwarden/goals/{goal_id}/handoff.md file.
`export_goal_report`	name_restates_behavior no_return_description	Aggregate subgoal completion, task evidence, and risk summary for a Goal session into a structured final report. Returns the report's file path and a structured summary object.
`import_speckit_tasks`	no_return_description	Import Spec Kit tasks into a Goal session as subgoals, mapping task files to scope hints and acceptance criteria. Returns the list of created subgoal IDs and any tasks skipped due to mapping errors.
`release_check`	no_return_description	Run a bounded release readiness check by wrapping the existing release gate. Local stages use existing guarded release-gate commands; remote stages are read-only. Does not publish, push, tag, or create a GitHub Release. Returns per-stage pass/fail/not_checked status up to target_stage.
`release_verify`	no_return_description	Verify npm/GitHub/CI release facts with read-only HTTPS requests. Does not run local shell commands and does not publish, push, tag, or create a GitHub Release. Returns the verification result for each checked fact (npm publish status, GitHub release status, CI status).
`release_cleanup`	no_return_description	Clean up release artifacts using project-policy auto_cleanup rules. Defaults to dry_run=true. Non-dry-run cleanup only removes low-risk ignored/untracked artifacts under repo_path and writes an audit summary. Returns the list of artifacts removed (or, in dry_run, that would be removed) plus the audit summary.
`merge_worktree`	no_return_description	Merge an isolated git worktree's changes back into the main workspace. Use after a subgoal task (created with isolate_worktree=true) is accepted. Updates worktree_status.json to status='merged'. Merge failures do NOT delete the worktree (preserved for manual inspection). Returns the merge outcome and updated worktree_status.json.
`discard_worktree`	no_return_description	Discard an isolated git worktree safely. Removes the worktree (git worktree remove --force), deletes its branch, and archives status as 'discarded'. Use when a subgoal is rejected or abandoned. All paths pass guardWorkspacePath + sensitiveGuard. Returns the resulting discard status.
`create_direct_session`	no_return_description	Create a Direct editing session for ChatGPT to apply patches directly. Requires enableDirectProfile: true in config. Returns the new session_id for use with apply_patch, run_verification, and finalize_direct_session.
`search_workspace`	no_return_description	Search file contents (grep-like) within a Direct session's repo_path. Skips .git, node_modules, dist, release, and sensitive files. Returns matching file paths with line numbers and bounded text previews.
`apply_patch`	params_unexplained no_return_description	Apply JSON patch operations (session_id identifies the Direct session; operations lists the replace_exact/insert_before/insert_after/replace_whole_file edits to perform) to a file within that session's repo_path. Validates expected_sha256 before applying. Returns the updated file's new sha256 and a summary of applied operations.
`create_file`	params_unexplained no_return_description	Create a new bounded UTF-8 text file inside an active Direct session (session_id). The target and its parent are revalidated, sensitive content is blocked, and existing targets are never overwritten. Returns the created file's path and sha256.
`mkdir`	params_unexplained	Create one new directory level (path) inside an active Direct session (session_id); pass review_id when a fresh review is required for this exact operation. Parent directories must already exist; linked, sensitive, internal, dependency, release, and build-output paths remain blocked.
`move_file`	params_unexplained no_return_description	Move one bounded regular text file within a Direct session (session_id) from source_path to target_path. Requires the current source file's expected_source_sha256 and never overwrites the target; pass review_id when a fresh review is required. Returns the moved file's new path and sha256.
`delete_file`	params_unexplained no_return_description	Delete one bounded regular text file (path) inside a Direct session (session_id). Requires the current expected_sha256 and confirm_delete=true; pass review_id when a fresh review is required. Directories and recursive deletion are unsupported. Returns the deletion confirmation status.
`request_direct_review`	params_unexplained	Request a policy-bound review for the exact Direct operation to be performed next, identified by operation_type (patch/create/mkdir/move/delete/verification/verification_bundle) within session_id. Supply whichever of path, source_path, target_path, expected_sha256, expected_source_sha256, operations, content, command, commands, and timeout_seconds match that operation, so the review matches it exactly. In enforce mode, pass the returned review_id unchanged to the matching operation before it expires.
`run_verification`	name_restates_behavior params_unexplained no_return_description	Run a whitelisted command (command) within a Direct session (session_id), optionally with a timeout_seconds override and a review_id when a fresh review is required. Command must be in the Direct allowlist. Returns the command's pass/fail result and structured status (without raw stdout/stderr).
`run_direct_verification_bundle`	params_unexplained	Run multiple allowlisted Direct verification commands sequentially within a session, using timeout_seconds as the per-command timeout. Returns only bounded structured status per command, omitting stdout/stderr tails and log content.
`finalize_direct_session`	no_return_description	Finalize a Direct session: capture after snapshot, generate diff/summary/change artifacts, mark session as finalized. Must be called before audit_session. Returns the finalized session's summary and generated artifact paths.
`sync_file`	no_return_description	Copy a file from source_path to target_path within the same Direct session repo. This legacy operation is outside the first Direct review MVP and is disabled when directReview.mode=enforce. Both paths must be inside the session repo_path. Returns the copied target file's sha256.

Selection evidence

Confusable tool pairs.

20 pairs where similar names or overlapping descriptions may send an agent toward the wrong tool.

Tool A	Tool B	Confidence	Why they collide
`finalize_direct_session`	`safe_finalize_direct_session`	high	Both finalize the same Direct session with identical required params; only difference is output verbosity (safe_ variant returns bounded evidence, omitting diff/log content). A task like 'finalize this session' gives no signal on which output shape is wanted, so an agent could pick either.
`check_release_gate`	`release_check`	high	Both check release readiness across the same five stages with nearly identical parameters (repo_path, target_stage, package_name, version, github_repo, branch). release_check wraps check_release_gate but their descriptions overlap heavily, making 'check if this is ready to release' ambiguous between them.
`create_task`	`create_subgoal_task`	medium	Both create a task from a plan/template with overlapping params (repo_path, test_command, agent, etc.). If the user is working within a goal session, 'create a task to do X' could plausibly go to either, though create_subgoal_task requires goal_id which somewhat disambiguates.
`get_result`	`get_result_json`	medium	Both read task result/acceptance data for a task_id; 'get the result of this task' doesn't clearly indicate whether markdown (get_result) or structured JSON (get_result_json) is wanted.
`get_task_stdout_tail`	`get_task_log_tail`	high	Both read the tail of stdout for a task_id with a 'lines' param; get_task_log_tail also supports file=stdout, making it functionally overlapping with get_task_stdout_tail for the common case of 'show me the last lines of output'.
`safe_finalize_direct_session`	`safe_audit_direct_session`	medium	Both are bounded-evidence Direct session operations taking session_id/max_items; a vague request like 'safely check this session' could be confused between finalizing and auditing since finalize is a prerequisite step to audit.
`audit_session`	`safe_audit_direct_session`	high	Both audit a Direct session by session_id and return pass/warn/fail-style evidence; the only difference is verbosity (full vs bounded), so 'audit this session' is ambiguous between them.
`get_task_lineage`	`get_task_progress`	low	Both read task-related summaries but lineage_id vs task_id as the key param and clearly different content (loop lineage vs phase/heartbeat) reduce real confusion; only plausible if user vaguely says 'show me task progress' in a loop context.
`get_task_status`	`get_task_summary`	medium	Both return task_id-keyed evidence about state/acceptance; a request like 'give me the status of this task' could plausibly route to either since get_task_summary also covers acceptance evidence overlapping with status.
`run_verification`	`run_direct_verification_bundle`	medium	Both run allowlisted verification commands in a Direct session; run_verification takes a single command while the bundle takes multiple, so 'run the tests in this session' is ambiguous when only one command is needed.
`safe_audit`	`safe_audit_direct_session`	medium	Both return bounded audit evidence for a task_id vs session_id respectively; if the user conflates 'task' and 'session' terminology (e.g., after create_direct_session), 'audit this' could be misrouted.
`safe_diff_summary`	`safe_direct_summary`	low	Both return bounded summaries without content, but one is diff-specific for a task while the other is a general Direct session summary; naming similarity is higher than functional overlap.
`safe_test_summary`	`safe_diff_summary`	low	Both are bounded task summaries but cover distinct evidence (test results vs changed files), so a specific ask like 'summarize the tests' or 'summarize the diff' disambiguates well despite shared shape.
`get_result`	`get_diff`	low	Both read task evidence with fallback structured availability info, but 'result' vs 'diff' are distinct enough concepts that confusion is unlikely except in very vague prompts like 'show me what happened'.
`move_file`	`delete_file`	low	Both operate on a single file within a Direct session with SHA-256 verification, but 'move' and 'delete' are clearly distinct actions; confusion is unlikely for concrete tasks.
`create_direct_session`	`safe_finalize_direct_session`	low	Both concern Direct sessions but represent opposite lifecycle ends (create vs finalize); low chance of genuine confusion despite shared tokens.
`merge_worktree`	`discard_worktree`	low	Both act on a worktree_id but represent opposite outcomes (merge vs discard); a clear task intent ('merge' vs 'discard/abandon') disambiguates well.
`cancel_task`	`kill_task`	medium	Both terminate a running task_id; 'stop this task' is genuinely ambiguous between graceful cancellation and immediate kill since the descriptions differ mainly in urgency/method, not user-facing intent.
`accept_subgoal`	`reject_subgoal`	low	Opposite actions (accept vs reject) on a subgoal_id; intent is almost always explicit in the task phrasing, so confusion is unlikely.
`audit_task`	`safe_audit`	medium	safe_audit explicitly wraps audit_task and returns a subset of its output; a generic 'audit this task' request doesn't indicate whether full markdown or bounded evidence is desired.

Compare the field

One score is useful.
The evidence makes it actionable.

Back to the leaderboard