Public assessment

goklab/guardvibe (guardvibe)

guardvibe · v3.31.0 · scanned Aug 3, 2026

What changed in the harness

Selection accuracy 88→90, token cost up 1%, unconfirmed writes 0%→0%.

Category breakdown

Where the score comes from.

Earned points across the four signals Gradable measures. Safety and Legibility are scored out of 30; Economics and Discoverability are scored out of 20.

01Safety

0.0 / 30

0.0 out of 30

02Legibility

22.3 / 30

22.3 out of 30

03Economics

13.7 / 20

13.7 out of 20

04Discoverability

15.1 / 20

15.1 out of 20

Highest-impact fix

Estimated gain +30 points

Add explicit identity and permission preflight tools

Expose machine-readable principal/tenant confirmation and a non-mutating permission check so agents can verify both before destructive actions.

Description evidence

Defects and rewrites.

9 defects found across the exposed tool descriptions. Suggested rewrites make purpose, inputs, boundaries, and returns easier for an agent to understand.

Tool	Defect types	Suggested rewrite
`check_code`	no_return_description	Analyze a code snippet for security vulnerabilities (OWASP Top 10, XSS, SQL injection, insecure patterns) and return the findings in the requested format (markdown or structured json). Pass code as a string parameter; for scanning files on disk, use scan_file instead. Example: check_code({code: 'app.get(...)', language: 'javascript'})
`check_dependencies`	no_return_description	Check npm, PyPI, or Go packages against the OSV database for known security vulnerabilities (CVEs) and return a list of affected packages with severity, CVE IDs, and remediation guidance. Use this before adding new dependencies or to audit existing ones.
`scan_dependencies`	no_return_description	Parse a lockfile or manifest (package.json, package-lock.json, requirements.txt, go.mod) and check all dependencies against the OSV database for known CVEs, returning findings with severity, affected versions, and fix recommendations. Reads the file directly. Use after installing dependencies, in CI, or when auditing existing projects for vulnerable packages.
`check_package_health`	no_return_description	Check npm packages for typosquat risk, maintenance status, adoption metrics, and deprecation, and return a health report per package with risk flags. Use this before adding new dependencies to catch suspicious or risky packages.
`audit_config`	no_return_description	Audit application config files (next.config, middleware, .env, vercel.json) for cross-file security gaps — missing headers, unprotected routes, exposed secrets — and return a report of the gaps found with locations and fix guidance. NOT the same as guardvibe_doctor, which checks AI host security. Example: audit_config({path: '.'})
`audit_mcp_config`	no_return_description	Scan MCP configuration files (.claude/settings.json, .cursor/mcp.json, .vscode/mcp.json) for security issues — malicious hooks (CVE-2025-59536), suspicious MCP servers, overly permissive tool access, shell injection patterns — and return findings per file with severity and remediation steps. Use this to verify MCP configurations are safe before use.
`scan_host_config`	no_return_description	Scan the host environment for AI security issues — API base URL hijacking (CVE-2026-21852), credential exposure in shell profiles, .env file leaks, environment variable sniffing — and return findings with locations and fixes. Checks .env files at project scope; add scope=host to also check shell profiles and global AI configs.
`guardvibe_doctor`	no_return_description	Run a security check on the AI host environment — MCP configurations, hooks, base URL hijacking, environment variable exposure — and return a report of issues found with severity and remediation steps. NOT the same as audit_config, which checks application config files (next.config, .env, headers). Use scope=project (default) for project-only checks, or scope=host to include shell profiles and global AI configs. Example: guardvibe_doctor({scope: 'project'})
`deep_scan`	no_return_description	Run an LLM-powered deep security analysis for vulnerabilities that pattern matching cannot detect — IDOR, business logic flaws, race conditions, stale auth, mass assignment, privilege escalation — and return findings with descriptions and remediation guidance. Defaults to Claude Haiku 4.5 (~cents per scan); pass model:'sonnet' for deeper analysis at higher cost. Requires ANTHROPIC_API_KEY or OPENAI_API_KEY env var.

Selection evidence

Confusable tool pairs.

17 pairs where similar names or overlapping descriptions may send an agent toward the wrong tool.

Tool A	Tool B	Confidence	Why they collide
`check_dependencies`	`scan_dependencies`	high	Both check dependencies for known CVEs via the OSV database but take different inputs (explicit package list vs manifest file path). A natural task like 'check my dependencies for vulnerable packages' leaves the input form ambiguous, so an agent could easily pick the wrong tool.
`scan_file`	`scan_changed_files`	medium	Tasks like 'scan the file(s) I modified' are ambiguous between a single-path scan and a git-diff-based changed-file scan; the agent must infer whether the target is one file or a set of changed files.
`scan_secrets`	`scan_secrets_history`	medium	Leaked-secret tasks rarely specify scope: 'scan for leaked secrets in the repo' could mean current files (scan_secrets) or committed history (scan_secrets_history), and the near-identical names make a wrong pick easy.
`analyze_dataflow`	`analyze_cross_file_dataflow`	medium	Both track user input flowing into dangerous sinks; 'analyze the data flow in my app for injection' forces the agent to decide single-snippet vs multi-file scope from context alone.
`audit_config`	`audit_mcp_config`	medium	Both audit 'config files' for security; a task like 'audit my project's configs' is ambiguous between application configs (next.config/.env) and MCP server configs, despite descriptions attempting to disambiguate.
`audit_config`	`full_audit`	medium	A task asking to 'audit my config/project security' could map to the targeted config audit or the all-checks full_audit; their overlapping coverage creates real selection ambiguity.
`explain_remediation`	`remediation_plan`	medium	Both produce remediation guidance; a request like 'explain/give me the remediation for my audit findings' is ambiguous between per-rule explanation (rule_id) and the section-by-section plan that follows full_audit.
`verify_fix`	`verify_remediation`	medium	Both 'verify' fixes; a user asking to verify a fix rarely specifies single-rule vs whole-audit scope, so either verification tool could be chosen.
`check_code`	`check_project`	low	'Check my project's code for vulnerabilities' could pick the inline snippet analyzer or the multi-file report tool depending on whether the agent holds code inline or file content; mild ambiguity.
`check_code`	`check_command`	low	A pasted shell script with 'check this for security issues' fits both the code analyzer (shell language) and the command analyzer; usually disambiguated by whether it is a command to run vs code.
`scan_directory`	`scan_file`	low	Generic 'scan this path' tasks could target a file or a directory, but the path vs file_path parameters usually disambiguate the intent.
`secure_this`	`secure_prompt`	low	Both start with 'secure'; a task like 'secure this before generating code' could mis-target the code-fixer vs the prompt-enhancer, though the code vs raw_prompt inputs largely separate them.
`scan_config_change`	`scan_changed_files`	low	Reviewing changed config files could be done via before/after diff or git diff-aware file scan; scope wording generally settles it but a generic 'scan my config changes' task is weakly ambiguous.
`audit_config`	`scan_config_change`	low	Both target config-file security, but one audits current state (path) while the other needs before/after content; a 'check my config for security' task could briefly match either.
`audit_mcp_config`	`scan_host_config`	low	Both scan AI tooling/environment configs for security (shell profiles/.env vs MCP configs), so a generic 'scan my AI setup for security issues' task is weakly ambiguous.
`remediation_plan`	`verify_remediation`	low	A generic 'handle remediation for my audit' task could target the planning step or the verification step; the sequential descriptions mostly disambiguate which comes first.
`scan_hallucinated_packages`	`check_package_health`	low	Both check packages for supply-chain risk (phantom/typosquat vs typosquat/maintenance), but one scans a repo while the other takes explicit package names, which usually clarifies the choice.

Compare the field

One score is useful.
The evidence makes it actionable.

Back to the leaderboard