01Safety
0.0 / 30
What changed in the harness
Selection accuracy 100→96, token cost up 8%, unconfirmed writes 0%→0%.
Category breakdown
Where the score comes from.
Earned points across the four signals Gradable measures. Safety and Legibility are scored out of 30; Economics and Discoverability are scored out of 20.
02Legibility
23.8 / 30
03Economics
16.8 / 20
04Discoverability
12.5 / 20
Highest-impact fix
Estimated gain +30 pointsAdd explicit identity and permission preflight tools
Expose machine-readable principal/tenant confirmation and a non-mutating permission check so agents can verify both before destructive actions.
Description evidence
Defects and rewrites.
16 defects found across the exposed tool descriptions. Suggested rewrites make purpose, inputs, boundaries, and returns easier for an agent to understand.
| Tool | Defect types | Suggested rewrite |
|---|---|---|
check_malicious |
params_unexplained |
Supply-chain malware check against OpenSSF/OSV. USE WHEN: about to suggest install of an unvetted/unfamiliar package; name came from a blog/tutorial. Call BEFORE check_package for untrusted pkgs. `ecosystem` is the registry the package belongs to (npm, pypi, cargo, go, ...); `package` is the exact package name to check in that ecosystem. RETURNS: {is_malicious, threat_tier, source}. |
check_typosquat |
params_unexplained |
Typosquat detector. USE WHEN: name differs from a well-known package by 1-2 chars (`lodsh`, `reqeusts`); copy-paste from unreliable source; downloads near zero but name looks familiar. `ecosystem` is the registry to search in (npm, pypi, cargo, ...); `package` is the suspicious name to scrutinize. RETURNS: {is_typosquat, likely_target, confidence}. |
package_exists |
params_unexplained |
Boolean registry existence check. USE WHEN: about to emit a package name in an install command but unsure it exists; verifying a name generated from training data. `ecosystem` is the registry to query (npm, pypi, cargo, go, ...); `package` is the name to look up. RETURNS: {exists}. |
get_latest_version |
params_unexplained |
Latest published version + deprecation flag — the cheapest call. USE WHEN: only a version string matters (pinning a dep, answering 'what version of X'). If you also need health/vulns use check_package. `ecosystem` is the registry (npm, pypi, cargo, go, ...); `package` is the package to look up. RETURNS: {latest, deprecated, published_at}. |
get_health_score |
params_unexplained |
Single 0-100 health score — cheapest go/no-go gate (>=70 safe). USE WHEN: CI gating or pkg already screened for malware/typos. NOT a first screen — run check_malicious + check_typosquat first. For a verbal verdict use get_package_prompt. `ecosystem` is the registry (npm, pypi, cargo, ...); `package` is the package to score. RETURNS: {score, verdict}. |
install_command |
params_unexplained |
Canonical install command(s) across every package manager of the ecosystem (npm/pnpm/yarn/bun, pip/uv/poetry, cargo, go, composer, maven+gradle, nuget, ...). USE WHEN: emitting an install line and you want correct flags. `ecosystem` is the registry (npm, pypi, cargo, ...); `package` is the package to install; `version` optionally pins an exact version (defaults to latest). RETURNS: {primary, variants[]}. |
get_package_prompt |
params_unexplained |
LLM-optimised package brief — plain text ~300 tokens (~75% cheaper than JSON). Verdict (SAFE/AVOID/URGENT/MALICIOUS) + health + vulns + alternatives + maintainer alerts. USE WHEN: you want to reason over a package and drop the output directly in context; 'is X safe'. PREFER THIS over check_package in 95% of LLM cases. `ecosystem` is the registry (npm, pypi, cargo, ...); `package` is the package to brief. RETURNS: plain-text brief. |
check_package |
params_unexplained |
Full machine-readable JSON report (~2k tokens). USE WHEN: you need to programmatically parse specific fields (CI gating, UI, sub-field extraction). Otherwise prefer get_package_prompt. `ecosystem` is the registry (npm, pypi, cargo, go, ...); `package` is the package name; `version` is an optional specific version (defaults to latest). RETURNS: {package, health:{score}, vulnerabilities[], latest, deprecated, maintainers, recommendation}. |
get_vulnerabilities |
params_unexplained |
CVE/OSV advisories affecting the latest published version of a package. USE WHEN: security-sensitive project; user asks 'any CVEs in X'; you already know the pkg exists. `ecosystem` is the registry (npm, pypi, cargo, ...); `package` is the package to check. RETURNS: {vulnerability_count, vulnerabilities[]: {id, severity, cvss, fixed_in}}. |
find_alternatives |
params_unexplained |
Curated replacements for deprecated/unhealthy packages, including stdlib built-ins (e.g. `fs.rm` for rimraf). USE WHEN: pkg flagged AVOID/URGENT; 'what to use instead of X'; before guessing a replacement name. `ecosystem` is the registry (npm, pypi, cargo, ...); `package` is the package to find alternatives for. RETURNS: {alternatives[]: {name, reason, is_stdlib}}. |
get_migration_path |
params_unexplained |
Prescriptive migration plan between DIFFERENT packages — rationale + literal code diff + breaking changes + effort minutes. USE WHEN: replacing `request`->`axios`, `moment`->`dayjs`, `flask`->`fastapi`, etc.; both endpoints known. `ecosystem` is the registry both packages belong to (npm, pypi, ...); `from_package` is the package to migrate away from; `to_package` is the replacement package. RETURNS: {rationale, diff, breaking_changes[], estimated_minutes}. |
get_breaking_changes |
params_unexplained |
Breaking changes between two versions of the SAME package (e.g. `next@14`->`15`). USE WHEN: user is bumping a major; before recommending a major upgrade. Different from get_migration_path (same pkg vs. different pkg). `ecosystem` is the registry (npm, pypi, cargo, ...); `package` is the package to analyze; `from_version` is the current version; `to_version` is the target version. RETURNS: {breaking_changes[]: {area, description, hint}}. |
pin_safe |
params_unexplained |
Highest version below the chosen CVE severity tier, respecting a semver constraint. USE WHEN: writing a package.json/requirements.txt line; resolving dependabot by lowest-risk patched version. `ecosystem` is the registry (npm, pypi, cargo, ...); `package` is the package to pin; `min_severity` is the lowest severity to exclude (default high, excludes critical+high); `constraint` is an npm-style semver constraint (^X, ~X, >=X, or exact); `include_prerelease` (default false) allows prerelease versions. RETURNS: {recommended_version, walk_log[]}. |
compare_packages |
params_unexplained |
Side-by-side comparison (health, vulns, downloads, maintainers, last release) of 2-10 packages in the same ecosystem. USE WHEN: 'X vs Y' / 'should I pick X or Y'. `ecosystem` is the registry all compared packages belong to (npm, pypi, cargo, ...); `packages` lists the names to compare. RETURNS: table-shaped JSON, one row per package. |
get_known_bugs |
params_unexplained |
Non-CVE known bugs for a specific package version. USE WHEN: unexpected behavior that is NOT a security issue; a pinned version misbehaves. `ecosystem` is the registry (npm, pypi, cargo, ...); `package` is the package name; `version` is the version to look up bugs for. RETURNS: {bugs[]: {title, fixed_in, workaround}}. |
get_trust_signals |
params_unexplained |
One-call aggregate of ALL non-CVE supply-chain trust signals: maintainer trust (bus factor, ownership changes), OpenSSF Scorecard, quality (criticality, release velocity, publish security), and SLSA/Sigstore provenance. USE WHEN: deep-vetting a package beyond CVEs (hardened/regulated env, SBOM/compliance, small-pkg ownership review, choosing between healthy candidates). Runs 4 backend endpoints in parallel. `ecosystem` is the registry (npm, pypi, cargo, ...); `package` is the package to vet. RETURNS: {maintainer, scorecard, quality, provenance} — each may be null if its backend call failed. |
Selection evidence
Confusable tool pairs.
10 pairs where similar names or overlapping descriptions may send an agent toward the wrong tool.
| Tool A | Tool B | Confidence | Why they collide |
|---|---|---|---|
check_bulk |
check_compatibility |
medium | Both take a set of packages and start with 'check'. A task like 'verify this package set before I pin/install it' is ambiguous: check_bulk is the pre-install existence/malware/typo filter, while check_compatibility confirms a version combo works together; an agent told just to 'check these packages' could pick either. |
check_bulk |
check_package |
medium | Both are 'check a package' tools but check_bulk accepts lists and returns only coarse per-item statuses while check_package returns a full JSON report per package. A task asking for a detailed assessment of several packages could route to the single bulk call even though it lacks report detail. |
check_malicious |
check_typosquat |
medium | Identical schemas (ecosystem+package) and both are single-package supply-chain security screens. A task like 'is this package suspicious/dangerous/safe to install' does not reveal whether malware or typosquatting is the concern, so an agent may pick the wrong member. |
check_malicious |
check_package |
low | Overlapping intent for 'is X safe to install/use' with the same schema; check_malicious explicitly frames itself as the pre-check run before check_package, so a generic 'check this package' request is genuinely ambiguous even if each description names its own trigger. |
check_package |
compare_packages |
medium | A task requesting details on two or more packages ('tell me about express and fastify') could be satisfied either as a compare_packages table (health/vulns/downloads) or as repeated check_package full reports; only an explicit 'X vs Y / which should I pick' maps unambiguously to compare_packages. |
get_health_score |
get_vulnerabilities |
medium | Both are go/no-go signals for the same question 'is X safe/risk-free': the score gate (>=70 safe) versus the CVE/OSV list. A safety-oriented task that omits the word 'vulnerabilities' can route to the wrong member despite the distinct names. |
get_latest_version |
get_vulnerabilities |
medium | Both operate on the 'latest version' of a package (reflected in shared tokens latest/version). A task like 'any issues with the latest version of X' or vague 'tell me about the latest version of X' is ambiguous between returning just the version string and returning its CVEs. |
get_breaking_changes |
get_known_bugs |
medium | Both report problems tied to a package version. 'What known issues will I hit upgrading next from 14 to 15' could mean breaking changes across majors (get_breaking_changes) or non-CVE bugs on a pinned version (get_known_bugs); the word 'issues' does not disambiguate, and both take ecosystem+package. |
get_package_prompt |
get_vulnerabilities |
low | get_package_prompt returns a plain-text brief that already embeds vulnerability info and owns the 'is X safe' task, so a request like 'should I worry about X's security' overlaps with get_vulnerabilities; only an explicit CVE question clearly maps to get_vulnerabilities. |
get_known_bugs |
get_trust_signals |
low | Both are explicitly non-CVE investigations of a package (shared token 'cve' in descriptions). A task like 'what do I need to know about X beyond CVEs' is ambiguous between version-specific non-CVE bugs and the maintainer/scorecard/provenance trust aggregate. |
Compare the field