# Gradable scoring methodology (v1) Gradable publishes agent-readiness scores for third-party MCP servers at [gradable.dev](https://gradable.dev). This document states exactly how those scores are produced, so a maintainer who disagrees with a score can check the arithmetic rather than take our word for it. It describes **what the scorer implements today**, not what the product spec aspires to. Where the two differ, that gap is called out below rather than papered over. The rubric is maintained in the open at [AgentWorkforce/agent-tool-spec](https://github.com/AgentWorkforce/agent-tool-spec), together with the JSON Schema for a conforming report. Disagreement with a weight, a defect type, or a score belongs there as an issue — that repository is where the rules can be contested. Source of truth for every formula here: `packages/score/src/index.ts`. --- ## 1. What is measured Gradable connects to an MCP server over stdio, reads its tool surface (names, descriptions, input and output schemas), and generates a set of natural-language tasks from that surface. It then asks a judge model, for each task, which tool it would call and with what arguments. **Gradable never executes your tools and never reads your source.** Every finding is derived from the declared surface plus the judge's observed selection behaviour. This is the single most important limitation to understand: gradable measures how legible your surface is to a model, not whether your implementation is correct. ### Harness metrics Each scan produces a `baseline` run and, where description defects were found, an `after_rewrite` run over the same tasks with gradable's suggested descriptions substituted. Both are recorded in the report; only `baseline` feeds the score. | Metric | Meaning | |---|---| | `selectionAccuracy` | Fraction of tasks where the judge chose the expected tool | | `firstCallSuccess` | Fraction where the first call was also well-formed against the schema | | `avgStepsToCompletion` | Mean tool calls per task | | `avgTokenCostPerTask` | Mean tokens consumed per task | | `destructiveSafetyRate` | Fraction of destructive candidate calls that were confirmed rather than issued blind | --- ## 2. The score Composite is the sum of four categories, out of 100. `clamp(x)` bounds a value to `[0, 1]`. ### Safety — 30 points ``` safety = 30 × clamp(destructiveSafetyRate) ``` Weighted highest because these are the failures that produce incidents rather than degraded output. Note this is **purely behavioural**: it measures whether destructive calls were confirmed during the harness run. It does not check for the presence of dry-run support, idempotency keys, or permission preflight. ### Legibility — 30 points ``` legibility = 30 × (0.60 × clamp(selectionAccuracy) + 0.25 × descriptionQuality + 0.15 × disambiguationQuality) descriptionQuality = 1 − clamp(defectWeight / 30) defectWeight = Σ over tools of (number of distinct defect types on that tool) disambiguationQuality = 1 − clamp(ambiguousPairWeight / 10) ambiguousPairWeight = Σ over confusable pairs of {low: 0.25, medium: 0.6, high: 1.0} ``` Selection dominates deliberately; the static and pairwise analysis explain *why* selection fails rather than standing on their own. ### Economics — 20 points ``` economics = 20 × (0.60 × definitionEfficiency + 0.40 × taskEfficiency) [no response data] = 20 × (0.50 × definitionEfficiency + 0.30 × taskEfficiency + 0.20 × responseEfficiency) [with response data] linearEfficiency(value, ideal, unacceptable): 1 if value ≤ ideal 0 if value ≥ unacceptable 1 − (value − ideal) / (unacceptable − ideal) otherwise ``` | Input | Ideal | Unacceptable | |---|---|---| | Definition tokens (whole tool surface) | 2,000 | 50,000 | | Avg tokens per task | 2,000 | 20,000 | | Avg response tokens | 500 | 8,000 | Reports declare `economics.definitionTokensSource` as `exact` or `estimated`. `estimated` means no tokenizer credential was available and the value is a UTF-8-bytes ÷ 4 heuristic. Treat estimated token scores as approximate. ### Discoverability — 20 points ``` discoverability = 20 × (0.40 × clamp(selectionAccuracy) + 0.60 × clamp(firstCallSuccess)) ``` This is explicitly a **proxy**. v1 records a selection-only trace, so first-call construction stands in for capability introspection and natural-name resolution. It is not a direct measurement of either. ### Grades | Score | Grade | |---|---| | ≥ 90 | A | | ≥ 80 | B | | ≥ 70 | C | | ≥ 60 | D | | < 60 | F | ### A note on double-counting `selectionAccuracy` appears in both legibility (0.60 of 30) and discoverability (0.40 of 20), so it drives 26 of the 100 available points. This is intentional — selection is the behaviour the product exists to measure — but it means a surface that a model picks well from scores well even if it is otherwise sparse. --- ## 3. Description defects Three defect types, judged per tool: | Type | A description has this defect when | |---|---| | `name_restates_behavior` | It mostly repeats the tool name instead of adding when/why/behaviour detail | | `params_unexplained` | A required parameter's meaning, accepted values, or default appear nowhere the agent can see | | `no_return_description` | It does not say what the call returns or yields, and `outputSchema` does not make it explicit | For each defective tool the judge also writes a suggested replacement description. **These suggestions are generated from the name, schema, and current description only** — the judge never sees the handler (`packages/analyze/src/judge.ts`). They are hypotheses to verify, not corrections to apply. Every published report ships a fix brief at `gradable.dev/reports//fix.md` that states this constraint and asks the fixer to confirm each suggestion against the implementation. Applying suggestions verbatim can raise a gradable score while making descriptions *less* accurate. That is a failure, not a win, and we would rather say so than harvest the score. --- ## 4. Reproducibility and comparability Every report records the judge identity that produced it: ```json "judge": { "cli": "opencode", "cliVersion": "...", "model": "deepseek/deepseek-v4-flash", "taskGeneratorVersion": "tasks-v1", "promptTemplateVersion": "selection-v1" } ``` **Scores from different judge configurations are not directly comparable.** Judge models differ in how strictly they read a description and how reliably they pick a tool, and those differences move scores. The leaderboard states how many distinct configurations are present in the current batch and drops the comparability claim when there is more than one. As of 2026-08-03 the corpus is **457 of 460 scored reports on a single configuration** (`opencode` / `deepseek/deepseek-v4-flash`, prompt `selection-v1`, generator `tasks-v1`). Cross-server ranking within that set is meaningful. The six exceptions are listed below. ### Known provenance defect Reports generated **before commit `5050db6`** recorded `judge.cli` and `judge.model` from the command-line flags, while the description and confusability stage always invoked the Claude CLI regardless of those flags. For those reports the recorded judge identity is accurate for task generation and selection, but **not** for the description defects that feed `descriptionQuality`. A regeneration pass on 2026-08-03 rebuilt 195 of the 201 affected reports. **Six could not be regenerated** because they fail to scan for reasons unrelated to the judge, so they retain their original pre-fix scores: | Report | Scanned | `judge` block says | Actually judged by | |---|---|---|---| | `agent-relay` | 2026-08-01 | claude / claude-sonnet-5 | accurate | | `mcp-sequential-thinking` | 2026-08-01 | claude / claude-sonnet-5 | accurate | | `relaycast` | 2026-08-01 | claude / claude-sonnet-5 | accurate | | `codemagic-mcp-server` | 2026-08-02 | opencode / deepseek-v4-flash | **description stage ran claude-sonnet-5** | | `geomelon-mcp` | 2026-08-02 | opencode / deepseek-v4-flash | **description stage ran claude-sonnet-5** | | `real-browser-mcp` | 2026-08-02 | opencode / deepseek-v4-flash | **description stage ran claude-sonnet-5** | For the last three, the `judge` block understates what produced the findings. Do not compare those six against the rest of the leaderboard. We publish this rather than quietly regenerate because a rubric that hides its own defects is not a rubric. --- ## 5. What this does not measure Of the nine capabilities in the product spec (`docs/spec.md` §3), v1 directly scores none as present/partial/absent. Not currently measured: - capability introspection as a manifest - identity and tenant confirmation - permission preflight - dry-run support - idempotency keys - natural-name resolution - actionable error semantics - quota and rate state as data - query-over-list support Safety, discoverability, and legibility are scored from observed harness behaviour, which correlates with several of these but substitutes for none of them. A high score means a model uses your surface well; it does not certify that your API is safe for autonomous agents. --- ## 6. Contesting a score A score is a measurement under a stated configuration, not a verdict. If you believe one is wrong: 1. Read the report's `judge` block and confirm which configuration produced it. 2. Re-run against your own judge: `gradable scan --target --judge-cli --judge-model `. 3. If results differ materially, that is a finding about judge sensitivity and we want it — open an issue with both reports attached. Scans can fail for reasons that are not the server's fault — ingestion timeouts, malformed judge output, argument-size limits. Failures are recorded as `failed.json` and are **not** scored or published as a grade. --- ## 7. Versioning This document describes methodology **v1**. Changes that move published scores will be recorded here with the commit that introduced them, so a score can always be traced to the rules in force when it was produced.