Public assessment

chessceo/chessceo-mcp (@chessceo/mcp)

chessceo-mcp · v0.1.0 · scanned Aug 3, 2026

What changed in the harness

Selection accuracy 98→96, token cost up 3%, unconfirmed writes 0%→0%.

Category breakdown

Where the score comes from.

Earned points across the four signals Gradable measures. Safety and Legibility are scored out of 30; Economics and Discoverability are scored out of 20.

01Safety

0.0 / 30

0.0 out of 30

02Legibility

20.0 / 30

20.0 out of 30

03Economics

8.4 / 20

8.4 out of 20

04Discoverability

12.2 / 20

12.2 out of 20

Highest-impact fix

Estimated gain +30 points

Add explicit identity and permission preflight tools

Expose machine-readable principal/tenant confirmation and a non-mutating permission check so agents can verify both before destructive actions.

Description evidence

Defects and rewrites.

23 defects found across the exposed tool descriptions. Suggested rewrites make purpose, inputs, boundaries, and returns easier for an agent to understand.

Tool	Defect types	Suggested rewrite
`get_prep_position`	params_unexplained	Query one position within a prep session created by `prepare_opponent`. Returns move statistics (frequency + win rate + last-played date per move) plus the actual games played from that position, in one call. Position input: prefer `file_id`+`node_id` when inside a prep file (server derives FEN from the tree). Otherwise pass `fen`. `limit` caps the number of games returned (default 10, max 50); `offset` is a pagination index into the game list — combine with `limit` to page through more than one page of games. AUTO-EVAL: if a cloud combo instance is running, the response includes `.eval` (Stockfish + Lc0 read at the position) so you don't need a separate cloud_analyse. Reading the response — CRITICAL: • Win % is one weight, not a verdict. Sample size matters (3 games at 66% is noise; 300 at 55% is signal). • Prep is symmetric information — both sides see the same history. Assume the opponent knows the weakness you spotted. • Recency > career. The last 12-24 months dominate — filter your session with `start_month` if the player's repertoire shifted. • Opponent will deviate early. Prep is a tree — cover the 2 most likely replies at each real branching point, not one 20-move line. For the full guide call `read_opening_prep_guide`.
`delete_prep_session`	no_return_description	Delete one prep session by token. Free-form cleanup — sessions do expire automatically, but this is useful when you're done with one or when you want to force a rebuild after upstream data changed. Returns a confirmation that the session was deleted.
`get_head_to_head`	params_unexplained	Complete head-to-head record between two players. Includes overall and per-colour W/D/L (from player A's perspective), splits by time control, most-played openings between them, first / last meeting, average game length, and the game list. `limit` caps how many games of the pairing appear in the returned game list (max 10); `offset` pages further into that list when there are more games than the limit.
`list_live_tournaments`	no_return_description	Returns the tournaments currently being broadcast live on chess.ceo, each with its tournament id (pass `tour_id` to `list_tournament_players` to get the entrants). Use this when the user asks 'what's on right now' / 'live tournaments today'.
`list_tournament_players`	no_return_description	Returns the players participating in one live-broadcast tournament, identified by `tour_id` (from `list_live_tournaments`).
`list_player_live_tournaments`	no_return_description	Returns the currently-live broadcasts a given player is competing in, identified by `fide_id` (from `search_player`). Use when the user asks 'is X playing anywhere right now'.
`start_cloud_engine`	no_return_description	Rent a combo GPU instance (Stockfish + Lc0 in the same container) on the user's chess.ceo account. Real money — billed per second while running. Returns the started instance's contract id (pass it to `stop_cloud_engine`) and its status. CRITICAL: `machine_type` must be an exact SKU from `list_cloud_machine_options` (e.g. 'rtx-5090-64', NOT 'rtx-5090'). Guessing SKUs will fail. Call list_cloud_machine_options first, show the user the display names + prices, get their confirmation, then pass the SKU here. Use list_cloud_engines first to check if the user already has one running; don't start a second combo unless the user asked for it. Requires an MCP token with agent access.
`list_cloud_engines`	no_return_description	Returns the user's currently running cloud engines, including each instance's contract_id (pass to `stop_cloud_engine`). Use before starting a new one: `cloud_analyse` auto-picks the only running combo, so listing is only necessary when the user might have zero or several.
`stop_cloud_engine`	no_return_description	Destroy a running cloud engine. Billing stops immediately. Use the `contract_id` from `list_cloud_engines` — don't guess. Returns a confirmation that the engine was stopped.
`search_prep_files`	no_return_description	Text search over the user's prep files ACROSS ALL their collections (matches PGN headers, comments, and content). Returns the matching prep files with their ids and PGN header metadata, ready to pass to `read_prep_file` or the mutation tools. Use when you know a keyword — e.g. search_prep_files(query='Firouzja') or search_prep_files(query='Najdorf'). Cheaper than paging list_collections + list_prep_files to find one file by name.
`delete_prep_file`	no_return_description	Soft-delete a prep file. Fully reversible: the file lands in the user's recycle bin, from which either the user (via app UI) OR you (via `restore_prep_file`) can bring it back. Permanent delete is intentionally NOT exposed on this surface — the user has to permanent-delete from the app themselves. Returns a confirmation that the file was moved to the recycle bin. Rare — usually you extend or edit instead.
`restore_prep_file`	no_return_description	Restore a previously soft-deleted prep file. Undo the `delete_prep_file` call — the file comes back with a fresh `game_number` (its slot in the collection is not preserved) and its full history intact. Returns a confirmation of the restoration. Use when you realize mid-session you shouldn't have deleted something; the LLM can fix its own mistake without asking the user to open the app.
`add_line`	params_unexplained	Append a linear sequence of moves under `parent_id`. Each SAN in the list becomes the mainline child of the previous — one call instead of N add_move calls for a straight variation. If the parent already has other children, this whole line is appended as a variation (promote_variation the first move if you want it as the mainline). Anti-pattern: pasting an engine PV as a single long `add_line`. Real prep is a tree, not a line. Almost every position along a variation has more than one plausible move — pasting a 12+-ply engine PV without branching at those points is the #1 documented failure mode of this MCP: it produces a page that reads as prep but ignores every decision the opponent actually gets to make. Long unbranched lines get a warning field in the response starting at ~9 plies and a strong warning at 14+ plies. Rule of thumb: if you added ≥8 plies in one call, at least half of them should have branched. Genuine exceptions exist (forced mates, obligated exchange sequences) — in those cases add a comment naming what makes the sequence forced (`{Every move here is forced by the mate threat.}`), so the reader knows it's forced by chess, not by LLM laziness. `expected_version` is an optimistic-lock check: pass the file `version` from your last read (read_prep_file or a prior mutation response); a mismatch means the file changed and the call errors instead of overwriting. Auto-saves. Returns `{node_id, line: [{node_id, san}, ...], version}` — `node_id` is the last (leaf) node's id, `line` is every node created in order so you can address any of them next. When long-and-linear, also includes `warning: "..."`.
`set_comment`	params_unexplained	Set (or clear, with empty string) the text comment on the node identified by `node_id`. Comments are for plans, prep-signal, and interpretation the app can't derive — NOT for describing moves that should be variations instead. `expected_version` is an optimistic-lock check: pass the file `version` from your last read; a mismatch means the file changed and the call errors instead of overwriting. Auto-saves. Two guardrails fire in the response as `warnings: [...]`: 1. Content scan — comments containing spread lists (`≈50, ≈42, …`), raw centipawn values (`≈−60`, `+0.35`, `at depth 24`), or roster restatement (`146 GM games — Nakamura, …`) are all restating what the app already renders. The warning names the fix (set the NAG and drop the number; label the character not the numbers; cite a specific game instead of a count). 2. Ungrounded prose — substantive comments (≥40 chars) on a node whose position was never passed to `describe_position` this session are prone to hallucinated structural claims (piece on wrong square, invented captures, misidentified pawn structure). Call `describe_position` with `file_id`+`node_id` BEFORE writing prose about the position; the same node's warning clears once the position is described.
`set_nags`	params_unexplained no_return_description	Replace the list of NAGs on the node identified by `node_id`. Empty array clears them. `id` is the prep file id; `node_id` is the target node (from read_prep_file / add_move); `expected_version` is an optimistic-lock check — pass the file `version` from your last read. NAGs are your EDITORIAL call — see read_pgn_authoring_guide for the discipline (novelty $146, sharp choice $5, decisive $18/$19, etc.). Do NOT set $10 '=' on every equal position; that's board noise. Auto-saves. Returns `{node_id, version}` for the updated node.
`set_annotations`	params_unexplained no_return_description	Replace the visual annotations (arrows + coloured squares) on the node identified by `node_id`. Passing empty arrays clears them. `id` is the prep file id; `node_id` is the target node. `arrows` are `{color, from, to}` square-to-square arrows, `highlights` are `{color, square}` coloured squares; colours are green, red, yellow, light-blue, dark-blue, orange. Keep it LIGHT: 1-3 arrows and 2-3 squares per move maximum. Twenty arrows is noise, not signal. `expected_version` is an optimistic-lock check — pass the file `version` from your last read. Auto-saves. Returns `{node_id, version}`.
`delete_subtree`	params_unexplained no_return_description	Delete the node identified by `node_id` and all its descendants. `id` is the prep file id; `expected_version` is an optimistic-lock check — pass the file `version` from your last read. Refuses to delete the root. Auto-saves. Returns a confirmation with the new file `version`.
`promote_variation`	params_unexplained no_return_description	Make the node identified by `node_id` its parent's mainline (children[0]), demoting the current mainline (and any other siblings) into variation order. Silently no-op if already the mainline. `id` is the prep file id; `expected_version` is an optimistic-lock check — pass the file `version` from your last read. Cannot be the root. Auto-saves. Returns a confirmation with the new file `version`.
`apply_mutations`	params_unexplained	Batch: apply a list of mutations in one call. One load-parse-mutate-export-save cycle for N ops, so building a 100-move repertoire costs one HTTP round-trip and one save instead of 100. This is the RIGHT way to build a file — use single mutations only for surgical follow-up edits. `id` is the prep file id; `expected_version` is an optimistic-lock check — pass the file `version` from your last read. `mutations` is the list of ops to apply: each is `{op, node_id \| parent_id, ...args}` where `op` is one of: add_move, add_line, set_comment, set_nags, set_annotations, delete_subtree, promote_variation, set_tag. Same arg shape as the individual tools. Ops apply in order; because node ids are content-derived (hash of parent_id + san), a node created by an early op has a deterministic id you can reference in later ops in the same batch. Any op error aborts the batch (nothing saved). Response is `{ok, results: [{node_id, line?}], version}` — one entry per op with the id it landed on (add_line also returns the full line array).
`auto_evaluate`	params_unexplained	Walk the tree from `node_id` (default `'r'` = whole file) and populate the persistent `ceoEval` on every descendant via cloud_analyse. Requires a running cloud combo instance. `id` is the prep file id. Async job — returns immediately. Response: `{ job_id, target_count, status: 'running', estimated_seconds }`. Then poll `auto_evaluate_status(job_id)` until `done: true`. Cancel a run with `auto_evaluate_cancel(job_id)` — partial progress is preserved. Do useful other work between polls (write more of the tree, walk the opponent's repertoire) — the engine runs in the background. Progress is checkpointed to the prep file every 8 successfully-evaluated nodes, so a cancel / crash / MCP restart mid-run leaves the tree partially populated rather than losing everything. On MCP restart the job record disappears; re-run auto_evaluate and `only_missing=true` naturally skips what was already saved. Does NOT set visible NAGs. NAG placement is your call, not the engine's — an opening tree full of 0.00 positions doesn't need a `$10` (=) glyph on every move. Use quote_engine_eval on individual nodes before writing prose that references engine numbers. Costs real money — one cloud_analyse per node. A 200-node walk at default movetime is ~5 min of engine time (calls serialise on the per-combo semaphore in the backend).
`set_tag`	params_unexplained	Set or clear a game-level PGN tag (Event, Site, Date, White, Black, Result, or any custom tag). Passing empty string removes the tag. `id` is the prep file id; `expected_version` is an optimistic-lock check — pass the file `version` from your last read. Auto-saves. Returns a confirmation with the new file `version`.
`read_opening_prep_guide`	no_return_description	Returns the chess.ceo opening-prep guide — the full preparation philosophy to follow when preparing against an opponent. CALL WHEN: the user asks about OPENING PREPARATION — 'prep me against X', 'what should I play vs the Najdorf', 'help me build a repertoire against 1.e4', 'walk this opponent's Sveshnikov'. This guide is chess-and-analysis philosophy, not storage semantics. Covers: why win% is one weight not a verdict, why prep is a two-player game with symmetric information (opponent sees your history too), how sample size and recency change the reading, when 'revealed weaknesses' are actionable vs already patched, how to choose between the GM-classical DB and the main DB, when to combine chesscom/lichess sources with FIDE, the three chess.com profile shapes (consistent / eclectic / split-personality), the reversed-colours scarcity trick, how to calibrate surprise (rare secondary lines inside the existing repertoire, not big first-move switches). Different tool: `read_prep_files_guide` covers the FILE STORAGE feature (how to list/create/save prep files) — call that only when about to manipulate files, not for opening questions.
`read_prep_files_guide`	no_return_description	Returns the guide to the prep FILE STORAGE feature — how to create, list, save, and delete prep files. CALL WHEN: you're about to CREATE, LIST, SAVE, or DELETE a prep file — the persistent file storage feature. Not for opening prep philosophy (that's `read_opening_prep_guide`) and not for how to write PGN (that's `read_pgn_authoring_guide`). Covers: the AI Prep folder, when to list vs search vs create (avoid duplicate 'Prep vs Firouzja' files), optimistic locking with `version`, naming conventions for the [Event] tag, node-id addressing basics.

Selection evidence

Confusable tool pairs.

16 pairs where similar names or overlapping descriptions may send an agent toward the wrong tool.

Tool A	Tool B	Confidence	Why they collide
`list_live_tournaments`	`list_player_live_tournaments`	medium	Both answer 'live tournaments' phrasing; a task like 'what live tournaments does Carlsen have right now' may make the agent call the unfiltered tournament-list tool instead of the player-scoped one, since the sole distinguishing cue is whether a fide_id is mentioned.
`list_tournament_players`	`list_player_live_tournaments`	medium	Inverse operations with mirror-like names ('players of a tournament' vs 'tournaments of a player'); a task such as 'who is playing in the live tournaments' lacks the direction cue, so the agent can pick the member whose required input (tour_id vs fide_id) it doesn't actually have.
`read_prep_file`	`read_example_prep_files`	low	A task like 'show me a sample prep file' should target read_example_prep_files, but the name read_prep_file lures the agent into reading a user file that it has no id for, or vice versa.
`read_prep_file`	`read_prep_files_guide`	low	A task like 'tell me how to read a prep file' could incorrectly open the guide while 'read the prep file' could fetch the guide document instead of file data; the guide-vs-data distinction is not obvious from the names alone.
`read_opening_prep_guide`	`read_prep_files_guide`	medium	Both are 'prep guides'; a task like 'read me the prep guide' or 'how should I do prep' is genuinely ambiguous between opening-prep philosophy and file-storage mechanics, and the wrong guide returns entirely different content.
`read_prep_files_guide`	`read_example_prep_files`	low	A task like 'what do good prep files look like' maps to the examples but could instead hit the storage guide, or vice versa; only the words 'guide' versus 'example' separate them.
`delete_prep_session`	`delete_prep_file`	low	A task like 'delete that Firouzja prep' is ambiguous over whether 'prep' means the opponent-prep session (identified by session_token) or the stored prep file (identified by id), and the two tools take different required ids.
`find_position_in_courses`	`read_course_at_position`	medium	The find→read pair both involve 'course' and 'position'; a task like 'what does my course say about this position' may jump straight to read_course_at_position without the prerequisite course_file_id from find, or stop at find and never fetch the actual commentary.
`find_position_in_files`	`find_position_in_courses`	medium	Near-identical names and schemas both search the user's positions by FEN/moves; a task like 'where does this position appear in my material' does not say editable prep files vs read-only courses, so the wrong corpus gets searched.
`get_prep_position`	`get_position_stats`	medium	Both return move statistics plus example games at a position; a task like 'how is this position scored' is satisfied by either the opponent-session tool (requires session_token) or the general GM-database tool depending on context, with no surface cue to disambiguate.
`list_cloud_machine_options`	`list_cloud_engines`	low	A task like 'what cloud engines can I use' is ambiguous between the rental catalog (machine options) and the user's currently running instances, since both tools read as 'list cloud ... engine(s)'.
`add_move`	`add_line`	low	A task like 'add this line: e4 e5 Nf3' should use add_line, but repeated add_move calls against the same parent would wrongly create sibling variations instead of a chain; 'add the move(s)' phrasing does not reveal the caller's intent.
`set_nags`	`set_annotations`	low	Both 'set' decorations on a node; a task like 'set the annotations on this move' could replace NAGs when the user meant arrows/highlights, since NAG symbols are also colloquially called annotations.
`cloud_analyse`	`deep_analyse`	low	Both analyse a FEN with Stockfish on the same combo instance; a task like 'analyze this position thoroughly' could start a slow async deep_analyse job when a 2s cloud_analyse was intended, or vice versa, since the descriptions share the analyse/stockfish vocabulary.
`list_nodes`	`list_transpositions`	medium	list_nodes offers a 'transpositions' filter and list_transpositions is the dedicated grouping tool; a task like 'which positions in this file transpose' is served by both, so the agent may pick the one returning the wrong shape (flat id list vs grouped transpositions).
`search_prep_files`	`find_position_in_files`	low	Both search across all the user's collections; a task phrased as 'search my files for this position' supplying moves could hit the text-search tool, which won't match a position, instead of the FEN/moves position search.

Compare the field

One score is useful.
The evidence makes it actionable.

Back to the leaderboard