Public assessment

ariekogan/ateam-mcp (@ateam-ai/mcp)

ateam-ai-mcp · v0.3.0 · scanned Aug 3, 2026

What changed in the harness

Selection accuracy 100→100, token cost up 1%, unconfirmed writes 0%→0%.

Category breakdown

Where the score comes from.

Earned points across the four signals Gradable measures. Safety and Legibility are scored out of 30; Economics and Discoverability are scored out of 20.

01Safety

0.0 / 30

0.0 out of 30

02Legibility

20.3 / 30

20.3 out of 30

03Economics

10.3 / 20

10.3 out of 20

04Discoverability

10.8 / 20

10.8 out of 20

Highest-impact fix

Estimated gain +30 points

Add explicit identity and permission preflight tools

Expose machine-readable principal/tenant confirmation and a non-mutating permission check so agents can verify both before destructive actions.

Description evidence

Defects and rewrites.

34 defects found across the exposed tool descriptions. Suggested rewrites make purpose, inputs, boundaries, and returns easier for an agent to understand.

Tool	Defect types	Suggested rewrite
`ateam_auth`	no_return_description	Authenticate with A-Team. Required before any tenant-aware operation (reading solutions, deploying, testing, etc.). The user can get their API key at https://mcp.ateam-ai.com/get-api-key. Only global endpoints (spec, examples, validate) work without auth. IMPORTANT: Even if environment variables (ADAS_API_KEY) are configured, you MUST call ateam_auth explicitly — env vars alone are not sufficient. For cross-tenant admin operations, use master_key instead of api_key. Returns an authentication confirmation for the supplied credentials (api_key or master_key), reporting whether authentication succeeded and establishing the tenant scope for subsequent tenant-aware calls.
`ateam_get_examples`	no_return_description	Get complete working examples that pass validation. Study these before building your own. Returns the requested complete example definition(s) that pass validation, or an index of all available examples when type='index'.
`ateam_spec_search`	no_return_description	Semantic search over the FULL ateam platform /spec documentation — the deep fallback behind ateam_design_advisor. Ask a natural-language 'how do I…' question and get the most relevant doc chunks (with their topic + heading), then read the full topic via ateam_get_spec(topic). Use this when the advisor's pointer isn't enough, or for details/examples on anything — including topics outside the curated capability list. Returns the top_k most relevant doc chunks ranked by relevance, each with its topic and heading (or an empty result if nothing matches). Read-only.
`ateam_build_and_run`	no_return_description	DEPLOY THE CURRENT MAIN BRANCH TO A-TEAM CORE. ⚠️ HEAVIEST OPERATION (60-180s): validates solution+skills → deploys all connectors+skills to Core (regenerates MCP servers) → health-checks → optionally runs a warm test → auto-pushes to GitHub. This tool ALWAYS deploys the `main` branch — there is no `ref` parameter. To deploy in-progress dev work, first promote it (ateam_github_promote). AUTO-DETECTS GitHub repo: if you omit mcp_store and a repo exists, connector code is pulled from main automatically; first deploy requires mcp_store. For small changes prefer ateam_patch (faster, incremental). Requires authentication. Returns the full deploy report: validation outcome, deployment and health-check status, the warm-test execution result (when test_message is provided), and the auto-push confirmation.
`ateam_test_skill`	params_unexplained	Send a test message to a deployed skill and get the execution result. Parameters: solution_id is the ID of the deployed solution that owns the skill being tested (tells the system which solution's skill to invoke). Wait modes (wait_for): • 'root' (default, back-compat) — wait until the message's root job completes, return single-job result. Fast, ignores any sub-skills the root delegated to via askAnySkill. • 'chain' — wait until EVERY job in the chain (root + handoffs + askAnySkill subcalls, recursively) reaches a terminal state, then return the full chain tree. Use when testing multi-skill flows (orchestrator → workers, builders → sub-builders, etc.). The response.chain field carries chainJobs[] with parentJobId/relation/depth and executionSteps[] with tool-nesting (opId/parentOpId/_toolDepth). Legacy: wait:false is equivalent to wait_for:'never' — returns job_id immediately for polling via ateam_test_status. wait:true is the same as the default wait_for:'root'.
`ateam_conversation`	params_unexplained	Send a chat message to a deployed solution. No skill_id needed — the system auto-routes to the right skill. Parameters: solution_id is the ID of the deployed solution to send the message to (resolves which solution's skills are used for auto-routing). ALWAYS ASYNC: returns a chain_id immediately — the assistant's reply is NOT in this response (a conversation can run for minutes across handoffs + subcalls, so a synchronous wait would hit the 100s edge timeout → 524). POLL BY CHAIN, NEVER BY JOB: an individual job can terminate while the chain is still running, so poll ateam_chain_status(chain_id) on a loop (~2s) and stop when chain_done === true (or pending_question is set — the assistant is waiting on the user). Use ateam_get_chain(chain_id) only ONCE at the end if you want the full tree / per-job detail — it's too heavy to loop on. Multi-turn: pass the actor_id from a previous response back in to continue the same thread (e.g. reply to a confirmation prompt). Each call starts a new chain; the same actor_id maintains conversation context.
`ateam_test_pipeline`	params_unexplained	Test the decision pipeline (intent detection → planning) for a skill WITHOUT executing tools. Returns intent classification, first planned action, and timing. Use this to debug why a skill classifies intent incorrectly or plans the wrong action. Parameters: solution_id is the ID of the deployed solution that owns the skill being tested.
`ateam_test_voice`	params_unexplained	Simulate a voice conversation with a deployed solution. Runs the full voice pipeline (session → caller verification → prompt → skill dispatch → response) using text instead of audio. Returns each turn with bot response, verification status, tool calls, and entities. Use this to test voice-enabled solutions end-to-end without making a phone call. Parameters: solution_id is the ID of the deployed solution whose skill(s) run the simulated voice conversation.
`ateam_patch`	params_unexplained	Surgically update ANY field in a skill or solution definition, redeploy, and optionally re-test — all in one step. Parameters: solution_id is the ID of the solution being patched (its own definition when target='solution', or the solution that owns the skill when target='skill'). ⚠️ MERGE-BY-DEFAULT (v0.4.0) — Arrays are protected from silent replace. Bare array writes on solution.linked_skills / ui_plugins / platform_connectors / handoffs / grants / triggers (etc.) and skill.tools / connectors / handoffs / scenarios are REFUSED to prevent sibling loss. Add or remove items with the _push / _delete / _update suffixes; opt into a full-array replace only when you really mean it. OPERATIONS (safe by construction): 1. Scalar (dot notation): { "problem.statement": "new value", "role.persona": "You are..." } 2. Deep nested: { "intents.thresholds.accept": 0.9, "policy.escalation.enabled": true } 3. Array APPEND: { "tools_push": [ { name: "new_tool", description: "..." } ] } 4. Array REMOVE: { "tools_delete": ["tool_name"] } 5. Array MODIFY-ONE: { "tools_update": [ { name: "existing_tool", description: "updated" } ] } 6. Full-array REPLACE (opt-in): { "linked_skills": [...], "linked_skills_replace": true } — or { _replace: true, ... } to opt every array in this call. PREVIEW BEFORE WRITING: pass dry_run:true to see the diff (arrays_merged, arrays_replaced, dropped_ids, added_ids) without applying.
`ateam_get_solution`	params_unexplained no_return_description	Read solution state — definition, skills, health, status, or export. Use this to inspect deployed solutions. Parameters: solution_id is the ID of the deployed solution to read. Returns a serialized 'content' blob holding the requested view ('definition', 'skills', 'health', 'status', 'export', 'validate', or 'connectors_health'); when the result would exceed the ~50KB output cap, page it via offset/limit and follow the response's _paging.next_offset (null when done), concatenating the content slices, then JSON.parse.
`ateam_list_solutions`	no_return_description	List all solutions deployed in the Skill Builder. Returns a list of the deployed solutions with their identifiers and identifying summary information, so callers can pick a solution_id for other tools.
`ateam_delete_solution`	no_return_description	⚠️ IRREVERSIBLE — kills Mongo state, running MCP processes, and Builder FS for the whole solution and every skill. REQUIRES `confirm:true` AND `confirm_solution_id` echoing the solution id you're destroying (defeats typos and hallucinated ids). RECOVERY: the GitHub repo is untouched; `ateam_github_pull` rebuilds the solution from `main`. Prefer that over re-deploying from memory. Returns a confirmation that the solution and all its skills were deleted, together with the ateam_github_pull recovery hint.
`ateam_delete_skill`	no_return_description	⚠️ IRREVERSIBLE in Core + Builder FS — kills the running MCP process, unregisters from skill registry, deletes the Mongo record, drops from solution.skills[] and solution.linked_skills, and removes the skill's files from Builder FS. REQUIRES `confirm:true`. RECOVERY: the skill still lives in GitHub — `ateam_github_pull` rebuilds the whole solution (no per-skill restore path). Returns a confirmation that the skill was removed from Core and Builder FS and dropped from the solution's skill lists, with the git-recovery path noted.
`ateam_delete_connector`	no_return_description	⚠️ CASCADING — any skill whose engine.bootstrap_tools or tools[] name a tool from this connector will FAIL its next execution. Stops and deletes the connector from A-Team Core; drops references from the solution definition (grants, platform_connectors, ui_plugins ids starting `mcp:<connector-id>:*`) and skill definitions (connectors array); cleans up mcp-store files. GitHub source is preserved — a follow-up `ateam_build_and_run(github:true)` can resurrect. REQUIRES `confirm:true`. Returns a confirmation that the connector was stopped and removed, including the cascade warning for any skills referencing its tools and the note that GitHub source is preserved.
`ateam_show_skill_minimal`	params_unexplained no_return_description	Show the minimal authoring view of a skill — persona + connectors + handoff_when + style + policy guardrails only. ~10× smaller than ateam_get_solution(view:'skills') for the same skill. Use this when you only need the irreducible author content (Phase 9 of the strip). Parameters: solution_id is the ID of the solution that owns the skill; skill_id is the ID of the skill to show. Returns the slim authoring view containing only those author fields.
`ateam_show_solution_minimal`	params_unexplained	Show the minimal authoring view of a solution — name + description + style + routing_mode + identity_mode + skill ids + connector ids only. Skips deployed metadata, handoffs (auto-generated), grants, ui_plugins, validation results. Use this for fast inspection without the verbose fields (Phase 9 of the strip). Parameters: solution_id is the ID of the solution to inspect.
`ateam_create_connector`	params_unexplained no_return_description	Scaffold a new MCP connector with server.js + package.json + README. Eliminates ~50% of identical boilerplate (MCP server setup, tool registration, stdio transport). You then fill in the tool implementations. Set ui_capable=true to include ui.listPlugins / ui.getPlugin stubs (plugin source files added separately via ateam_create_plugin). After scaffolding, the files are uploaded to Core via the same path as ateam_upload_connector. Parameters: solution_id is the ID of the solution the new connector belongs to. Returns the scaffold upload result to Core, confirming the connector was registered with its starter files.
`ateam_create_plugin`	params_unexplained no_return_description	Scaffold a UI plugin (iframe HTML, React Native TSX, or both) inside an existing connector. Eliminates ~50% of identical plugin boilerplate (imports, theme/bridge hooks, postMessage protocol, default export shape). You then fill in the component body. Use kind='iframe' for web-only, 'rn' for mobile-only, 'adaptive' for both. Also writes ui-dist/<plugin>/manifest.json with the required render block. ⚠️ RENDERING IS NOT AUTOMATIC. At deploy, Phase 5 discovers plugins by calling each connector's ui.listPlugins + ui.getPlugin — a plugin only appears (and renders) if the connector ADVERTISES it there with a render.{mode, iframeUrl?, reactNative?} block. Dropping the scaffold files alone does NOT register it. If the connector generates its plugin list from ui-dist/<plugin>/manifest.json, the emitted manifest is picked up automatically; if the connector has a HARDCODED list (e.g. personal-assistant-ui-mcp: UI_PLUGINS[] + PLUGIN_MANIFESTS{} in server.js), you MUST add this plugin there (copy the render block from the manifest.json). Verify after deploy with ateam_get_solution(solution_id, 'connectors_health') or ateam_get_widget_catalog. Then declare it at solution ui_plugins[] so a skill can open it via sys.focusUiPlugin. The scaffold MERGES into the existing connector (server.js + other files preserved). Parameters: solution_id is the ID of the solution whose connector receives the plugin. Returns the plugin scaffold files created inside the connector (including ui-dist/<plugin>/manifest.json).
`ateam_upload_connector`	params_unexplained	Upload connector code to Core and restart — WITHOUT redeploying skills. Parameters: solution_id is the ID of the solution that owns the connector being uploaded. MERGES with the GitHub state at `ref` by default (default ref: 'dev'). Sending a partial file set ONLY overlays those files — the rest of the connector is preserved from GitHub. To fully replace the connector dir (historical behavior), pass replace:true. Modes: • github:true (no files) — deploy the GitHub state at `ref` as-is. • github:true + files:[] — GitHub state at `ref` as BASE, your files overlay on top (incoming wins). • files:[] (no github) — default MERGE with GitHub state at `ref`. Refuses if no GitHub base exists (no silent nuke). • files:[] + replace:true — full replace. Wipes connector dir + writes only the provided files. Use deliberately. Multi-file connectors (server.js + dashboard HTML + RN bundle + package/manifest): pass each file with content_base64 (a single-line, escape-safe base64 string) instead of content — so you don't hand-escape ~90KB of HTML/JS/JSON inside one tool call. This is the CANONICAL agent path for a full connector; do NOT hand-roll `curl` against the raw endpoint (that skips connector registration / PAT provisioning).
`ateam_test_status`	params_unexplained	Poll the progress of an async skill test. Returns iteration count, tool call steps, status (running/completed/failed), and result when done. Parameters: solution_id is the ID of the solution that owns the test; skill_id is the ID of the skill being tested. Set include_chain:true to ALSO include the full chain tree (every job in the chain, rooted at this job_id, with parent/child linkage). Use when this job dispatched askAnySkill subcalls and you want a single snapshot of the whole multi-skill state instead of polling each child job_id separately.
`ateam_test_abort`	params_unexplained no_return_description	Abort a running skill test. Stops the job execution at the next iteration boundary. (Advanced.) Parameters: solution_id is the ID of the solution that owns the test; skill_id is the ID of the skill being tested. Returns a confirmation that the job was aborted at the next iteration boundary.
`ateam_test_connector`	params_unexplained	Call a tool on a running connector and get the result. Use this to test individual connector tools (e.g., triggers.list, entities.list, google.command) without deploying to a client. The connector must be connected and running. Parameters: solution_id is the ID of the solution that owns the running connector.
`ateam_github_push`	no_return_description	Push the current deployed solution to GitHub. Auto-creates the repo on first use. Commits the full bundle (solution + skills + connector source) atomically. Use after ateam_build_and_run to version your solution, or anytime you want to snapshot the current state. Returns a confirmation of the commit and push (including whether the repo was auto-created).
`ateam_github_pull`	no_return_description	Deploy a solution FROM its GitHub repo. Reads .ateam/export.json + connector source from the repo and feeds it into the deploy pipeline. Use this to restore a previous version or deploy from GitHub as the source of truth. Returns the deploy status (success/failure) of the state pulled from the repo.
`ateam_github_status`	params_unexplained no_return_description	Check if a solution has a GitHub repo, its URL, and the latest commit. Use this to verify GitHub integration is working for a solution. Parameters: solution_id is the ID of the solution whose GitHub integration is being checked. Returns whether the repo exists, the repo URL, and the latest commit (or an error if no repo is configured).
`ateam_github_read`	params_unexplained	Read any file from a solution's GitHub repo. Returns the file content. Use this to read connector source code, skill definitions, or any versioned file. Default reads from `main` (deployed/prod state). Pass `ref: 'dev'` to read in-progress work. Parameters: solution_id is the ID of the solution whose GitHub repo the file is read from.
`ateam_github_patch`	params_unexplained	Edit a file in the solution's GitHub repo and commit. Two modes: 1. FULL FILE: provide `content` — replaces entire file (good for new files or small files) 2. SEARCH/REPLACE: provide `search` + `replace` — surgical edit without sending full file (preferred for large files like server.js). Always use search/replace for large files (>5KB). Always read the file first with ateam_github_read to get the exact text to search for. DEFAULTS TO `dev` BRANCH — writes don't touch prod. Use ateam_github_promote to ship dev→main when ready. Pass ref:'main' only for emergency hotfixes. Parameters: solution_id is the ID of the solution whose GitHub repo is being edited.
`ateam_github_write`	params_unexplained no_return_description	Write a file to the solution's GitHub repo. Use this to create new connector files or replace existing ones — one file per call. This is the PRIMARY way to write connector code after first deploy. Write each file individually (server.js, package.json, UI assets), then call ateam_github_promote() to ship to prod (dev→main), then ateam_build_and_run() to deploy. DEFAULTS TO `dev` BRANCH. Parameters: solution_id is the ID of the solution whose GitHub repo is being written to. Returns a confirmation that the file was written and committed.
`ateam_github_log`	params_unexplained no_return_description	View commit history for a solution's GitHub repo. Default reads from `main` (prod). Pass `ref: 'dev'` to see in-progress work. Parameters: solution_id is the ID of the solution whose commit history is requested. Returns the recent commit entries with their messages, SHAs, timestamps, and links.
`ateam_github_diff`	params_unexplained	PRE-FLIGHT BEFORE PROMOTE. Compares `dev` (head) vs `main` (base) by default — shows exactly which commits and files are about to ship if you call ateam_github_promote() next. Use this when you want to: • Review changes before promoting to prod • See if dev is ahead of main at all (returns ahead_by: 0 if nothing to promote) • Inspect arbitrary branch/tag/commit comparisons (override base/head). Parameters: solution_id is the ID of the solution whose branches/refs are being compared.
`ateam_github_promote`	params_unexplained	SHIP DEV TO PROD. Merges the `dev` branch into `main` and auto-tags the new main HEAD as safe-YYYY-MM-DD-NNN. Use after testing your dev work, when you're ready to deploy changes to production. Workflow: 1) ateam_github_patch (writes to dev) → 2) ateam_github_promote (merges dev→main) → 3) ateam_build_and_run (deploys main). Pass dry_run:true to see what's about to ship without merging. On merge conflict the call returns 409 — resolve manually on GitHub (open a PR or use the web UI), then retry. Parameters: solution_id is the ID of the solution whose dev branch is merged into main.
`ateam_github_rollback`	params_unexplained no_return_description	Roll prod (`main` branch) back to a previous state. ADDITIVE — does NOT destroy history. Creates a new commit on top of main whose tree matches the target's tree. The history of everything between target and current main is preserved (you can roll back the rollback). Workflow: 1) ateam_github_list_versions (find a safe-* tag) → 2) ateam_github_rollback(target: 'safe-...') → 3) ateam_build_and_run (deploys the reverted state). Parameters: solution_id is the ID of the solution whose main branch is being rolled back. Returns a confirmation of the additive revert commit created on main.
`ateam_github_list_versions`	params_unexplained no_return_description	List all available checkpoints (safe-* tags) for a solution. Use before rollback to see available safe points. Parameters: solution_id is the ID of the solution whose checkpoints are listed. Returns the available checkpoints with their tag name, date, counter, and commit SHA.
`ateam_redeploy`	no_return_description	Re-deploy skills WITHOUT changing any definitions. ⚠️ HEAVY OPERATION: regenerates MCP servers (Python code) for every skill, pushes each to A-Team Core, restarts connectors, and verifies tool discovery. Takes 30-120s depending on skill count. Use after connector restarts, Core hiccups, or stale state. For incremental changes, prefer ateam_patch (which updates + redeploys in one step). Returns the redeploy status: per-skill MCP server generation/registration results, connector restart status, and tool-discovery verification outcome.

Selection evidence

Confusable tool pairs.

10 pairs where similar names or overlapping descriptions may send an agent toward the wrong tool.

Tool A	Tool B	Confidence	Why they collide
`ateam_patch`	`ateam_github_patch`	medium	Both are 'patch' tools on the same solution; a task like 'apply this patch to the solution' is ambiguous between updating the solution/skill definition (ateam_patch, which redeploys) and editing a repo file with a commit (ateam_github_patch, which writes to the dev branch without deploying).
`ateam_verify`	`ateam_verify_consistency`	medium	Nearly identical names and a generic 'verify the solution' task is ambiguous: ateam_verify checks the real runtime end-state (connectors/widgets/skills) while ateam_verify_consistency only checks Builder-FS vs GitHub file sync — an agent could pick either for a plain 'make sure the solution is in good shape' request.
`ateam_github_patch`	`ateam_github_write`	medium	Both create or update a single file in the solution's GitHub repo; 'overwrite/update this file in the repo' maps validly to both, since patch's full-file mode is functionally identical to write, differing only in search/replace capability and default branch semantics.
`ateam_github_push`	`ateam_github_write`	medium	Both put content into GitHub; a task like 'save/commit my changes to the repo' is ambiguous between snapshotting the whole deployed bundle atomically (push) and writing one individual file to the dev branch (write).
`ateam_get_spec`	`ateam_spec_search`	medium	Both retrieve spec content; a natural task like 'look up how to do X in the spec' maps plausibly to either get_spec(search=...) or spec_search(query=...), since get_spec's search param overlaps semantically with the semantic-search tool's query interface.
`ateam_get_chain`	`ateam_chain_status`	medium	Both inspect a chain given an id; 'check on the chain' is ambiguous between the slim whole-chain aggregate poll (chain_status) and the full chain-tree analysis (get_chain) — the descriptions themselves explicitly warn against using get_chain for polling, signaling real selection risk.
`ateam_github_pull`	`ateam_github_read`	medium	Both fetch from GitHub; 'pull the latest from the repo' is ambiguous between deploying/restoring the solution from the repo (pull) and reading a single file's content (read), especially since read defaults to the deployed main branch state.
`ateam_show_skill_minimal`	`ateam_show_solution_minimal`	low	A vague task like 'show me the minimal authoring view' without naming whether the entity is a skill or a solution could route to either; typically the user names one, keeping confusion low.
`ateam_github_diff`	`ateam_github_promote`	low	A review-style task such as 'show me what would change before shipping to prod' could trigger either diff (the documented pre-flight) or promote(dry_run:true), which also returns the about-to-ship diff; verb choice usually disambiguates.
`ateam_delete_skill`	`ateam_delete_connector`	low	Both are irreversible, confirm-flagged deletes of a component within a solution; a terse task like 'delete the <id> component' where the id doesn't reveal the kind could pick the wrong target, but explicit skill/connector wording normally disambiguates.

Compare the field

One score is useful.
The evidence makes it actionable.

Back to the leaderboard