Public assessment

todah-zg/codemagic-mcp (codemagic-mcp-server)

codemagic-mcp-server · v1.0.5 · scanned Aug 2, 2026

What changed in the harness

Selection accuracy 95→100, token cost up 3%, unconfirmed writes 0%→0%.

Category breakdown

Where the score comes from.

Earned points across the four signals Gradable measures. Safety and Legibility are scored out of 30; Economics and Discoverability are scored out of 20.

01Safety

0.0 / 30

0.0 out of 30

02Legibility

17.5 / 30

17.5 out of 30

03Economics

11.5 / 20

11.5 out of 20

04Discoverability

10.7 / 20

10.7 out of 20

Highest-impact fix

Estimated gain +30 points

Add explicit identity and permission preflight tools

Expose machine-readable principal/tenant confirmation and a non-mutating permission check so agents can verify both before destructive actions.

Description evidence

Defects and rewrites.

39 defects found across the exposed tool descriptions. Suggested rewrites make purpose, inputs, boundaries, and returns easier for an agent to understand.

Tool	Defect types	Suggested rewrite
`list_applications`	no_return_description	List all applications in your Codemagic account. Call this first to get the app IDs needed by all other Codemagic tools. Returns each application's ID, name, and repository details.
`get_build`	params_unexplained	Get full details for a single build, including status, metadata, and artifact download URLs. Pass a build_id returned by trigger_build, wait_for_build, or list_builds. Artifact URLs require x-auth-token — use them with upload_to_testflight, upload_to_google_play, or create_public_artifact_url.
`cancel_build`	no_return_description	Cancel a running or queued Codemagic build. Use when a triggered build is no longer needed — for example if the wrong branch was used or an error was found after triggering. Has no effect on builds that have already finished. Returns the updated build status confirming cancellation.
`add_application`	no_return_description	Add a new application to Codemagic by connecting a Git repository. For HTTPS URLs: if you have connected your GitHub, GitLab, or Bitbucket account via Codemagic Settings → Integrations, private repositories are accessible with just the URL — no credentials needed. For SSH URLs (git@... or ssh://git@...): a fresh Ed25519 deploy key is generated automatically. The private key is stored directly in Codemagic and the public key is added to GitHub automatically if the gh CLI is installed and authenticated, or shown for manual setup otherwise. Note: after adding, the app shows 'Set up build' in the Codemagic UI — this is expected. Returns the newly created application's ID and details.
`create_variable_group`	no_return_description	Create a new variable group in Codemagic. Requires a team_id (personal accounts do not support global variable groups) or an app_id for app-level groups. After creating, add non-secret variables via add_variable, or add secret values directly in the Codemagic UI. Returns the new group's ID and name.
`add_variable`	no_return_description	Add a non-secret variable to a Codemagic variable group. For secret values (API keys, certificates, tokens) use the Codemagic UI instead — secrets should never pass through the agent. Returns the created variable's ID, name, and value.
`update_variable`	no_return_description	Update the name or value of an existing non-secret variable in a Codemagic variable group. The variable_id comes from list_variable_groups. For secret values use the Codemagic UI — secrets should never pass through the agent. Returns the updated variable's ID, name, and value.
`delete_variable`	no_return_description	Delete a variable from a Codemagic variable group. The variable_id comes from list_variable_groups. This cannot be undone. Returns a confirmation that the variable was deleted.
`get_webhook_url`	no_return_description	Get the incoming webhook URL for a Codemagic app. Paste this URL into your Git provider (GitHub, GitLab, or Bitbucket) repository settings to trigger builds automatically on push or pull request events. Returns the webhook URL as a string.
`list_webhooks`	name_restates_behavior no_return_description	Retrieve the webhook subscriptions configured for a Codemagic app, including each webhook's ID, target URL, and subscribed event types. Use the webhook IDs with delete_webhook.
`delete_webhook`	name_restates_behavior no_return_description	Remove a webhook subscription from a Codemagic app so it stops receiving event notifications for that app. Get the webhook_id from list_webhooks. Returns a confirmation that the webhook was deleted.
`list_caches`	no_return_description	List build caches for a Codemagic app. Each cache is scoped to a workflow. Use the cache IDs with delete_cache to free up storage or force a clean build. Returns each cache's ID, workflow name, and size.
`list_asc_apps`	no_return_description	List apps in App Store Connect. Call this first to get the ASC app ID needed by all other App Store Connect tools. Returns each app's ID, name, and bundle identifier.
`list_asc_builds`	no_return_description	List TestFlight builds for an app in App Store Connect. Find the highest version number, increment by 1, and pass that as BUILD_NUMBER in trigger_build variables before triggering a release build. Returns each build's ID, version number, and processing state.
`list_testflight_groups`	no_return_description	List TestFlight beta groups for an app. Use a group name from this list in the beta_group parameter of upload_to_testflight to distribute to testers automatically after upload. Returns each group's ID, name, and whether it is internal or external.
`get_asc_review_status`	no_return_description	Get the current App Store review status for an app. Call this after submitting to the App Store to monitor progress and check for blockers. Returns the current review state and any rejection or blocker details.
`get_asc_release_status`	no_return_description	Get a full release pipeline status dashboard for an app, including latest build, TestFlight, App Store version, and submission state. Returns a structured status object covering each stage of the release pipeline.
`set_version_metadata`	params_unexplained no_return_description	Update App Store version localization metadata — What's New text, description, keywords, support and marketing URLs, and more. What's New is required for every release before submitting for review. support_url and marketing_url are the app's support and marketing web pages shown on the App Store listing. Call once per locale — en-US is the required default; add other locales if the app supports them. Use validate_app_submission afterward to confirm the update resolved the blocker. Returns the updated version metadata.
`set_export_compliance`	no_return_description	Set the export compliance declaration for an iOS build. Required before App Store submission and for TestFlight external distribution. Most apps only use standard HTTPS/TLS — set uses_non_exempt_encryption to false. Only set it to true if the app implements custom or proprietary encryption beyond standard protocols. Defaults to the latest build for the app; pass build_id to target a specific build. Returns a confirmation of the updated compliance setting.
`release_version`	no_return_description	Release an App Store version that has been approved and is waiting in 'Pending Developer Release' state. This immediately makes the update available to all users (or starts the phased rollout if one was configured). Use set_phased_release with action=create before submission if you want a gradual rollout instead of an instant release. Returns the updated version release state.
`set_phased_release`	no_return_description	Manage a phased rollout for an App Store version. Phased rollout gradually releases the update over 7 days: 1% → 2% → 5% → 10% → 20% → 50% → 100%. Actions: 'create' — configure phased rollout before submitting for review. 'pause' — pause an in-progress rollout (use if a critical bug is found after release). 'resume' — resume a paused rollout. 'complete' — immediately release to all remaining users. Returns the current phased release state and rollout percentage.
`submit_beta_review`	no_return_description	Submit a build for TestFlight beta app review. Required before external beta groups can install the build — Apple reviews it once, then all external groups can access it. Internal groups (Apple employees / org members) do not require beta review. Get the build ID from list_asc_builds. Returns the submission's review status.
`add_testflight_tester`	no_return_description	Add a tester to TestFlight by email address. Optionally assign them to a specific beta group — use list_testflight_groups to get group names. The tester receives an invitation email from Apple. Returns the created tester's ID and invitation status.
`create_testflight_group`	no_return_description	Create a new TestFlight beta group for an app. External groups require beta app review before testers can install builds. Internal groups (Apple org members) do not require review — useful for fast internal QA. Returns the new group's ID and name.
`upload_to_testflight`	no_return_description	Download an IPA artifact from Codemagic and upload it to TestFlight via App Store Connect. Optionally distribute to a beta group after upload. Returns the uploaded build's App Store Connect ID and processing state.
`submit_for_app_store_review`	no_return_description	Attach a processed build to an App Store version and submit it for review. The build must have processingState VALID — confirm with list_asc_builds before calling. Requires version metadata (What's New) and export compliance to be set first. Use validate_app_submission to catch blockers before submitting. Returns the submission's review status.
`upload_ios_screenshots`	no_return_description	Download screenshot images from URLs and upload them to App Store Connect for a specific device type and locale. Apple allows up to 10 screenshots per set. Supported formats: PNG (no alpha) and JPEG. Max 10 MB per file. Call list_ios_screenshot_types first to get valid device_type values and required dimensions. Set replace=true to delete existing screenshots before uploading (recommended when refreshing a set). Returns the list of uploaded screenshot IDs and their processing state.
`list_google_play_tracks`	no_return_description	List Google Play tracks (internal, alpha, beta, production) with current release info and version codes. Find the highest versionCode across all tracks, increment by 1, and pass that as BUILD_NUMBER in trigger_build variables before triggering a release build. Returns each track's name, release status, and version codes.
`list_google_play_bundles`	no_return_description	List all uploaded App Bundles (AAB) for an app on Google Play, with their version codes. Use this to audit what has already been uploaded before triggering a new build. Returns each bundle's version code, SHA, and upload date.
`promote_google_play_release`	no_return_description	Promote a release between Google Play tracks (e.g. internal → alpha → beta → production) without re-uploading. Set user_fraction to enable staged rollout on the target track (0.1 = 10% of users). To halt an in-progress staged rollout: set source_track=target_track='production' and release_status='halted'. To resume a halted rollout: same tracks with release_status='inProgress' and a user_fraction. Returns the updated release status on the target track.
`set_rollout_fraction`	no_return_description	Adjust the staged rollout percentage for an existing release on a Google Play track. Use this to gradually expand a rollout (e.g. 10% → 25% → 50% → 100%). Requires the version code of the release currently in the staged rollout. Returns the updated rollout fraction and release status.
`upload_android_screenshots`	no_return_description	Download screenshot images from URLs and upload them to Google Play for a specific language and device type. Google allows up to 8 screenshots per device type. Supported formats: JPEG and 24-bit PNG (no alpha). Max 8 MB per file. Common image types: phoneScreenshots, sevenInchScreenshots, tenInchScreenshots, tvScreenshots, wearScreenshots. Set replace=true to delete all existing screenshots of this type before uploading (recommended when refreshing a set). All uploads are committed atomically — if any upload fails, no changes go live. Returns the list of uploaded screenshot URLs.
`reply_to_google_play_review`	no_return_description	Post or update a developer reply to a Google Play user review. Replies are limited to 350 characters. If the review already has a developer reply, this call replaces it. Get the review_id from list_google_play_reviews. Write a personal, helpful reply — responding to negative reviews improves store ratings and user trust. Returns the posted reply text and timestamp.
`set_android_data_safety`	no_return_description	Submit the data safety declaration for a Google Play app. The declaration describes what data the app collects, how it is used, and whether it is shared. Accepts the raw CSV exported from Play Console → App content → Data safety → Export CSV. Re-upload whenever data practices change (new data type, updated retention policy, etc.). Takes effect immediately — there is no staging step and no GET endpoint to retrieve current labels. Returns a confirmation that the declaration was submitted.
`validate_codemagic_yaml`	name_restates_behavior no_return_description	Check a codemagic.yaml file's contents against the official Codemagic JSON schema and report any structural or field errors before you use it in trigger_build. Returns a validity flag plus a list of any schema violations found.
`get_yaml_template`	no_return_description	Get a starter codemagic.yaml template for a given project type. Templates cover build and signing only — publishing is handled separately via App Store Connect tools. Call list_yaml_template_types to see all valid project_type values. IMPORTANT: Android templates use linux_x2 by default (cheaper, no Mac needed). Personal accounts (no team) cannot use linux_x2 — replace it with mac_mini_m2 for personal accounts. Returns the template's YAML content as a string.
`list_yaml_template_types`	name_restates_behavior no_return_description	Enumerate the project types supported by get_yaml_template (e.g. flutter, ios-native, android-native, react-native), so you can pick a valid project_type value. Returns an array of supported project type strings.
`prepare_release_notes`	no_return_description	Validate localized release notes before submitting to the App Store or Google Play. Checks that each locale is a valid BCP-47 code (e.g. en-US, fr-FR, zh-Hans) and that text fits within platform char limits (Android: 500, iOS: 4000). Pass platform='both' to validate against the stricter Android limit for notes that will go to both stores. Returns per-locale validation results, flagging any invalid codes or over-limit text.
`check_publish_readiness`	no_return_description	Aggregate publish-readiness checks for iOS or Android into a single pass/fail report. API-verifiable checks (valid build, store listing completeness, binary validation) run live. Items that have no API (age rating, privacy labels, content policy, legal agreements) are always listed as 'human required' so nothing is silently skipped. Each item is tagged as 'agent can fix' or 'human required', giving a clear action plan. Call this before submit_for_app_store_review (iOS) or promoting to production (Android). Use first_publish_ios or first_publish_android prompts for the one-time account/app-record setup. Returns a structured checklist of readiness items with pass/fail status and required action.

Selection evidence

Confusable tool pairs.

14 pairs where similar names or overlapping descriptions may send an agent toward the wrong tool.

Tool A	Tool B	Confidence	Why they collide
`upload_to_testflight`	`upload_build_to_asc`	high	Both download an IPA from a Codemagic artifact URL and upload it to App Store Connect/TestFlight for the same app_id+ipa_url inputs; a task like 'upload this build to TestFlight' doesn't clearly indicate whether the beta-distribution shortcut or the full ASC submission-pipeline upload is wanted.
`submit_beta_review`	`submit_for_app_store_review`	high	Both are 'submit for review' actions on a build/app; a request like 'submit my app for review' is ambiguous between TestFlight beta review (submit_beta_review, takes build_id) and full App Store review (submit_for_app_store_review, takes app_id+version+build_id).
`list_builds`	`list_asc_builds`	medium	A generic request like 'list my builds' or 'show recent builds for this app' doesn't specify Codemagic CI builds vs App Store Connect/TestFlight builds, and both tools return build lists with status/version info.
`get_asc_review_status`	`get_asc_release_status`	high	Near-identical names and both take only app_id; a request like 'check the status of my app' is ambiguous between review status specifically and the broader release pipeline dashboard.
`upload_to_google_play`	`promote_google_play_release`	medium	Both move a release onto a Google Play track; 'release this to production' could mean uploading a fresh AAB to the production track (upload_to_google_play) or promoting an existing release from another track (promote_google_play_release).
`list_variable_groups`	`list_testflight_groups`	medium	Both are 'list groups for an app' tools; an ambiguous instruction like 'show me the groups for this app' could target Codemagic variable groups or TestFlight beta tester groups.
`create_variable_group`	`create_testflight_group`	medium	Both are 'create a new group' actions taking a name; a vague request like 'create a group called Beta Testers' could be misrouted to variable groups instead of TestFlight groups or vice versa.
`get_ios_store_listing`	`get_android_store_listing`	medium	Both fetch current store listing text for a version/app; if the platform isn't explicit in a request like 'get the store listing text' the agent could pick the wrong platform-specific tool.
`set_ios_store_listing`	`set_android_store_listing`	medium	Both update store listing fields (name/description/etc.) for a locale; 'update the store description' without stating iOS or Android is genuinely ambiguous between the two.
`upload_ios_screenshots`	`upload_android_screenshots`	medium	Both upload screenshot URLs to a store listing with very similar parameters; a platform-unspecified request like 'upload these screenshots to the store' could select either.
`add_variable`	`update_variable`	medium	A request like 'set FLUTTER_VERSION to 3.19' doesn't indicate whether the variable already exists, so the agent could plausibly call add_variable when update_variable (or vice versa) is correct.
`list_google_play_tracks`	`list_google_play_bundles`	medium	Both surface version-code information for an Android app; 'what versions have been released/uploaded' is ambiguous between track release info and uploaded AAB bundle history.
`get_build`	`get_build_logs`	medium	Both take build_id and relate to build diagnostics; 'check what happened with this build' could plausibly call either full build details or the failure logs.
`list_variable_groups`	`list_variables`	low	Names are similar ('list variable groups' vs 'list variables') but list_variables requires a group_id, making the distinction usually clear from context — confusion is possible only in a very underspecified request like 'show me the variables'.

Compare the field

One score is useful.
The evidence makes it actionable.

Back to the leaderboard