Public assessment

todah-zg/codemagic-mcp (codemagic-mcp-server)

codemagic-mcp-server · v1.0.5 · scanned Aug 4, 2026

What changed in the harness

Selection accuracy 98→100, token cost up 4%, unconfirmed writes 0%→0%.

Category breakdown

Where the score comes from.

Earned points across the four signals Gradable measures. Safety and Legibility are scored out of 30; Economics and Discoverability are scored out of 20.

01Safety

0.0 / 30

0.0 out of 30

02Legibility

18.8 / 30

18.8 out of 30

03Economics

11.8 / 20

11.8 out of 20

04Discoverability

11.2 / 20

11.2 out of 20

Highest-impact fix

Estimated gain +30 points

Add explicit identity and permission preflight tools

Expose machine-readable principal/tenant confirmation and a non-mutating permission check so agents can verify both before destructive actions.

Description evidence

Defects and rewrites.

40 defects found across the exposed tool descriptions. Suggested rewrites make purpose, inputs, boundaries, and returns easier for an agent to understand.

Tool	Defect types	Suggested rewrite
`list_applications`	no_return_description	List all applications in your Codemagic account. Returns each application's ID and name — use the app ID with all other Codemagic tools that require one. If team_id is omitted, lists apps for the authenticated user.
`get_build`	params_unexplained no_return_description	Fetch full details for a single build, including its status and artifact download URLs. The build_id is the ID returned by trigger_build or list_builds. Artifact URLs require x-auth-token — use them with upload_to_testflight, upload_to_google_play, or create_public_artifact_url.
`cancel_build`	no_return_description	Cancel a running or queued Codemagic build and return its updated status. Use when a triggered build is no longer needed — for example if the wrong branch was used or an error was found after triggering. Has no effect on builds that have already finished.
`add_application`	no_return_description	Add a new application to Codemagic by connecting a Git repository, returning the new application's details including its app ID. For HTTPS URLs: if you have connected your GitHub, GitLab, or Bitbucket account via Codemagic Settings → Integrations, private repositories are accessible with just the URL — no credentials needed. For SSH URLs (git@... or ssh://git@...): a fresh Ed25519 deploy key is generated automatically. The private key is stored directly in Codemagic and the public key is added to GitHub automatically if the gh CLI is installed and authenticated, or shown for manual setup otherwise. Note: after adding, the app shows 'Set up build' in the Codemagic UI — this is expected.
`create_variable_group`	no_return_description	Create a new variable group in Codemagic and return the created group's ID. Requires a team_id (personal accounts do not support global variable groups) or an app_id for app-level groups. After creating, add non-secret variables via add_variable, or add secret values directly in the Codemagic UI.
`add_variable`	no_return_description	Add a non-secret variable to a Codemagic variable group and return the created variable's details. For secret values (API keys, certificates, tokens) use the Codemagic UI instead — secrets should never pass through the agent.
`update_variable`	no_return_description	Update the name or value of an existing non-secret variable in a Codemagic variable group and return the updated variable's details. The variable_id comes from list_variable_groups. For secret values use the Codemagic UI — secrets should never pass through the agent.
`delete_variable`	no_return_description	Delete a variable from a Codemagic variable group and return confirmation of the deletion. The variable_id comes from list_variable_groups. This cannot be undone.
`get_webhook_url`	no_return_description	Get the incoming webhook URL for a Codemagic app. Returns the URL — paste it into your Git provider (GitHub, GitLab, or Bitbucket) repository settings to trigger builds automatically on push or pull request events.
`list_webhooks`	name_restates_behavior no_return_description	List the webhook subscriptions configured for a Codemagic app. Returns each webhook's ID and configuration, which you can use with delete_webhook to remove a subscription.
`delete_webhook`	name_restates_behavior no_return_description	Remove a webhook subscription from a Codemagic app so it stops triggering builds and return confirmation of the deletion. The webhook_id comes from list_webhooks.
`list_caches`	no_return_description	List build caches for a Codemagic app. Returns each build cache with its ID, scoped to a workflow. Use the cache IDs with delete_cache to free up storage or force a clean build.
`list_asc_apps`	no_return_description	List apps in App Store Connect. Returns the ASC app ID for each app — the ID needed by all other App Store Connect tools. Call this first.
`list_asc_builds`	no_return_description	List TestFlight builds for an app in App Store Connect. Returns the builds including their version numbers — find the highest version, increment it by 1, and pass that as BUILD_NUMBER in trigger_build variables before triggering a release build.
`list_testflight_groups`	no_return_description	List TestFlight beta groups for an app. Returns the group names you can pass as the beta_group parameter of upload_to_testflight to distribute a build to those testers automatically after upload.
`get_asc_review_status`	no_return_description	Get the current App Store review status for an app. Returns the review state for app_id so you can monitor progress and check for blockers after submitting to the App Store.
`get_asc_release_status`	no_return_description	Get a full release pipeline status dashboard for an app. Returns the latest build, TestFlight state, App Store version, and submission state for app_id so you can review the whole release pipeline in one call.
`set_version_metadata`	params_unexplained no_return_description	Update App Store version localization metadata — What's New text, description, keywords, and more. What's New is required for every release before submitting for review. Call once per locale — en-US is the required default; add other locales if the app supports them. support_url is the Support URL shown on the App Store page of the version, and marketing_url is an optional marketing page URL for the version. Returns confirmation that the metadata for the given locale and version was updated; run validate_app_submission afterward to confirm the update resolved the blocker.
`set_export_compliance`	no_return_description	Set the export compliance declaration for an iOS build. Required before App Store submission and for TestFlight external distribution. Most apps only use standard HTTPS/TLS — set uses_non_exempt_encryption to false. Only set it to true if the app implements custom or proprietary encryption beyond standard protocols. Targets the latest build by default, or the specific build passed as build_id. Returns confirmation that the compliance declaration was recorded for the target build.
`release_version`	no_return_description	Release an App Store version that has been approved and is waiting in 'Pending Developer Release' state. Returns confirmation that the update is now available to all users (or that the configured phased rollout has started). Use set_phased_release with action=create before submission if you want a gradual rollout instead of an instant release.
`set_phased_release`	no_return_description	Manage a phased rollout for an App Store version. Phased rollout gradually releases the update over 7 days: 1% to 2% to 5% to 10% to 20% to 50% to 100%. Actions: 'create' — configure phased rollout before submitting for review. 'pause' — pause an in-progress rollout (use if a critical bug is found after release). 'resume' — resume a paused rollout. 'complete' — immediately release to all remaining users. Returns the updated phased rollout state for the version.
`submit_beta_review`	no_return_description	Submit a build for TestFlight beta app review. Required before external beta groups can install the build — Apple reviews it once, then all external groups can access it. Internal groups (Apple employees / org members) do not require beta review. Get the build ID from list_asc_builds. Returns the beta review submission state for the build.
`add_testflight_tester`	no_return_description	Add a tester to TestFlight by email address. Optionally assign them to a specific beta group — use list_testflight_groups to get group names. Returns confirmation that the tester was added and that an invitation email has been sent by Apple.
`create_testflight_group`	no_return_description	Create a new TestFlight beta group for an app. External groups require beta app review before testers can install builds. Internal groups (Apple org members) do not require review — useful for fast internal QA. Returns the created group so its name can be used in upload_to_testflight or add_testflight_tester.
`upload_to_testflight`	no_return_description	Download an IPA artifact from Codemagic and upload it to TestFlight via App Store Connect. Returns the upload result and, if beta_group is provided, confirms the build was distributed to that group after upload.
`upload_build_to_asc`	no_return_description	Download an IPA from a Codemagic artifact URL and upload it to App Store Connect. Returns the build UUID immediately once the upload commits — it does NOT wait for Apple's processing pipeline. After calling, poll list_asc_builds until the build's processingState is VALID, then pass the returned build UUID as build_id to submit_for_app_store_review.
`submit_for_app_store_review`	no_return_description	Attach a processed build to an App Store version and submit it for review. The build must have processingState VALID — confirm with list_asc_builds before calling. Requires version metadata (What's New) and export compliance to be set first. Use validate_app_submission to catch blockers before submitting. Returns the review submission state for the version.
`upload_ios_screenshots`	no_return_description	Download screenshot images from URLs and upload them to App Store Connect for a specific device type and locale. Apple allows up to 10 screenshots per set. Supported formats: PNG (no alpha) and JPEG. Max 10 MB per file. Call list_ios_screenshot_types first to get valid device_type values and required dimensions. Set replace=true to delete existing screenshots before uploading (recommended when refreshing a set). Returns confirmation of the uploaded screenshots for the device type and locale.
`list_google_play_tracks`	no_return_description	List Google Play tracks (internal, alpha, beta, production) with current release info and version codes. Returns the tracks for the package so you can find the highest versionCode across all tracks, increment by 1, and pass that as BUILD_NUMBER in trigger_build variables before triggering a release build.
`list_google_play_bundles`	no_return_description	List all uploaded App Bundles (AAB) for an app on Google Play. Returns the bundles with their version codes so you can audit what has already been uploaded before triggering a new build.
`promote_google_play_release`	no_return_description	Promote a release between Google Play tracks (e.g. internal to alpha to beta to production) without re-uploading. Set user_fraction to enable staged rollout on the target track (0.1 = 10% of users). To halt an in-progress staged rollout: set source_track=target_track='production' and release_status='halted'. To resume a halted rollout: same tracks with release_status='inProgress' and a user_fraction. Returns the resulting release state on the target track.
`set_rollout_fraction`	no_return_description	Adjust the staged rollout percentage for an existing release on a Google Play track. Use this to gradually expand a rollout (e.g. 10% to 25% to 50% to 100%). Requires the version code of the release currently in the staged rollout. Returns the updated rollout fraction for the release.
`upload_android_screenshots`	no_return_description	Download screenshot images from URLs and upload them to Google Play for a specific language and device type. Google allows up to 8 screenshots per device type. Supported formats: JPEG and 24-bit PNG (no alpha). Max 8 MB per file. Common image types: phoneScreenshots, sevenInchScreenshots, tenInchScreenshots, tvScreenshots, wearScreenshots. Set replace=true to delete all existing screenshots of this type before uploading (recommended when refreshing a set). All uploads are committed atomically — if any upload fails, no changes go live. Returns the screenshot URLs that were successfully uploaded along with the language and image type they were committed to, confirming what went live.
`reply_to_google_play_review`	no_return_description	Post or update a developer reply to a Google Play user review. Replies are limited to 350 characters. If the review already has a developer reply, this call replaces it. Get the review_id from list_google_play_reviews. Write a personal, helpful reply — responding to negative reviews improves store ratings and user trust. Returns the review ID and the published reply text, confirming the reply went live.
`set_android_data_safety`	no_return_description	Submit the data safety declaration for a Google Play app. The declaration describes what data the app collects, how it is used, and whether it is shared. Accepts the raw CSV exported from Play Console → App content → Data safety → Export CSV. Re-upload whenever data practices change (new data type, updated retention policy, etc.). Takes effect immediately — there is no staging step and no GET endpoint to retrieve current labels. Returns confirmation that the declaration was submitted successfully for the given package name.
`validate_codemagic_yaml`	name_restates_behavior no_return_description	Validate a codemagic.yaml file against the official Codemagic JSON schema. Call this before triggering a build to catch configuration errors such as invalid fields, wrong value types, or unknown keys. Returns whether the file is valid and, if not, the list of schema violations with their messages so you can fix them before starting a build.
`get_yaml_template`	no_return_description	Get a starter codemagic.yaml template for a given project type. Templates cover build and signing only — publishing is handled separately via App Store Connect tools. Call list_yaml_template_types to see all valid project_type values. IMPORTANT: Android templates use linux_x2 by default (cheaper, no Mac needed). Personal accounts (no team) cannot use linux_x2 — replace it with mac_mini_m2 for personal accounts. Returns the template contents as YAML text to use as your starting point.
`list_yaml_template_types`	name_restates_behavior no_return_description	List all available Codemagic project types that have starter templates, useful as input to get_yaml_template. Returns the list of valid project_type identifiers along with a short label for each so you know which template to request.
`prepare_release_notes`	no_return_description	Validate localized release notes before submitting to the App Store or Google Play. Checks that each locale is a valid BCP-47 code (e.g. en-US, fr-FR, zh-Hans) and that text fits within platform char limits (Android: 500, iOS: 4000). Pass platform='both' to validate against the stricter Android limit for notes that will go to both stores. Returns a per-locale validation result showing whether each note passed, its character count, and any errors found.
`check_publish_readiness`	no_return_description	Aggregate publish-readiness checks for iOS or Android into a single pass/fail report. API-verifiable checks (valid build, store listing completeness, binary validation) run live. Items that have no API (age rating, privacy labels, content policy, legal agreements) are always listed as 'human required' so nothing is silently skipped. Each item is tagged as 'agent can fix' or 'human required', giving a clear action plan. Call this before submit_for_app_store_review (iOS) or promoting to production (Android). Use first_publish_ios or first_publish_android prompts for the one-time account/app-record setup. Returns the overall pass/fail verdict plus each check with its status and 'agent can fix' or 'human required' tag.

Selection evidence

Confusable tool pairs.

11 pairs where similar names or overlapping descriptions may send an agent toward the wrong tool.

Tool A	Tool B	Confidence	Why they collide
`list_builds`	`list_asc_builds`	high	A task like 'list the builds' is ambiguous because one returns Codemagic CI builds (team_id) while the other returns App Store Connect TestFlight builds (app_id); both tool names literally read 'list builds'.
`submit_beta_review`	`submit_for_app_store_review`	high	A task like 'submit the build for review' matches both tool names and descriptions; only whether the intent is TestFlight beta review vs full App Store submission disambiguates them.
`upload_to_testflight`	`upload_build_to_asc`	medium	Both download an IPA artifact and upload it to App Store Connect; 'upload the ipa to Apple/the App Store' could route to either, differing only in TestFlight distribution vs App Store submission pipeline.
`upload_to_google_play`	`promote_google_play_release`	medium	'Publish the app to production' has two valid routes: uploading a new AAB to the production track or promoting an existing internal/alpha/beta release to production.
`get_build`	`get_build_logs`	medium	'Show me what happened with the build' could select get_build (full details, status, artifacts) instead of get_build_logs (step logs), since both surface build output.
`get_build`	`get_latest_build_number`	medium	'Get the latest build number' is split: get_latest_build_number returns the highest Play versionCode, while get_build fetches a specific Codemagic build's details; the phrase 'latest build' is ambiguous between them.
`list_variable_groups`	`list_variables`	medium	'List the variables' could select list_variable_groups (returns groups) instead of list_variables (which needs a group_id), so a task referencing 'variables' is ambiguous between the two levels.
`list_variable_groups`	`list_testflight_groups`	medium	'List the groups for this app' is ambiguous: one lists Codemagic variable groups, the other TestFlight beta groups; both use the generic term 'groups' and share an app-scoped parameter.
`list_google_play_tracks`	`list_google_play_bundles`	medium	Both list Play content with version codes; 'show me what's uploaded/released on Google Play' could route to tracks (release info) or bundles (uploaded AABs).
`get_asc_review_status`	`get_asc_release_status`	medium	'Check the app's App Store status' fits both a per-build review status and a full release pipeline dashboard; review vs release status is not clear from the task alone.
`create_variable_group`	`create_testflight_group`	medium	'Create a new group' without a qualifier is ambiguous between a Codemagic variable group and a TestFlight beta group; the definitions require domain context the task may not supply.

Compare the field

One score is useful.
The evidence makes it actionable.

Back to the leaderboard