Public assessment

GitHub MCP (community server, listTools works unauthenticated)

github-mcp · v0.6.2 · scanned Aug 3, 2026

What changed in the harness

Selection accuracy 95→98, token cost up 11%, unconfirmed writes 0%→0%.

Category breakdown

Where the score comes from.

Earned points across the four signals Gradable measures. Safety and Legibility are scored out of 30; Economics and Discoverability are scored out of 20.

01Safety

0.0 / 30

0.0 out of 30

02Legibility

18.5 / 30

18.5 out of 30

03Economics

17.5 / 20

17.5 out of 20

04Discoverability

13.5 / 20

13.5 out of 20

Highest-impact fix

Estimated gain +30 points

Add explicit identity and permission preflight tools

Expose machine-readable principal/tenant confirmation and a non-mutating permission check so agents can verify both before destructive actions.

Description evidence

Defects and rewrites.

26 defects found across the exposed tool descriptions. Suggested rewrites make purpose, inputs, boundaries, and returns easier for an agent to understand.

Tool	Defect types	Suggested rewrite
`create_or_update_file`	name_restates_behavior no_return_description	Create or update a single file at the given path in a repository on the specified branch, committing content with message. Provide sha when updating an existing file so the correct version is replaced; omit it to create a new file. Returns the committed file with its SHA.
`search_repositories`	name_restates_behavior no_return_description	Search public GitHub repositories using GitHub search syntax (query), with pagination via page and perPage (max 100). Returns a list of matching repositories with metadata such as stars, language, and URLs.
`create_repository`	name_restates_behavior no_return_description	Create a new repository in your GitHub account with a name and optional description, private visibility, and autoInit to add a README. Returns the created repository with its clone URLs.
`get_file_contents`	name_restates_behavior no_return_description	Read a file or directory at path in a repository, optionally on a specific branch; binary files are base64-encoded. Returns the file content or a directory listing with metadata.
`push_files`	name_restates_behavior no_return_description	Stage and commit an array of files (each with path and content) to the branch in a single commit using message. Returns the resulting commit with its SHA.
`create_issue`	name_restates_behavior params_unexplained no_return_description	Create a new issue in the repository identified by owner/repo with a required title, plus optional body, assignees (usernames), milestone (number), and labels. Returns the created issue with its number, URL, and state.
`create_pull_request`	name_restates_behavior no_return_description	Open a pull request in the repository that merges the head branch into base, with a title and optional body; can be created as draft or with maintainer_can_modify. Returns the created pull request with its number and URL.
`fork_repository`	name_restates_behavior no_return_description	Copy the repository (owner/repo) into your personal account or the organization, preserving version history. Returns the forked repository with its new clone URLs.
`create_branch`	name_restates_behavior no_return_description	Create a new branch named branch in the repository, branching from from_branch or the default branch when from_branch is omitted. Returns the new branch reference with its commit SHA.
`list_commits`	params_unexplained no_return_description	List commits on the default branch, a branch sha, or a specific commit sha in the repository, with pagination (page, perPage). Returns commit details including SHAs, messages, authors, and dates.
`list_issues`	name_restates_behavior params_unexplained no_return_description	List issues in the repository, optionally filtered by state, labels, sort, direction, and since date, with pagination (page, per_page). Returns a page of issues with their titles, numbers, and states.
`update_issue`	name_restates_behavior params_unexplained no_return_description	Modify the issue with issue_number in the repository, updating any of title, body, assignees, milestone, labels, or state (open/closed). Returns the updated issue.
`add_issue_comment`	name_restates_behavior params_unexplained no_return_description	Post a comment with body text on the issue identified by issue_number in the repository. Returns the created comment with its ID.
`search_code`	name_restates_behavior params_unexplained no_return_description	Search code across GitHub repositories using a search query (q) with optional order and pagination (page, per_page, max 100). Returns matching code results with file paths and repository metadata.
`search_issues`	name_restates_behavior params_unexplained no_return_description	Search issues and pull requests across GitHub repositories using a query (q), optionally sorted by created/updated/comments etc. with order and pagination. Returns matching issues and pull requests.
`search_users`	name_restates_behavior params_unexplained no_return_description	Search GitHub users by query (q), optionally sorted by followers, repositories, or joined date, with order and pagination. Returns matching user profiles.
`get_issue`	name_restates_behavior params_unexplained no_return_description	Retrieve full details of the issue with issue_number in the repository, including its title, body, state, labels, assignees, and comment count. Returns the issue object.
`get_pull_request`	name_restates_behavior no_return_description	Retrieve full details of pull request pull_number, including its title, body, state, head/base refs, mergeable status, and review status. Returns the pull request object.
`list_pull_requests`	name_restates_behavior no_return_description	List pull requests in the repository, optionally filtered by state, head, or base, sorted and paginated (per_page max 100). Returns a page of pull requests.
`create_pull_request_review`	name_restates_behavior no_return_description	Submit a review (event: APPROVE, REQUEST_CHANGES, or COMMENT) on pull request pull_number, with an optional body and inline comments (each specifying path and either position or line, not both). Returns the created review.
`merge_pull_request`	name_restates_behavior no_return_description	Merge pull request pull_number into its base branch using merge_method, with optional commit_title and commit_message. Returns the merge result with the merge commit SHA.
`get_pull_request_files`	no_return_description	Get the files changed in pull request pull_number, including each file's status (added/modified/deleted), additions, deletions, and changes. Returns the list of changed files.
`get_pull_request_status`	no_return_description	Get the combined status of all CI/status checks for pull request pull_number, reporting the overall state (success, failure, pending). Returns the combined status with per-check details.
`update_pull_request_branch`	name_restates_behavior no_return_description	Update the branch of pull request pull_number with the latest changes from its base branch, optionally verifying expected_head_sha when provided. Returns the update result.
`get_pull_request_comments`	name_restates_behavior no_return_description	Get the inline review comments on pull request pull_number, including comment body, author, and the path/position commented on. Returns the list of review comments.
`get_pull_request_reviews`	name_restates_behavior no_return_description	Fetch the list of reviews submitted on a pull request, identified by pull_number in the owner/repo repository. Use this to inspect review status and reviewer feedback on a PR. Returns an array of review objects with details such as the reviewer, review state, and when the review was submitted.

Selection evidence

Confusable tool pairs.

12 pairs where similar names or overlapping descriptions may send an agent toward the wrong tool.

Tool A	Tool B	Confidence	Why they collide
`create_pull_request`	`create_pull_request_review`	medium	A task like 'review pull request #4' or 'submit a review on the PR' can be read as either creating a PR or writing an approval/comment review, since 'review' modifies the same tokens and both are PR-scoped actions.
`get_pull_request`	`get_pull_request_comments`	medium	'Show me the comments on pull request #5' names the same tokens and the generic get_pull_request could be selected instead of the comments-specific fetch if the agent treats comments as subsumed by PR details.
`get_pull_request`	`get_pull_request_files`	medium	'What changed in pull request #5?' could route to generic PR details rather than the changed-files endpoint; both share owner/repo/pull_number inputs, so the natural task fits either tool name.
`get_pull_request`	`get_pull_request_reviews`	medium	'Show the reviews on pull request #5' maps to the same token set; an agent may pick the base get_pull_request tool instead of the reviews endpoint.
`get_pull_request`	`get_pull_request_status`	high	'What is the status of pull request #5?' is genuinely ambiguous: get_pull_request returns the PR state/merged info while get_pull_request_status returns CI check status, and both are plausible responses to 'status'.
`get_pull_request_comments`	`get_pull_request_reviews`	medium	'Show me the review comments on PR #5' can be satisfied by either tool; users routinely conflate inline review comments with the list of reviews, and both have identical signatures.
`create_pull_request_review`	`get_pull_request_reviews`	medium	'Can you review pull request #5?' is ambiguous between writing a review (create) and reading existing reviews (get), since 'review' functions as both verb and noun.
`create_or_update_file`	`push_files`	high	'Update this file in the repo' or 'commit my changes' could be routed to either tool; the only distinguishing feature is single vs multiple files, and a single-file request can be wrongly sent to push_files or a multi-file request to create_or_update_file.
`search_repositories`	`search_code`	medium	A generic task like 'search GitHub for X' is underspecified; both tools take a query and return GitHub search results, so without an explicit 'repositories' or 'code' qualifier an agent may choose the wrong search tool.
`search_repositories`	`search_issues`	low	'Find issues about X' vs 'find repos about X' is usually disambiguated by the noun, but a vaguer phrase like 'search GitHub for X' could route to the wrong search endpoint.
`search_code`	`search_issues`	low	Searching for a term such as an error message or API name could plausibly target either code or issues depending on phrasing, e.g. 'where does this error appear'.
`list_issues`	`search_issues`	low	'Show all issues about X in this repo' might be sent to list_issues, which lacks a text-query parameter, instead of search_issues, despite the natural-language intent requiring search.

Compare the field

One score is useful.
The evidence makes it actionable.

Back to the leaderboard