Public assessment

GitHub MCP (community server, listTools works unauthenticated)

github-mcp · v0.6.2 · scanned Aug 1, 2026

What changed in the harness

Selection accuracy 98→95, token cost up 73%, unconfirmed writes 0%→0%.

Category breakdown

Where the score comes from.

Earned points across the four signals Gradable measures. Safety and Legibility are scored out of 30; Economics and Discoverability are scored out of 20.

01Safety

0.0 / 30

0.0 out of 30

02Legibility

17.6 / 30

17.6 out of 30

03Economics

16.8 / 20

16.8 out of 20

04Discoverability

11.4 / 20

11.4 out of 20

Highest-impact fix

Estimated gain +30 points

Add explicit identity and permission preflight tools

Expose machine-readable principal/tenant confirmation and a non-mutating permission check so agents can verify both before destructive actions.

Description evidence

Defects and rewrites.

26 defects found across the exposed tool descriptions. Suggested rewrites make purpose, inputs, boundaries, and returns easier for an agent to understand.

Tool	Defect types	Suggested rewrite
`create_or_update_file`	name_restates_behavior no_return_description	Creates a new file or updates an existing one in a GitHub repository via a commit; requires the file's blob SHA when overwriting an existing file. Returns the commit and content metadata for the new file version.
`search_repositories`	name_restates_behavior no_return_description	Searches GitHub for repositories matching a query using GitHub's repository search syntax, with pagination support. Returns a paginated list of matching repositories and their metadata.
`create_repository`	name_restates_behavior no_return_description	Creates a new GitHub repository under the authenticated user's account, optionally private and initialized with a README. Returns the newly created repository's details.
`get_file_contents`	name_restates_behavior no_return_description	Retrieves the raw content of a file, or the listing of a directory, at a given path and branch in a GitHub repository. Returns the decoded file content and metadata, or an array of directory entries.
`push_files`	name_restates_behavior no_return_description	Commits and pushes an array of file path/content pairs to a branch in a GitHub repository as one atomic commit, avoiding a separate commit per file. Returns the resulting commit's details.
`create_issue`	name_restates_behavior params_unexplained no_return_description	Opens a new issue in a GitHub repository. 'owner' and 'repo' identify the repository, 'title' is the issue title, 'body' is the description text, 'assignees' is a list of usernames to assign, 'milestone' is the milestone number, and 'labels' is a list of label names to apply. Returns the created issue's details, including its number and URL.
`create_pull_request`	name_restates_behavior no_return_description	Opens a new pull request proposing to merge the 'head' branch into the 'base' branch of a GitHub repository, optionally as a draft. Returns the created pull request's details, including its number and URL.
`fork_repository`	name_restates_behavior no_return_description	Creates a fork of a GitHub repository under the authenticated user's account or a specified organization. Returns the details of the newly created fork.
`create_branch`	name_restates_behavior no_return_description	Creates a new branch in a GitHub repository, branching from a given source branch or the repository's default branch if none is specified. Returns the new branch's reference and commit SHA.
`list_commits`	params_unexplained no_return_description	Lists the commit history for a GitHub repository. 'owner' and 'repo' identify the repository, 'sha' optionally specifies a branch, tag, or commit to start listing from (defaults to the default branch), and 'page'/'perPage' control pagination. Returns a paginated array of commits with SHA, author, message, and date.
`list_issues`	name_restates_behavior params_unexplained no_return_description	Lists issues in a GitHub repository with filtering and sorting. 'owner' and 'repo' identify the repository, 'state' filters by open/closed/all, 'labels' filters by label names, 'sort' and 'direction' control ordering (by created/updated/comments, ascending/descending), 'since' returns issues updated after this timestamp, and 'page'/'per_page' control pagination. Returns a paginated array of matching issues.
`update_issue`	name_restates_behavior params_unexplained no_return_description	Updates fields on an existing GitHub issue identified by 'owner', 'repo', and 'issue_number'. Any of 'title', 'body', 'assignees', 'milestone', 'labels', or 'state' (open/closed) may be supplied to change that field, leaving omitted fields unchanged. Returns the updated issue's details.
`add_issue_comment`	name_restates_behavior params_unexplained no_return_description	Posts a new comment on an existing GitHub issue or pull request. 'owner' and 'repo' identify the repository, 'issue_number' identifies the issue or pull request, and 'body' is the comment text. Returns the created comment's details, including its ID and URL.
`search_code`	name_restates_behavior params_unexplained no_return_description	Searches source code across GitHub repositories using GitHub's code search syntax. 'q' is the search query, 'order' sets ascending or descending sort, and 'page'/'per_page' control pagination. Returns a paginated list of matching code fragments with their file and repository locations.
`search_issues`	name_restates_behavior params_unexplained no_return_description	Searches issues and pull requests across GitHub repositories using GitHub's issue search syntax. 'q' is the search query, 'sort' selects the ranking field (created, updated, comments, reactions, etc.), 'order' sets ascending or descending, and 'page'/'per_page' control pagination. Returns a paginated list of matching issues and pull requests.
`search_users`	name_restates_behavior params_unexplained no_return_description	Searches for GitHub user accounts using GitHub's user search syntax. 'q' is the search query, 'sort' ranks results by followers, repositories, or join date, 'order' sets ascending or descending, and 'page'/'per_page' control pagination. Returns a paginated list of matching user profiles.
`get_issue`	name_restates_behavior params_unexplained no_return_description	Fetches a single GitHub issue identified by 'owner', 'repo', and 'issue_number'. Returns the issue's full details, including title, body, state, labels, and assignees.
`get_pull_request`	name_restates_behavior no_return_description	Fetches a single pull request identified by 'owner', 'repo', and 'pull_number' in a GitHub repository. Returns the pull request's full details, including title, body, state, branches, and merge status.
`list_pull_requests`	name_restates_behavior no_return_description	Lists pull requests in a GitHub repository, filterable by state, head/base branch, and sortable by creation, update, popularity, or duration. Returns a paginated array of matching pull requests.
`create_pull_request_review`	name_restates_behavior no_return_description	Submits a review on a pull request with an overall verdict (approve, request changes, or comment) and optional inline comments on specific file lines or diff positions. Returns the created review's details.
`merge_pull_request`	name_restates_behavior no_return_description	Merges an open pull request into its base branch using the specified method (merge, squash, or rebase), with optional custom commit title and message. Returns the merge result, including the resulting commit SHA.
`get_pull_request_files`	no_return_description	Lists the files changed in a pull request, identified by 'owner', 'repo', and 'pull_number'. Returns an array of changed files with their status, additions, deletions, and patch diff.
`get_pull_request_status`	no_return_description	Fetches the combined status of all CI/status checks for the head commit of a pull request, identified by 'owner', 'repo', and 'pull_number'. Returns the overall state (success/failure/pending) plus the individual status checks.
`update_pull_request_branch`	name_restates_behavior no_return_description	Updates a pull request's branch by merging in the latest changes from its base branch, optionally verifying the branch's current head SHA first. Returns confirmation of the update, including the new head SHA.
`get_pull_request_comments`	name_restates_behavior no_return_description	Fetches the inline review comments on a pull request's diff, identified by 'owner', 'repo', and 'pull_number'. Returns an array of comments with their file path, line/position, author, and text.
`get_pull_request_reviews`	name_restates_behavior no_return_description	Fetches the reviews submitted on a pull request, identified by 'owner', 'repo', and 'pull_number'. Returns an array of reviews with their author, verdict (approved/changes requested/commented), and body text.

Selection evidence

Confusable tool pairs.

44 pairs where similar names or overlapping descriptions may send an agent toward the wrong tool.

Tool A	Tool B	Confidence	Why they collide
`create_pull_request`	`create_pull_request_review`	medium	A task like 'submit feedback on PR #5' could be phrased ambiguously enough that an agent confuses creating a new PR with creating a review on an existing one, though the verbs (create vs review) usually disambiguate.
`get_pull_request`	`get_pull_request_comments`	medium	A request like 'show me PR #12' could plausibly resolve to either the general PR details or specifically its comments if the agent overgeneralizes 'details' to include comments.
`get_pull_request`	`get_pull_request_files`	low	get_pull_request returns general metadata while get_pull_request_files is clearly scoped to file diffs; a task asking 'what changed in PR X' leans toward files but could be answered by either tool, creating mild ambiguity.
`get_pull_request`	`get_pull_request_reviews`	low	Both relate to PR state, but 'reviews' is a distinct enough concept (approvals/comments by reviewers) that confusion is unlikely except in vague prompts like 'what's the status of reviews on PR X'.
`get_pull_request`	`get_pull_request_status`	medium	A prompt like 'check the status of PR #7' is genuinely ambiguous between fetching general PR details (which includes state) and the dedicated CI status-checks endpoint.
`get_pull_request_comments`	`get_pull_request_reviews`	medium	Comments and reviews are closely related GitHub concepts (review comments vs review summaries); a task like 'what did reviewers say on PR X' could map to either tool.
`create_pull_request_review`	`get_pull_request_reviews`	low	Create vs get are distinct verbs so genuine selection confusion is unlikely, though both concern the same 'reviews' resource.
`get_pull_request_files`	`get_pull_request_comments`	low	Files changed vs review comments are distinct concepts; unlikely to be confused except in very vague 'show me PR feedback' requests.
`get_pull_request_files`	`get_pull_request_reviews`	low	Distinct resource types (file diffs vs review approvals) make confusion unlikely beyond generic 'PR details' phrasing.
`get_pull_request_files`	`get_pull_request_status`	low	Files changed and CI status checks are clearly different concepts, minimal risk of confusion.
`get_pull_request_status`	`get_pull_request_comments`	low	CI status and review comments are semantically distinct enough that confusion is unlikely.
`create_pull_request`	`list_pull_requests`	low	Create and list are clearly different actions; unlikely to be confused despite shared tokens.
`create_pull_request`	`merge_pull_request`	low	Distinct lifecycle actions (create vs merge) are unlikely to be confused by a reasonable NL task.
`list_pull_requests`	`merge_pull_request`	low	Listing vs merging are clearly distinct operations with little ambiguity.
`create_issue`	`create_branch`	low	Both are 'create new X' actions in a repo, but issue vs branch are unmistakably different entities.
`search_repositories`	`search_code`	low	A vague 'search GitHub for X' could theoretically apply to either, but repo search vs code search target different result types and are usually clear from context.
`create_repository`	`create_branch`	low	Creating a new repo vs a new branch are distinct enough concepts that confusion is unlikely.
`create_repository`	`create_issue`	low	Distinct entity types (repository vs issue) make confusion unlikely despite shared 'create new' phrasing.
`search_repositories`	`search_users`	low	Searching for repos vs users targets clearly different entities, low ambiguity.
`create_pull_request`	`get_pull_request_status`	low	Create and get-status are different verbs on the same resource type but not plausibly confused.
`create_pull_request_review`	`merge_pull_request`	low	Reviewing and merging are distinct PR actions unlikely to be confused.
`list_pull_requests`	`create_pull_request_review`	low	Listing PRs vs creating a review are clearly different operations.
`list_pull_requests`	`update_pull_request_branch`	low	Listing vs updating a branch are distinct actions, low ambiguity.
`merge_pull_request`	`update_pull_request_branch`	medium	A task like 'bring PR #3 up to date and merge it' could cause confusion about which tool 'update the PR' refers to, since update_pull_request_branch syncs with base while merge_pull_request finalizes it.
`search_code`	`search_issues`	low	Code search vs issue/PR search target clearly different content types, unlikely to be confused.
`create_issue`	`update_issue`	low	Create vs update are distinct verbs; a task specifying new vs existing issue clearly resolves to one or the other.
`create_repository`	`fork_repository`	medium	A task like 'make me a copy of repo X' is ambiguous between forking an existing repo and creating a new one, since both result in a new repository under the user's account.
`search_repositories`	`search_issues`	low	Searching repos vs issues/PRs are distinct enough result types to avoid confusion.
`create_issue`	`get_issue`	low	Create vs get are clearly distinct verbs, low ambiguity.
`list_issues`	`update_issue`	low	Listing vs updating issues are distinct operations, unlikely to be confused.
`update_issue`	`get_issue`	low	Update vs get are clearly different verbs despite sharing the issue resource.
`search_code`	`search_users`	low	Code search and user search target very different entities, minimal ambiguity.
`list_commits`	`list_issues`	low	Commits and issues are distinct resource types, unlikely to be confused.
`list_issues`	`get_issue`	low	List vs get-single are distinct enough operations, low ambiguity.
`list_issues`	`search_issues`	medium	A task like 'find open issues about X in this repo' could plausibly map to either listing with filters or searching, since both can filter by state/repo.
`search_issues`	`get_issue`	low	Search across repos vs get a specific known issue are distinct enough use cases.
`create_or_update_file`	`push_files`	medium	A task like 'add these files to the repo' is ambiguous between creating/updating a single file and pushing multiple files in one commit, especially if the agent isn't sure how many files are involved.
`create_or_update_file`	`create_branch`	low	Creating/updating a file vs creating a branch are clearly distinct actions.
`create_or_update_file`	`update_issue`	low	Updating a file vs updating an issue are distinct enough entities to avoid confusion.
`create_or_update_file`	`create_repository`	low	Creating a file vs creating a repository are clearly different scopes of action.
`get_file_contents`	`push_files`	low	Reading vs writing files are opposite operations, unlikely to be confused.
`create_branch`	`list_commits`	low	Creating a branch vs listing commits are distinct operations with little overlap.
`get_file_contents`	`list_commits`	low	Getting file contents vs listing commit history are clearly distinct actions.
`push_files`	`list_commits`	low	Pushing files vs listing commits are distinct write vs read operations, low ambiguity.

Compare the field

One score is useful.
The evidence makes it actionable.

Back to the leaderboard