01Safety
30.0 / 30
Public leaderboard
Public assessment
arXiv MCP Server (@cyanheads/arxiv-mcp-server, Smithery popular)
What changed in the harness
Selection accuracy 98%, destructive-action safety rate 100% (baseline only -- no rewrite pass applied).
Category breakdown
Where the score comes from.
Earned points across the four signals Gradable measures. Safety and Legibility are scored out of 30; Economics and Discoverability are scored out of 20.
02Legibility
29.1 / 30
03Economics
18.3 / 20
04Discoverability
14.8 / 20
Highest-impact fix
Estimated gain +5 pointsMake target tools discoverable on the first call
Clarify tool names, decision boundaries, and required argument schemas so an agent can choose and construct the target call without exploratory steps.
Description evidence
Defects and rewrites.
0 defects found across the exposed tool descriptions. Suggested rewrites make purpose, inputs, boundaries, and returns easier for an agent to understand.
| Tool | Defect types | Suggested rewrite |
|---|---|---|
| No description defects were flagged in this assessment. | ||
Selection evidence
Confusable tool pairs.
3 pairs where similar names or overlapping descriptions may send an agent toward the wrong tool.
| Tool A | Tool B | Confidence | Why they collide |
|---|---|---|---|
arxiv_get_metadata |
arxiv_read_paper |
medium | Both take a known paper_id and return information about that paper; a task phrased 'show me details about paper 2401.12345' or 'tell me about this paper' is ambiguous between metadata (title/abstract) and full-text body, so an agent could pick the wrong one when the user's intent was the other. |
arxiv_search |
arxiv_get_metadata |
low | A task phrased with 'search/look up the paper 2401.12345' could drive the agent to arxiv_search (querying the ID as text) even though arxiv_get_metadata is the intended known-ID lookup; the ambiguity is real but the descriptions ('Use when you have known IDs') usually disambiguate. |
arxiv_search |
arxiv_read_paper |
low | A task like 'find and read the paper on attention' lacking an ID is ambiguous on which step to invoke: an agent may call arxiv_search alone (getting only metadata/abstract) when the user wanted full text, or call arxiv_read_paper without an ID and fail; the read tool's full-text focus usually prevents this. |
Compare the field