01Safety
0.0 / 30
What changed in the harness
Selection accuracy 96→98, token cost down 1%, unconfirmed writes 0%→0%.
Category breakdown
Where the score comes from.
Earned points across the four signals Gradable measures. Safety and Legibility are scored out of 30; Economics and Discoverability are scored out of 20.
02Legibility
26.0 / 30
03Economics
6.4 / 20
04Discoverability
12.8 / 20
Highest-impact fix
Estimated gain +30 pointsAdd explicit identity and permission preflight tools
Expose machine-readable principal/tenant confirmation and a non-mutating permission check so agents can verify both before destructive actions.
Description evidence
Defects and rewrites.
1 defect found across the exposed tool descriptions. Suggested rewrites make purpose, inputs, boundaries, and returns easier for an agent to understand.
| Tool | Defect types | Suggested rewrite |
|---|---|---|
search_papers |
params_unexplained |
Search Scholar Feed's 600k+ CS/AI/ML paper corpus and return matching papers as a paged list of paper records (arxiv_id, title, authors, year, categories, has_code, github_url, citation_count, venue_name, llm_summary, llm_significance, llm_novelty_score, impact_pct) plus total count and next_cursor. Defaults to semantic (embedding) search, which finds conceptually related papers even when the wording doesn't match the title/abstract; pass mode='keyword' for exact-string full-text search. q is REQUIRED unless (a) anchor_paper_id or scope_to_citations_of is set (anchor mode ignores q and returns similar papers carrying similarity_score), or (b) sort is 'trending'/'recent'/'impactful' (a query-less browse-the-frontier feed capped at the FIRST 200 RESULTS, paging past offset 200 is a 422); filters alone do not substitute for q. sort is a relevance/impact dial: 'relevance' (default) = best topical match; 'balanced' = relevant AND well-cited; 'impactful' = most-cited among those relevant, the right choice for 'the important/seminal papers on topic X'; 'trending' = rising FORECAST impact (impact_pct, last ~90 days); 'recent' = newest first; 'community' = GitHub adoption (stars + star-velocity). Impact semantics, don't confuse: impact_pct (0-100) is a cohort-relative ML forecast of citations computed within a paper's own arXiv category over the last ~90 days, so impact_min is a RISING-WORK filter that DROPS all established/canonical papers; use sort='impactful' or min_citations for established work. github_stars is 0-defaulted for never-fetched repos, so treat 0 as 'unknown', not 'unpopular', and sort='community' reflects measured adoption within its window, not corpus-wide adoption. Filters: category, novelty_min, min_citations (PROVEN citations, keeps established papers), min_stars (only papers KNOWN to have >= N stars, excludes never-fetched repos), has_code, github_url_exists, days (rolling lookback) or an explicit window via published_after/published_before ('YYYY-MM-DD'), method_category, method_name, task, dataset, contribution_type, task_category, and exclude_ids for deduplication across chained calls. Pagination: either page (a 1-based page number, default 1; with limit (default 20, max 50) it returns the results offset by (page-1)*limit, and it is capped at offset 200 for query-less browse) or cursor (an opaque value taken from a previous response's next_cursor, for stable keyset pagination). verbose=true or an explicit fields list (e.g. 'arxiv_id,title,llm_summary,llm_novelty_score') returns the full 28-field paper shape with method/task/dataset extraction; otherwise the lean default shape above is used. |
Selection evidence
Confusable tool pairs.
14 pairs where similar names or overlapping descriptions may send an agent toward the wrong tool.
| Tool A | Tool B | Confidence | Why they collide |
|---|---|---|---|
list_library |
list_collections |
medium | Both descriptions literally talk about listing the user's 'saved papers' (library = flat list, collections = named groups). A task like 'show me everything I've saved' or 'list my saved papers' is ambiguous: picking list_collections yields folder names and counts instead of the actual papers, or vice versa. |
save_paper |
like_paper |
medium | like_paper is explicitly 'Distinct from save_paper,' but both are one-click 'add this paper to my stuff' actions on feed papers. A task like 'bookmark this paper, I really like it' or 'favorite it' is ambiguous between bookmarks (save) and feed taste-signal (like); the wrong pick either fails to tune the feed or fails to persist a bookmark. |
search_papers |
like_paper |
medium | Both serve a 'more like this' intent: search_papers has anchor_paper_id that 'replicates find_similar' and like_paper is described as a 'more like this calibration signal.' A task like 'show me more papers like DPO' or 'I like this one, find me more like it' could route to the wrong tool — like_paper returns nothing visible and doesn't surface similar papers. |
get_paper |
get_citations |
medium | 'Get the citations for paper X' is genuinely ambiguous between the citation count (get_paper returns citation_count) and the citation graph (get_citations returns citing/cited works). A user asking for 'the citations of 2407.15831' could get metadata without a graph, or a graph of papers when they wanted the count. |
list_watches |
check_watches |
medium | Both describe surfacing 'new matching papers since the last digest delivery': list_watches returns pending_hits counts and check_watches returns the actual new hits. A task like 'check my watches for updates' or 'anything new since the last digest?' is ambiguous between status/counts and pulling the papers themselves. |
create_collection |
add_to_collection |
medium | add_to_collection get-or-creates collections by name and auto-saves, so a task like 'create a collection for KV papers and put this in it' is satisfiable by either. The ambiguity cuts both ways: an agent may use add_to_collection (which needs an arxiv_id) for a pure 'make an empty collection X' request, or create_collection only, forgetting to file the papers. |
create_watch |
preview_watch |
medium | preview_watch is the dry-run 'tuning loop' that returns match_count and samples without persisting. A task like 'set up a watch for RAG papers, see what it catches first' can route to preview_watch (nothing actually created) or to create_watch (persists without showing the matches) — the user's intent about committing vs testing is easily misread. |
get_field_orientation |
get_foundational_lineage |
medium | Both claim to return foundational/canonical papers, differentiated only by anchor: field_orientation is topic-anchored, lineage is anchor-paper-anchored. A task like 'give me the foundational work for transformer interpretability' where the user mentions a paper title (not an arXiv ID) or conversely an area name can plausibly route to the wrong one; lineage errors if no anchor_paper_id is supplied. |
get_citations |
get_foundational_lineage |
medium | Both traverse the citation graph and return ranked related papers. A task like 'trace this paper's citation roots' or 'what does this paper build on / what are its founding works' is ambiguous between the raw citation graph (get_citations) and the niche-root/lineage analysis (get_foundational_lineage), which are conceptually overlapping outputs. |
delete_watch |
update_watch |
low | A task like 'change my watch on LoRA to a quant filter' or 'replace my watch with a different criteria' is ambiguous between in-place update and delete-and-recreate; the delete_watch description explicitly warns to use update_watch instead of delete-and-recreate, signaling this is a real slip point. |
update_watch |
preview_watch |
low | A task like 'make my watch stricter and show me what it would return' is ambiguous between mutating in place (update_watch) and dry-running the new filter (preview_watch). The wrong pick either persists without showing results or shows results without persisting. |
list_library |
ask_library |
low | A task like 'tell me about the papers I've saved' or 'what do I have saved on LoRA' is ambiguous between a flat listing (list_library, which has no topic filter) and a grounded synthesis over the library (ask_library). The agent can pick the wrong member and give either an unfiltered dump or an overbuilt synthesis. |
unsave_paper |
like_paper |
low | A task like 'I don't like this paper, get it out of my feed' is ambiguous between removal (unsave_paper, which strips it from the library and all collections) and expressing taste for feed ranking (like_paper, which is insert-only and can't express dislikes). An agent keying on the word 'like' can pick the wrong, non-removing tool. |
find_author |
co_author_graph |
low | A task like 'find who collaborates with Yann LeCun' or 'show this author's network' requires resolving the person to an author_id via find_author before co_author_graph can run. An agent can stop at find_author (returning the profile + top papers, not collaborators) or jump straight to co_author_graph without the required author_ids. |
Compare the field