01Safety
30.0 / 30
What changed in the harness
Selection accuracy 98→95, token cost up 1%, unconfirmed writes 100%→100%.
Category breakdown
Where the score comes from.
Earned points across the four signals Gradable measures. Safety and Legibility are scored out of 30; Economics and Discoverability are scored out of 20.
02Legibility
26.1 / 30
03Economics
15.7 / 20
04Discoverability
18.7 / 20
Highest-impact fix
Estimated gain +4 pointsReduce the tool-definition and task token payload
Consolidate overlapping tools and trim repetitive schema text while preserving behavior, parameter, and return semantics.
Description evidence
Defects and rewrites.
7 defects found across the exposed tool descriptions. Suggested rewrites make purpose, inputs, boundaries, and returns easier for an agent to understand.
| Tool | Defect types | Suggested rewrite |
|---|---|---|
health_check |
name_restates_behavior no_return_description |
Checks whether the ForgeMesh Anomaly Tracker API backend is reachable and operational. Returns a health status (e.g. ok/degraded) and the API's current uptime duration. |
whale_alerts |
no_return_description |
Get recent whale movements, CEX inflows/outflows, bridge activity, and stablecoin mints/burns from monitored addresses (Binance, Coinbase, Kraken, OKX, Bybit, major bridges, Tether, Circle). Returns a list of matching transaction events with amounts, addresses, and timestamps within the lookback window. Costs $0.02 USDC on Base mainnet. |
address_scan |
no_return_description |
Scan any wallet address for anomalous transaction patterns. Classifies each transaction by checking counterparties against known CEX wallets, bridges, and stablecoin issuers. Returns an anomaly score, story label, and the list of classified transactions for the address. Costs $0.03 USDC on Base mainnet. |
model_status |
no_return_description |
Get SequenceMiner model health and training stats per chain. Returns, for each monitored chain, the model's health state, training sequence count, and last retrain timestamp. Costs $0.01 USDC on Base mainnet. |
flight_scan |
no_return_description |
Sequence anomaly analysis for a named airspace region — detects unusual clustering of emergency squawks, rapid descents, or speed anomalies using the SequenceMiner engine. Returns a story label and anomaly score describing the detected pattern for the region. Costs $0.03 USDC on Base mainnet. |
trending_signal |
no_return_description |
GitHub repos with anomalous star velocity right now — early signals before mainstream discovery. Detects repos going viral (200+ stars/day), overnight explosions, fork surges, and issue floods. Returns a ranked list of trending repos with their velocity metrics and detected anomaly signals. Costs $0.02 USDC on Base mainnet. |
claude_feature_watch |
no_return_description |
Watch Anthropic/Claude repos for recently merged features — SDK changes, Claude Code updates, new capabilities. Scans anthropic-sdk-python, anthropic-sdk-typescript, claude-code, courses, and anthropic-cookbook. Returns a list of recently merged PRs/features per repo within the lookback window. Costs $0.02 USDC on Base mainnet. |
Selection evidence
Confusable tool pairs.
5 pairs where similar names or overlapping descriptions may send an agent toward the wrong tool.
| Tool A | Tool B | Confidence | Why they collide |
|---|---|---|---|
token_scan |
nft_scan |
high | Both take a generic 'token' contract-address parameter with near-identical description templates ('best-effort', 'source status', 'coverage notes'); a request like 'scan this token contract 0x...' gives no cue whether it's an ERC-20 or an NFT, so the agent could easily pick the wrong one. |
address_scan |
nft_scan |
medium | A bare 0x address in a prompt ('check this address for weird activity') doesn't indicate whether it's a wallet (address_scan) or an NFT contract (nft_scan), since both accept raw hex addresses. |
address_scan |
defi_scan |
medium | defi_scan accepts either a protocol name or a raw contract address, overlapping with address_scan's wallet-address input; a request naming a contract address for a lending/DEX pool could be routed to either tool if the agent doesn't recognize it as a known protocol. |
anomaly_scan |
mempool_anomaly |
medium | Both are chain-wide (no specific target) anomaly detectors for the same chain enum; a generic prompt like 'any anomalies on ethereum right now?' doesn't clearly indicate whether historical/windowed (anomaly_scan) or real-time mempool (mempool_anomaly) analysis is wanted. |
github_watch |
claude_feature_watch |
medium | Both watch GitHub repo activity for anomalous patterns; a request like 'watch the claude-code repo for new stuff' could be sent to github_watch (generic repo param) or claude_feature_watch (fixed Anthropic repo set with days param), since claude_feature_watch's scope isn't obvious without reading its description closely. |
Compare the field