01Safety
0.0 / 30
Public leaderboard
Public assessment
pineforge-4pass/pineforge-codegen-mcp (@pineforge/backtest-mcp)
What changed in the harness
Selection accuracy 95→95, token cost up 1%, unconfirmed writes 0%→0%.
Category breakdown
Where the score comes from.
Earned points across the four signals Gradable measures. Safety and Legibility are scored out of 30; Economics and Discoverability are scored out of 20.
02Legibility
27.6 / 30
03Economics
16.1 / 20
04Discoverability
12.6 / 20
Highest-impact fix
Estimated gain +30 pointsAdd explicit identity and permission preflight tools
Expose machine-readable principal/tenant confirmation and a non-mutating permission check so agents can verify both before destructive actions.
Description evidence
Defects and rewrites.
2 defects found across the exposed tool descriptions. Suggested rewrites make purpose, inputs, boundaries, and returns easier for an agent to understand.
| Tool | Defect types | Suggested rewrite |
|---|---|---|
list_coverage_topics |
no_return_description |
Returns a coverage overview of the PineForge Pine v6 engine before writing, porting, or backtesting a strategy: an object containing the list of every coverage topic (each with its id, one-line status — supported / partial / unsupported / via_transpiler — and summary), the legend explaining status semantics (via_transpiler still works end-to-end; unsupported means parsed-and-skipped or rejected), and the coverage version. PineForge implements a SUBSET of Pine v6, so checking coverage first avoids a strategy that compiles but silently misbehaves vs TradingView. Cheap, free, local — no engine run, no I/O. Then drill in with get_coverage_topic for one area's full supported/unsupported lists, or check_pine_feature to look up a single identifier. |
pull_engine_image |
no_return_description |
Runs `docker pull` to fetch the pineforge-release runtime Docker image on the user's machine so it is present locally before the first backtest_pine call. Returns the outcome of the pull (success/failure with the docker output or error), indicating whether the runtime image is ready to use for transpile/backtest runs. |
Selection evidence
Confusable tool pairs.
4 pairs where similar names or overlapping descriptions may send an agent toward the wrong tool.
| Tool A | Tool B | Confidence | Why they collide |
|---|---|---|---|
backtest_pine |
backtest_pine_grid |
high | Both accept the same Pine source + OHLCV CSV and share the backtest_pine prefix. Tasks like 'try a few different fast lengths and see which performs best' or 'run my strategy with several settings' straddle the single-vs-sweep boundary: unless the user says optimize/sweep/tune, an agent can pick single-run backtest_pine when a grid was wanted, or the grid tool for an ambiguous one-off. |
list_coverage_topics |
get_coverage_topic |
medium | Near-identical names and both are local coverage lookups. 'Check what PineForge supports for request.security' could route to list_coverage_topics (topic overview) instead of get_coverage_topic (detail + supported/unsupported lists), giving an incomplete answer; conversely 'what's covered?' could route to the drill-in tool before the topic id is known. |
pull_engine_image |
check_engine_image |
medium | Both act on the same local pineforge-release Docker image and overlap in behavior (check_engine_image with auto_pull performs the pull). 'Make sure my engine image is up to date / set up' is ambiguous between downloading (pull) and verifying currentness (check), so the agent may call pull when a status check was intended or the report-only check when a pull was wanted. |
transpile_pine |
backtest_pine |
low | Both take Pine v6 source and share pine/script tokens. 'Compile/run/process this Pine strategy and show what it does' is genuinely ambiguous between emitting the C++ translation unit (transpile) and executing the strategy (backtest); the cross-references reduce but don't eliminate a wrong pick for vague 'run this script' phrasing. |
Compare the field