Public assessment

vincentvella/devloop (devloop-mcp)

devloop-mcp · v0.10.2 · scanned Aug 3, 2026

What changed in the harness

Selection accuracy 98→96, token cost up 3%, unconfirmed writes 0%→0%.

Category breakdown

Where the score comes from.

Earned points across the four signals Gradable measures. Safety and Legibility are scored out of 30; Economics and Discoverability are scored out of 20.

01Safety

0.0 / 30

0.0 out of 30

02Legibility

18.9 / 30

18.9 out of 30

03Economics

16.0 / 20

16.0 out of 20

04Discoverability

13.4 / 20

13.4 out of 20

Highest-impact fix

Estimated gain +30 points

Add explicit identity and permission preflight tools

Expose machine-readable principal/tenant confirmation and a non-mutating permission check so agents can verify both before destructive actions.

Description evidence

Defects and rewrites.

28 defects found across the exposed tool descriptions. Suggested rewrites make purpose, inputs, boundaries, and returns easier for an agent to understand.

Tool	Defect types	Suggested rewrite
`browser_back`	no_return_description	Go back one entry in the active pane's history (no-op if there's nothing to go back to). Returns the resolved URL and HTTP status of the page now shown. Pairs with browser_forward; use browser_navigate to go to a new URL.
`browser_forward`	no_return_description	Go forward one entry in the active pane's history (no-op if there's nothing to go forward to). Returns the resolved URL and HTTP status of the page now shown. Pairs with browser_back; use browser_navigate to go to a new URL.
`browser_reload`	name_restates_behavior no_return_description	Re-fetch the active pane's current page from the server; set hard=true to bypass the cache (default false, Cockpit only; ignored under Puppeteer). Returns the resolved URL and HTTP status of the reloaded page. Use browser_navigate to go to a different URL instead.
`browser_click`	no_return_description	Click the element matching a CSS selector with a real mouse click. Returns the click result or an error if the selector matches nothing. For keyboard keys use browser_press; to only hover use browser_hover.
`browser_type`	no_return_description	Type text into the element matching a CSS selector (focuses it first, then types). Returns the result or an error if the element isn't found. For single keys use browser_press; for <select> use browser_select.
`browser_hover`	name_restates_behavior no_return_description	Move the pointer over the element matching the selector so hover-dependent UI (menus, tooltips) appears without triggering a click. Returns the result or an error if the selector matches nothing. For actual clicks use browser_click.
`browser_select`	no_return_description	Set the value of a <select> (or input) and fire input/change events so listeners update. Returns the result or an error if the element isn't found. For typing free text use browser_type.
`browser_press`	no_return_description	Press a single key (e.g. Enter, Escape, Tab, ArrowDown); optionally focus a CSS selector first. Returns the result or an error if the target element isn't found. For typing text use browser_type.
`browser_emulate`	no_return_description	Emulate a device/viewport on the active page — pass a `device` preset, custom `width`+`height`, or `reset: true` to restore a desktop viewport. Returns the applied emulation settings. For network conditions use browser_throttle instead.
`browser_throttle`	no_return_description	Throttle the active page's network conditions to a named profile (slow-3g, fast-3g, offline); pass `none` to clear throttling. Returns the profile now applied. For viewport/device emulation use browser_emulate.
`browser_clear_storage`	no_return_description	Clear the active page's storage — cookies, localStorage, IndexedDB, cache, service workers — for the current origin, or the whole session with allOrigins:true. Use to log out or test a fresh-user flow; reload afterward. Returns an ok status / count of cleared storage.
`clear_logs`	no_return_description	Clear the event buffer (irreversible). Call before reproducing an issue for a clean window. Returns the number of events removed. Note: repro clears for you unless clear=false.
`export_har`	no_return_description	Export captured network requests as a HAR 1.2 document (importable into Chrome DevTools / Charles) and return it as JSON. Covers ALL requests (the full network ring, independent of DEVLOOP_NET_THRESHOLD — bodies kept for the curated subset: failures + status ≥ threshold). To browse requests in JSON use get_network.
`get_network`	no_return_description	List captured network requests (the full ring — every request, independent of DEVLOOP_NET_THRESHOLD). Each row has method/url/status/timing/headers (bodies for the curated subset); returns rows most recent first. Unlike get_logs (curated timeline only); for an importable HAR file use export_har.
`dev_start`	no_return_description	Start a dev server and tee its logs into the buffer. Specify it as (1) a saved `project` (resolves cmd/cwd), (2) explicit `cmd`+`cwd`, or (3) neither — `cwd` defaults to the server's directory and `cmd` is auto-detected from package.json; explicit cmd/cwd override the project's. Returns the started process info (cmd/cwd/pid). Stop it with dev_stop; check it with dev_status.
`project_list`	no_return_description	Return the saved projects (name, cwd, cmd, url) from the registry. Add with project_add, remove with project_remove.
`project_add`	no_return_description	Save (or replace) a project in the registry so you can dev_start it by name; replaces any existing project with the same name. Returns the saved project. name and cwd are required; cmd is auto-detected if omitted and url is optional.
`project_remove`	no_return_description	Remove a saved project from the registry by its unique name (irreversible; doesn't stop a running server). Returns the removed project or an ok status. List names with project_list.
`pane_list`	no_return_description	Return the browser panes (multi-target), each with id, url, active. The active pane is what browser_*/repro act on. Switch with pane_select. (Cockpit only.)
`pane_new`	no_return_description	Open a new browser pane, make it active, and return its pane id/url. Optionally navigate it to a URL and/or scope it to a project: pass a saved `project` (resolves cwd/cmd/url/name) or explicit `cwd`/`cmd`/`label` — cwd isolates its storage partition and names the pane. With none of these it's a blank, unscoped pane. To switch panes use pane_select; to close use pane_close. (Cockpit only.)
`pane_select`	no_return_description	Make the pane with the given id active so subsequent browser_*/repro calls target it; returns an ok status. Find ids with pane_list. (Cockpit only.)
`pane_close`	no_return_description	Close the browser pane with the given id (irreversible) and return an ok status. To detach a pane into its own window instead, use pane_pop. (Cockpit only.)
`pane_pop`	no_return_description	Detach the pane with the given id into its own standalone window so you can view targets side-by-side; closing that window re-docks the pane. Returns the detached pane info. pane_close removes a pane entirely. (Cockpit only.)
`pane_set_label`	no_return_description	Set a pane's display label (the tab name), which is also what `app` filters match in get_logs/diagnose; returns the updated pane. (Cockpit only.)
`dev_status`	no_return_description	Return { running, cmd, cwd, pid } reporting whether the dev server is running and its details. Start/stop with dev_start/dev_stop.
`native_close`	no_return_description	Close the active native target so browser_* route back to the pane's web content; returns an ok status. (Cockpit only.) Open one with native_open.
`ext_list`	no_return_description	Return the loaded + disabled Chrome extensions, each with id, name, version, enabled. (Cockpit only.) Install with ext_install, toggle with ext_set_enabled.
`ext_remove`	no_return_description	Remove (uninstall/unload) the Chrome extension with the given id (irreversible for store extensions) and return the updated extension list. (Cockpit only.) To keep it but turn it off, use ext_set_enabled.

Selection evidence

Confusable tool pairs.

10 pairs where similar names or overlapping descriptions may send an agent toward the wrong tool.

Tool A	Tool B	Confidence	Why they collide
`browser_wait_for_idle`	`browser_wait_for`	high	Identical 'wait' prefix, overlapping timeoutMs schema, and tasks like 'wait until the page is ready' genuinely ambiguous between network idle and element presence; descriptions even cross-reference each other.
`get_logs`	`get_logs_around`	high	Nearly identical names both returning logs; 'get logs around the error time' could map to either, and an agent without an explicit ts may pick get_logs_around and wrongly fail, or pick get_logs and miss correlation.
`browser_back`	`browser_forward`	high	Mirrored paired history tools; a task phrased as 'go to the previous/next page in history' without the explicit word back/forward can route to the wrong direction, and summaries often conflate the two.
`browser_click`	`browser_type`	medium	Both take a CSS selector for an element and are interaction verbs; tasks like 'click into the search field and enter text' are mostly clear, but phrasing like 'enter the field' or 'press the input' can misroute. Verb distinction usually disambiguates.
`browser_emulate`	`browser_throttle`	medium	Both simulate conditions on the active page (device vs network) and cross-reference each other in descriptions; a task like 'simulate a slow mobile connection' or 'make the page mobile-like' requires reading carefully to pick device vs network.
`browser_snapshot`	`browser_wait_for`	medium	Both take a selector and concern page state; 'get the state once the element appears' could be served by either, and an agent may snapshot prematurely or wait without capturing.
`export_bundle`	`export_har`	medium	Both export captured data including network; a task like 'export the network capture for the report' is ambiguous between the full bug bundle and the HAR document.
`get_logs`	`clear_logs`	medium	Shared 'logs' token; phrasing like 'reset the logs' or 'give me a fresh log view' is ambiguous between retrieving and wiping, though get vs clear is usually distinct.
`get_logs`	`get_network`	medium	Both retrieve captured events; 'get the failing requests' could mean the curated timeline via get_logs or the full request ring via get_network, and agents may not realize get_logs filters by threshold.
`pane_new`	`pane_select`	medium	Both concern making a pane active; a task like 'open the dev pane for project X' is ambiguous between creating a new scoped pane and selecting an existing one without checking pane_list first.

Compare the field

One score is useful.
The evidence makes it actionable.

Back to the leaderboard