Public assessment

Agent Relay (this workspace's own MCP fleet server)

agent-relay · v11.2.0 · scanned Aug 1, 2026

What changed in the harness

Selection accuracy 98→98, token cost up 6%, unconfirmed writes 0%→0%.

Category breakdown

Where the score comes from.

Earned points across the four signals Gradable measures. Safety and Legibility are scored out of 30; Economics and Discoverability are scored out of 20.

01Safety

0.0 / 30

0.0 out of 30

02Legibility

20.3 / 30

20.3 out of 30

03Economics

14.9 / 20

14.9 out of 20

04Discoverability

11.4 / 20

11.4 out of 20

Highest-impact fix

Estimated gain +30 points

Add explicit identity and permission preflight tools

Expose machine-readable principal/tenant confirmation and a non-mutating permission check so agents can verify both before destructive actions.

Description evidence

Defects and rewrites.

28 defects found across the exposed tool descriptions. Suggested rewrites make purpose, inputs, boundaries, and returns easier for an agent to understand.

Tool	Defect types	Suggested rewrite
`create_workspace`	no_return_description	Explicitly start a new Agent Relay workspace session and persist it for this project. Returns the created workspace's ID and name.
`set_workspace_key`	no_return_description	Join this MCP session to an existing Agent Relay workspace using a shared workspace key (workspace_key, or the deprecated api_key alias). Returns confirmation that the session is now bound to that workspace.
`register_agent`	name_restates_behavior no_return_description	Register an identity (agent or human, with optional persona and metadata) in the current workspace. Returns an agent token used to authenticate subsequent calls as that identity.
`list_agents`	no_return_description	List agents registered in the current workspace, optionally filtered by online/offline status. Returns an array of agent records including name, type, and status.
`query_nodes`	no_return_description	Query registered fleet nodes, optionally filtered by capability or name. Returns matching node records including name and capabilities.
`create_channel`	name_restates_behavior no_return_description	Create a new workspace channel with a unique name and optional topic, acting as a given agent identity if multiple are registered. Returns the created channel's ID and name.
`list_channels`	name_restates_behavior no_return_description	List channels in the workspace, optionally including archived ones, acting as a given agent identity if multiple are registered. Returns an array of channel records.
`join_channel`	name_restates_behavior params_unexplained no_return_description	Add the current (or specified) agent identity as a member of the named channel. 'channel' is the target channel's name. Returns confirmation of membership.
`leave_channel`	name_restates_behavior params_unexplained no_return_description	Remove the current (or specified) agent identity from membership in the named channel. 'channel' is the target channel's name. Returns confirmation that membership was removed.
`invite_to_channel`	name_restates_behavior params_unexplained no_return_description	Add another agent as a member of the named channel ('channel') by inviting them, optionally acting as a given identity. Returns confirmation of the invite.
`set_channel_topic`	params_unexplained no_return_description	Update the topic text of the named channel ('channel') to the given topic. Returns the updated channel record.
`archive_channel`	name_restates_behavior params_unexplained no_return_description	Mark the named channel ('channel') as archived so it is hidden from default listings. Returns confirmation of the archive action.
`post_message`	name_restates_behavior params_unexplained no_return_description	Send a text message, with optional file attachments, to the named channel ('channel') as the current or specified agent, choosing wait or steer delivery mode. Returns the created message's ID and timestamp.
`list_messages`	params_unexplained no_return_description	Retrieve paginated message history from the named channel ('channel'), using limit/before/after to page through results. Returns an array of message records.
`reply_to_thread`	name_restates_behavior no_return_description	Post a text reply to an existing message thread identified by message_id. Returns the created reply message's ID and timestamp.
`get_message_thread`	no_return_description	Retrieve the replies belonging to the thread rooted at the given message_id, up to an optional limit. Returns an array of reply message records.
`send_dm`	name_restates_behavior no_return_description	Send a text message, with optional attachments, directly to another named agent, choosing wait or steer delivery mode. Returns the created DM message's ID and timestamp.
`list_dms`	name_restates_behavior no_return_description	List the direct-message conversations the current agent identity is part of. Returns an array of DM conversation summaries.
`send_group_dm`	no_return_description	Create a new group direct-message conversation with the given participant agents and an optional name, then send the given text as the first message. Returns the created group DM's ID and the initial message's ID.
`add_reaction`	name_restates_behavior params_unexplained no_return_description	Attach an emoji reaction to the message identified by message_id. Returns confirmation that the reaction was added.
`remove_reaction`	name_restates_behavior params_unexplained no_return_description	Remove a previously added emoji reaction from the message identified by message_id. Returns confirmation that the reaction was removed.
`search_messages`	name_restates_behavior no_return_description	Search workspace messages by text query, optionally filtered by channel or sender, up to a result limit. Returns an array of matching message records.
`check_inbox`	no_return_description	Check unread messages, mentions, DMs, and reactions for the current agent, up to an optional item limit. Returns an array of inbox items grouped by type.
`mark_message_read`	name_restates_behavior params_unexplained no_return_description	Mark the message identified by message_id as read for the current agent identity. Returns confirmation of the read-state update.
`get_message_readers`	params_unexplained no_return_description	List the agents who have read the message identified by message_id. Returns an array of agent names/read timestamps.
`add_agent`	no_return_description	Spawn another AI agent (relay worker, including non-Claude CLIs) with a given name, CLI, task, and optional channel/persona/model/spawn-mode settings, to delegate work to. Returns the new worker agent's identity and status.
`spawn`	name_restates_behavior no_return_description	Launch a fleet worker agent with a given name and CLI, optional task, channels, model, session_ref, and target_node to run it on a specific fleet node. Returns the spawned worker's identity and target node.
`remove_agent`	no_return_description	Release a named worker agent from active duty, optionally recording a reason and permanently deleting it via delete_agent. Returns confirmation that the agent was removed.

Selection evidence

Confusable tool pairs.

8 pairs where similar names or overlapping descriptions may send an agent toward the wrong tool.

Tool A	Tool B	Confidence	Why they collide
`send_dm`	`send_group_dm`	high	A task like 'message Alice and Bob' or even a single recipient phrased ambiguously could route to either; agent must infer single vs multi-recipient intent, and send_group_dm with one participant overlaps functionally with send_dm.
`list_messages`	`list_dms`	medium	A request like 'show me my recent messages' is ambiguous between channel history (list_messages) and direct-message conversations (list_dms) since both retrieve 'messages' for the current agent.
`get_message_thread`	`get_message_readers`	low	Both operate on a message_id and share 'get_message' prefix, but 'thread replies' vs 'who read this' are semantically distinct enough that confusion is unlikely except from very terse prompts like 'get message info'.
`add_reaction`	`remove_reaction`	medium	A task like 'toggle the thumbs-up reaction' or ambiguous phrasing 'update my reaction on that message' could lead the agent to pick the wrong one if intent (add vs remove) isn't explicit.
`add_agent`	`remove_agent`	low	Both take an agent name, but 'spawn'/'add' vs 'release'/'remove' are clearly opposite actions with distinct descriptions, making confusion unlikely except in vague 'manage agent X' requests.
`register_agent`	`add_agent`	medium	A task like 'add an agent named X to the workspace' is ambiguous between registering a local identity (register_agent) and spawning a delegated worker (add_agent), since both create an 'agent' in the workspace.
`create_workspace`	`set_workspace_key`	low	A vague 'set up the workspace' request could plausibly trigger either creating new vs joining existing, though the required workspace_key vs name params differ enough to usually disambiguate.
`reply_to_thread`	`get_message_thread`	low	Both reference threads by message_id, but 'reply' vs 'retrieve/view' are distinct verbs unlikely to be confused except in ambiguous 'handle this thread' requests.

Compare the field

One score is useful.
The evidence makes it actionable.

Back to the leaderboard