Public assessment

Agent Relay (this workspace's own MCP fleet server)

agent-relay · v11.4.0 · scanned Aug 4, 2026

What changed in the harness

Selection accuracy 100→100, token cost up 12%, unconfirmed writes 0%→0%.

Category breakdown

Where the score comes from.

Earned points across the four signals Gradable measures. Safety and Legibility are scored out of 30; Economics and Discoverability are scored out of 20.

01Safety

0.0 / 30

0.0 out of 30

02Legibility

21.4 / 30

21.4 out of 30

03Economics

18.1 / 20

18.1 out of 20

04Discoverability

13.9 / 20

13.9 out of 20

Highest-impact fix

Estimated gain +30 points

Add explicit identity and permission preflight tools

Expose machine-readable principal/tenant confirmation and a non-mutating permission check so agents can verify both before destructive actions.

Description evidence

Defects and rewrites.

28 defects found across the exposed tool descriptions. Suggested rewrites make purpose, inputs, boundaries, and returns easier for an agent to understand.

Tool	Defect types	Suggested rewrite
`create_workspace`	no_return_description	Explicitly start a new Agent Relay workspace session and persist it for this project, returning the created workspace including its shared workspace key so other sessions can rejoin it via set_workspace_key.
`set_workspace_key`	no_return_description	Join this MCP session to an existing Agent Relay workspace using a shared workspace key, returning the joined workspace details so subsequent calls operate within it.
`register_agent`	name_restates_behavior no_return_description	Register a new agent identity in the current workspace so it can act on channels and messages, returning the agent token used to authenticate as this identity in subsequent calls.
`list_agents`	no_return_description	List agents registered in the current workspace, optionally filtering by online or offline status, and return the matching agent identities.
`query_nodes`	no_return_description	Query registered fleet nodes by capability or name and return the matching node details.
`create_channel`	name_restates_behavior no_return_description	Create a new named channel in the current workspace for agent communication, optionally with a topic, and return the created channel so it can be used for posting messages.
`list_channels`	name_restates_behavior no_return_description	List channels available in the current workspace, optionally including archived ones, and return each channel with its name and topic.
`join_channel`	name_restates_behavior params_unexplained no_return_description	Join an existing channel in the current workspace, identified by its channel name, so the acting agent can read and post in it, and return the joined channel.
`leave_channel`	name_restates_behavior params_unexplained no_return_description	Leave a channel in the current workspace, identified by its channel name, so the acting agent can no longer read or post in it, and return confirmation of the leave.
`invite_to_channel`	name_restates_behavior params_unexplained no_return_description	Invite another registered agent, identified by agent name, to join a channel in the current workspace, identified by channel name, and return the invitation result.
`set_channel_topic`	params_unexplained no_return_description	Update the topic of an existing channel in the current workspace, identified by channel name, to the new topic text, and return confirmation of the update.
`archive_channel`	name_restates_behavior params_unexplained no_return_description	Archive an existing channel in the current workspace, identified by channel name, so it is no longer active but remains available via list_channels with include_archived set, and return confirmation.
`post_message`	name_restates_behavior params_unexplained no_return_description	Post a new message to a channel in the current workspace, identified by channel name, as the acting agent, optionally attaching files and choosing wait or steer delivery, and return the posted message so it can be replied to or reacted on.
`list_messages`	params_unexplained no_return_description	Retrieve message history from a channel in the current workspace, identified by channel name, optionally limited by count or by before and after cursors, and return the messages.
`reply_to_thread`	name_restates_behavior no_return_description	Post a reply in the thread of an existing parent message, identified by message_id, as the acting agent, and return the created reply message.
`get_message_thread`	no_return_description	Retrieve the full thread of replies to a parent message, identified by message_id, optionally limited to a maximum number of replies, and return the parent message with its replies.
`send_dm`	name_restates_behavior no_return_description	Send a private direct message to another registered agent, identified by recipient name, optionally attaching files and choosing wait or steer delivery, and return the sent message.
`list_dms`	name_restates_behavior no_return_description	List the current agent's direct message conversations with other agents and return each conversation with its participants.
`send_group_dm`	no_return_description	Create a group DM with the given participant agents and send the first message, optionally naming the group, and return the created conversation.
`add_reaction`	name_restates_behavior params_unexplained no_return_description	Add an emoji reaction, given as a character or shortcode, to a message identified by message_id, as the acting agent, and return the added reaction.
`remove_reaction`	name_restates_behavior params_unexplained no_return_description	Remove an emoji reaction, given as a character or shortcode, from a message identified by message_id, as the acting agent, and return confirmation of the removal.
`search_messages`	name_restates_behavior no_return_description	Search messages across the workspace by text query, optionally filtered by channel or sender and limited by result count, and return the matching messages.
`check_inbox`	no_return_description	Check the current agent's inbox for unread messages, mentions, DMs and reactions, optionally limited to a maximum number of items, and return the pending items.
`mark_message_read`	name_restates_behavior params_unexplained no_return_description	Mark a message, identified by message_id, as read for the acting agent so it no longer appears as unread in the inbox, and return confirmation.
`get_message_readers`	params_unexplained no_return_description	List the agents who have read a message, identified by message_id, and return the matching reader agents.
`add_agent`	no_return_description	Spawn another AI agent (relay worker) to delegate a task to. This is how you create workers — including non-Claude ones. Use it for any "spawn a <tool> agent" request. Examples: "spawn a codex agent" → cli:"codex"; "spawn an opus claude agent" → cli:"claude", model:"claude-opus-4-8"; "spawn a sonnet claude agent" → cli:"claude", model:"claude-sonnet-4-6". Do NOT use the built-in Agent/Tool task for relay workers. Returns the spawned worker agent (with its assigned name and pinned model, if any) so you can track its status and later release or delete it.
`spawn`	name_restates_behavior no_return_description	Spawn a new AI agent worker in the fleet and give it optional initial task instructions. Requires name and cli; optionally pin a model, join channels, target a specific fleet node via target_node, or act as a registered identity via as. Returns the spawned agent and its session reference, which you can pass back as session_ref in a later spawn to resume the session.
`remove_agent`	no_return_description	Release a worker agent from active duty so it stops participating in fleet work. Set delete_agent to true to permanently delete the agent rather than merely releasing it; reason can record why it was removed. Returns the outcome of the removal operation.

Selection evidence

Confusable tool pairs.

4 pairs where similar names or overlapping descriptions may send an agent toward the wrong tool.

Tool A	Tool B	Confidence	Why they collide
`send_dm`	`send_group_dm`	medium	A task like 'send a direct message to Alice and Bob' or 'DM the whole team' is ambiguous: the agent may pick send_dm for its single 'to' field when multiple recipients require send_group_dm, since both share the 'send message' phrasing.
`list_messages`	`list_dms`	medium	'Show me my messages' or 'list my DMs' can map to either: list_messages returns channel history (needs a channel) while list_dms returns DM conversations, so an agent might pick the wrong one when a DM's message history is requested.
`list_messages`	`search_messages`	medium	A task like 'get the messages Alice sent about pricing' could be satisfied by either browsing channel history (list_messages) or querying across the workspace (search_messages); the missing 'search' keyword makes the choice ambiguous.
`register_agent`	`add_agent`	medium	'Add a new agent named Bob' is genuinely ambiguous: register_agent creates an agent identity in the workspace, while add_agent spawns a worker AI agent, and both accept a name, so only tool descriptions disambiguate.

Compare the field

One score is useful.
The evidence makes it actionable.

Back to the leaderboard