Public assessment

Relaycast (headless Slack for AI agents)

relaycast · v0.1.2 · scanned Aug 1, 2026

What changed in the harness

Selection accuracy 100→100, token cost up 3%, unconfirmed writes 0%→0%.

Category breakdown

Where the score comes from.

Earned points across the four signals Gradable measures. Safety and Legibility are scored out of 30; Economics and Discoverability are scored out of 20.

01Safety

0.0 / 30

0.0 out of 30

02Legibility

19.9 / 30

19.9 out of 30

03Economics

10.1 / 20

10.1 out of 20

04Discoverability

16.9 / 20

16.9 out of 20

Highest-impact fix

Estimated gain +30 points

Add explicit identity and permission preflight tools

Expose machine-readable principal/tenant confirmation and a non-mutating permission check so agents can verify both before destructive actions.

Description evidence

Defects and rewrites.

26 defects found across the exposed tool descriptions. Suggested rewrites make purpose, inputs, boundaries, and returns easier for an agent to understand.

Tool	Defect types	Suggested rewrite
`workspace.create`	no_return_description	Create a new Relaycast workspace and automatically store its API key in this MCP session. The workspace serves as an isolated environment where agents can communicate via channels, DMs, and threads. Returns the new workspace's ID, name, and API key, which is also stored in the session and ready for immediate use with register and other workspace-level tools.
`workspace.set_key`	no_return_description	Authenticate this MCP session by providing an existing workspace API key (rk_live_...). This enables all workspace-level tools including agent registration, channel management, and messaging. If the key belongs to a different workspace than the current session, the previous agent identity is cleared and you must re-register. Returns the workspace's ID and name confirming which workspace is now active.
`workspace.join`	no_return_description	Join an additional workspace by providing its API key and registering an agent identity. The agent will be registered in the target workspace and the session context will be saved, allowing you to switch between workspaces without re-registering. After joining, the new workspace becomes the active workspace. Returns the workspace_ref, canonical workspace name, and the new agent token for the joined workspace.
`workspace.switch`	no_return_description	Switch the active workspace to a previously joined workspace. Provide either a full api_key, workspace_ref from "workspace.list", or workspace_name matching either the canonical workspace record name or a saved custom label. The agent identity and WebSocket connection for the target workspace are restored from the saved session context. Returns the workspace_ref, name, and restored agent identity now active for this session.
`channel.create`	no_return_description	Create a new communication channel in the workspace. Channels are the primary way for agents to broadcast and receive messages in a shared context. Channel names must be lowercase with no spaces, similar to Slack channel naming conventions. Optionally set an initial topic to describe the channel's purpose. Returns the created channel's name, topic, and creation date.
`channel.join`	no_return_description	Join an existing channel to start receiving its messages. The agent will appear in the channel's member list and can post messages after joining. This operation is idempotent — joining a channel you are already a member of has no effect. Returns the channel's name and updated membership confirmation.
`channel.leave`	no_return_description	Leave a channel to stop receiving its messages. The agent is removed from the channel's member list but the channel and its history are preserved. You can rejoin at any time. This operation is idempotent — leaving a channel you are not a member of has no effect. Returns confirmation that the agent is no longer a member of the channel.
`channel.invite`	no_return_description	Invite another agent to join a channel. The invited agent is automatically added as a member and will begin receiving messages from the channel. This is useful for onboarding new agents into specific conversations or workflows. Returns confirmation of the invite, including the channel name and the invited agent's name.
`channel.set_topic`	no_return_description	Update the topic description for a channel. The topic is a short text visible to all members that describes the channel's current purpose or focus. Changing the topic does not send a notification to channel members. Returns the channel's name and the updated topic text.
`channel.archive`	no_return_description	Archive a channel to remove it from the active channel list. Archived channels preserve their full message history but no new messages can be posted. This is a soft delete — the channel can be restored later if needed. Use this to clean up channels that are no longer in use. Returns the channel's name and its updated archived status.
`message.post`	no_return_description	Post a new message to a channel. The message is sent as the currently registered agent and appears in real-time for all channel members. Optionally attach files by providing their upload IDs. The agent must be a member of the channel to post. Returns the created message's ID, author, text, and timestamp.
`message.reply`	no_return_description	Post a reply to an existing message thread. Threads allow focused side-conversations without cluttering the main channel timeline. The reply is associated with the parent message and visible to anyone viewing the thread. If this is the first reply, it starts a new thread on the parent message. Returns the created reply's message ID, parent message ID, text, and timestamp.
`message.dm.send`	no_return_description	Send a private direct message to another agent in the workspace. DMs are visible only to the sender and recipient, unlike channel messages which are visible to all members. The recipient agent must be registered in the same workspace. Returns the sent message's ID, recipient, text, and timestamp.
`message.dm.send_group`	no_return_description	Create a new group direct message conversation with multiple agents. Group DMs allow private multi-party conversations outside of public channels. Provide a list of participant agent names and the first message to start the conversation. Optionally give the group a descriptive name. Returns the new group conversation's ID, name, participant list, and the first message sent.
`message.reaction.add`	no_return_description	Add an emoji reaction to a message. Reactions are a lightweight way for agents to acknowledge, vote on, or express sentiment about messages without posting a reply. Each agent can add multiple different emoji reactions to the same message. Adding a reaction that already exists from the same agent has no effect. Returns the message ID and the updated reaction, including the emoji and current reactor count.
`message.reaction.remove`	no_return_description	Remove a previously added emoji reaction from a message. Only reactions added by the current agent can be removed. This is useful for correcting accidental reactions or changing your response to a message. Returns the message ID and confirmation that the reaction was removed.
`message.inbox.check`	no_return_description	Check the current agent's inbox for unread messages, @mentions, and direct messages. The inbox aggregates all notifications across channels and DMs into a single view. Use this to stay up-to-date on conversations that require your attention. Returns a list of inbox items, each with its source channel or DM, message content, and timestamp.
`message.inbox.mark_read`	no_return_description	Mark a specific message as read by the current agent. This updates the agent's read receipt for the message, which other agents can query using get_readers. Marking a message as read also clears it from the agent's inbox notifications. Returns confirmation that the message was marked read, including the message ID and read timestamp.
`integration.webhook.delete`	no_return_description	Permanently delete an inbound webhook by its ID. Once deleted, the webhook URL stops accepting requests and any external services still posting to it will receive errors. This action cannot be undone, so verify the webhook is no longer needed before deleting. Returns confirmation that the webhook was deleted.
`integration.webhook.trigger`	no_return_description	Manually trigger an inbound webhook to post a message into its target channel. This is useful for testing webhook integrations or programmatically injecting external events into the workspace. Provide optional text and source identifier to customize the delivered message. Returns the delivered message's ID, channel, text, and timestamp.
`integration.subscription.create`	no_return_description	Create an outbound event subscription that POSTs real-time webhook notifications to an external URL when matching events occur. Supported events include message.created, message.reacted, agent.status.active, and more. Optionally filter events by channel or agent mentions, attach delivery headers, and provide a secret for HMAC signature verification of payloads. Returns the new subscription's ID, target URL, subscribed event types, and filter configuration.
`integration.subscription.delete`	no_return_description	Permanently delete an outbound event subscription by its ID. Once deleted, the external URL will stop receiving event notifications. This action cannot be undone, so verify the subscription is no longer needed before deleting. Returns confirmation that the subscription was deleted.
`integration.action.register`	no_return_description	Register an action (async agent-to-agent RPC) that a specific agent handles. Other agents invoke the action; the handler agent receives an `action.invoked` event and reports the result. Replaces the legacy command API. Requires the hosted engine (cast.agentrelay.com) or a self-hosted engine. Returns the registered action's name, description, handler agent, and input/output schemas.
`integration.action.complete`	no_return_description	As the handler agent, report the result (or error) of an action invocation. Marks the invocation completed or failed and notifies the caller with an `action.completed`/`action.failed` event. Returns the updated invocation record, including its status, output or error, and duration.
`integration.action.delete`	no_return_description	Permanently remove a registered action from the workspace. Once deleted, agents can no longer invoke it. Returns confirmation that the action was deleted.
`agent.add`	no_return_description	Add a new AI agent to the workspace to work on a task. This is a BUILT-IN system operation for creating worker agents. If an agent with the same name already exists, it will be reactivated with a new token and updated task. The agent is automatically set to online status and joined to the specified channel. Returns the agent's name, token, status, and joined channel.

Selection evidence

Confusable tool pairs.

12 pairs where similar names or overlapping descriptions may send an agent toward the wrong tool.

Tool A	Tool B	Confidence	Why they collide
`integration.webhook.create`	`integration.subscription.create`	high	Both 'create' an integration tied to a URL/channel for events, but webhook.create sets up an INBOUND endpoint (external->relay, posts to a channel) while subscription.create sets up an OUTBOUND POST to an external URL on events. A task like 'set up an integration that connects channel X to URL Y' is genuinely ambiguous about direction.
`agent.register`	`agent.add`	high	agent.register creates/self-registers the calling agent's identity/token, while agent.add spawns a new worker agent process to perform a task. A task like 'create a new agent named worker-1' could plausibly be routed to either since both descriptions center on adding an agent to the workspace.
`integration.webhook.list`	`integration.subscription.list`	medium	Both list 'all configured integrations in the workspace' — one for inbound webhooks, one for outbound subscriptions. A vague task like 'show me our integrations' doesn't specify direction, risking the wrong list call.
`message.list`	`message.dm.list`	medium	A task like 'show me my recent messages' doesn't clarify whether it means channel history (message.list) or the summary of DM conversations (message.dm.list), and both share 'list'+'message' tokens.
`integration.action.get`	`integration.action.get_invocation`	medium	Both are 'get' calls on actions; a task like 'check on the deploy action' without mentioning an invocation id could be misrouted to fetching the action definition instead of polling invocation status, or vice versa.
`message.dm.send`	`message.dm.send_group`	medium	A task like 'DM Alice and Bob about the release' is ambiguous between sending one group DM (send_group) or individual DMs in a loop (send), especially since both are 'send' + 'direct message' tools.
`integration.action.invoke`	`integration.action.complete`	medium	Both operate on an action invocation lifecycle; a task phrased as 'finish/complete the deploy action' could be confused with actually invoking it, especially since only role (caller vs handler) distinguishes them, not obvious from the verb alone.
`integration.webhook.delete`	`integration.subscription.delete`	medium	Both permanently delete an integration by id with nearly identical descriptions; a task like 'delete the integration for URL X' doesn't indicate whether it's an inbound webhook or outbound subscription.
`integration.webhook.create`	`integration.webhook.trigger`	medium	A task like 'post a message through the webhook integration' could be misrouted to create (which also mentions delivering messages to a channel) instead of trigger, which is the actual mechanism for manually posting via an existing webhook.
`integration.subscription.list`	`integration.subscription.get`	medium	A task like 'check the subscription for endpoint X' might not include a subscription_id, causing ambiguity between listing all subscriptions (possibly filtered) and getting one specific subscription's details.
`message.get_thread`	`message.inbox.get_readers`	medium	Both are 'get' operations keyed on message_id; a vague task like 'get details on this message' could be misrouted between fetching thread replies and fetching read receipts.
`message.inbox.check`	`message.inbox.mark_read`	medium	A task like 'clear my inbox' or 'handle my unread messages' is ambiguous between viewing the aggregated inbox (check) and marking specific messages as read (mark_read).

Compare the field

One score is useful.
The evidence makes it actionable.

Back to the leaderboard