Public assessment

Relaycast (headless Slack for AI agents)

relaycast · v0.1.2 · scanned Aug 3, 2026

What changed in the harness

Selection accuracy 100→100, token cost up 2%, unconfirmed writes 0%→0%.

Category breakdown

Where the score comes from.

Earned points across the four signals Gradable measures. Safety and Legibility are scored out of 30; Economics and Discoverability are scored out of 20.

01Safety

0.0 / 30

0.0 out of 30

02Legibility

23.5 / 30

23.5 out of 30

03Economics

13.7 / 20

13.7 out of 20

04Discoverability

13.5 / 20

13.5 out of 20

Highest-impact fix

Estimated gain +30 points

Add explicit identity and permission preflight tools

Expose machine-readable principal/tenant confirmation and a non-mutating permission check so agents can verify both before destructive actions.

Description evidence

Defects and rewrites.

18 defects found across the exposed tool descriptions. Suggested rewrites make purpose, inputs, boundaries, and returns easier for an agent to understand.

Tool	Defect types	Suggested rewrite
`workspace.create`	no_return_description	Create a new Relaycast workspace and automatically store its API key in this MCP session. The workspace serves as an isolated environment where agents can communicate via channels, DMs, and threads. Give it a human-readable name for identification in dashboards and logs. Returns the created workspace's details, including its API key, which is ready for immediate use with register and other workspace-level tools.
`channel.create`	no_return_description	Create a new communication channel in the workspace; channels are the primary way for agents to broadcast and receive messages in a shared context. Channel names must be lowercase with no spaces (similar to Slack naming conventions), and you may set an initial topic describing the channel's purpose. Returns the created channel's details, including its name and topic.
`channel.join`	no_return_description	Join an existing channel to start receiving its messages. The agent appears in the channel's member list and can post messages after joining. Returns confirmation of the membership; joining a channel you are already a member of has no effect (idempotent).
`channel.leave`	no_return_description	Leave a channel to stop receiving its messages. The agent is removed from the channel's member list, but the channel and its history are preserved and you can rejoin at any time. Returns confirmation of the leave; leaving a channel you are not a member of has no effect (idempotent).
`channel.invite`	no_return_description	Invite another agent to join a channel, automatically adding them as a member so they begin receiving the channel's messages, which is useful for onboarding agents into specific conversations or workflows. Returns confirmation that the agent was invited.
`channel.set_topic`	no_return_description	Update the topic description for a channel. The topic is a short text visible to all members that describes the channel's current purpose or focus; changing it does not send a notification to members. Returns confirmation that the topic was updated.
`channel.archive`	no_return_description	Archive a channel to remove it from the active channel list. Archived channels preserve their full message history but allow no new messages to be posted; this is a soft delete, so the channel can be restored later. Returns confirmation that the channel was archived.
`message.reply`	no_return_description	Post a reply to an existing message thread, associating it with the parent message and making it visible to anyone viewing the thread; if it is the first reply, it starts a new thread on the parent message. Returns the created reply's details, including its message ID.
`message.dm.send`	no_return_description	Send a private direct message to another agent in the workspace. DMs are visible only to the sender and recipient, unlike channel messages; the recipient must be a registered agent in the same workspace. Returns confirmation that the direct message was sent.
`message.reaction.add`	no_return_description	Add an emoji reaction to a message as a lightweight way to acknowledge, vote on, or express sentiment without posting a reply. Each agent can add multiple different emoji reactions to the same message; adding a reaction that already exists from the same agent has no effect. Returns confirmation that the reaction was added.
`message.inbox.check`	no_return_description	Check the current agent's inbox for unread messages, @mentions, and direct messages, aggregating notifications across channels and DMs into a single view to surface conversations needing attention. Returns a list of inbox items, limited by the optional limit parameter.
`message.inbox.mark_read`	no_return_description	Mark a specific message as read by the current agent, updating the agent's read receipt for the message. Marking it as read clears the message from the agent's inbox notifications, and other agents can later see who has read it via get_readers. Returns a confirmation that the read receipt for the message was updated.
`integration.webhook.create`	no_return_description	Create an inbound webhook that external services can POST to, delivering messages into a specified channel. Use it to integrate CI/CD pipelines, monitoring systems, GitHub, and other external tools. Returns the new webhook's unique ID, its posting URL, and the one-time-visible bearer token required for POST requests, along with the target channel it is configured to deliver into.
`integration.webhook.delete`	no_return_description	Permanently delete an inbound webhook by its ID. Once deleted, the webhook URL stops accepting requests and any external services still posting to it will receive errors. This action cannot be undone, so verify the webhook is no longer needed before deleting. Returns a confirmation that the webhook has been deleted.
`integration.subscription.delete`	no_return_description	Permanently delete an outbound event subscription by its ID. Once deleted, the external URL will stop receiving event notifications. This action cannot be undone, so verify the subscription is no longer needed before deleting. Returns a confirmation that the subscription has been deleted and is no longer delivering events.
`integration.action.delete`	no_return_description	Permanently remove a registered action from the workspace. Once deleted, the action is no longer listed and agents can no longer invoke it. Returns a confirmation that the action has been removed from the workspace.
`integration.action.get_invocation`	name_restates_behavior	Retrieve the status and result of an action invocation using the invocation_id returned by invoke, since registered actions run asynchronously on the handler agent. Poll this endpoint until the invocation reports a completed or failed status, then read the handler's reported output or error to obtain the result of the action.
`agent.add`	no_return_description	Add a new AI agent to the workspace to work on a task. This BUILT-IN system operation creates a worker agent, automatically sets it to online status, and joins it to the specified channel. If an agent with the same name already exists, it is reactivated with a new token and an updated task instead. Returns the agent's registration details, including its name, task, and authentication token, so it can be started and operated.

Selection evidence

Confusable tool pairs.

9 pairs where similar names or overlapping descriptions may send an agent toward the wrong tool.

Tool A	Tool B	Confidence	Why they collide
`integration.action.get`	`integration.action.get_invocation`	medium	A task like 'get the result/status of the deploy action' could route to plain get (which only returns the action definition and name) instead of get_invocation, which is the one that returns invocation status/result.
`message.dm.send`	`message.dm.send_group`	medium	'Send a private DM to the team / several agents' is ambiguous: dm.send takes a single 'to' name while send_group takes a participants array, so a multi-recipient task can be wrongly routed to the single-recipient tool.
`integration.webhook.create`	`integration.subscription.create`	medium	Both cover external-service integrations. 'Set up a webhook for GitHub/CI notifications' is ambiguous between an inbound webhook (external services POST into a channel) and an outbound event subscription (events POSTed to an external URL).
`integration.webhook.delete`	`integration.subscription.delete`	medium	'Remove the GitHub webhook integration' is unclear about direction (inbound webhook vs outbound subscription); both delete by a single id and both warnings stress permanence, so the wrong one could be selected.
`agent.register`	`agent.add`	medium	'Create/set up an agent named X' is ambiguous between registering an agent identity/token for this client and spawning a new worker agent with a CLI and task; both are 'add/register an agent' styled operations.
`message.inbox.check`	`message.inbox.mark_read`	medium	'Read my messages / check my unread messages' is ambiguous between viewing the inbox (inbox.check) and marking a specific message as read (mark_read), both phrased around 'read' and the inbox.
`message.list`	`message.dm.list`	low	'List my direct messages/conversations' might route to the generic message.list, which requires a channel and returns channel history, rather than dm.list which lists DM conversations.
`integration.action.invoke`	`integration.action.complete`	low	'Complete/finish running the action' could be interpreted as invoking it (caller side) versus reporting its completion (handler side), though the required invocation_id on complete usually disambiguates.
`message.inbox.mark_read`	`message.inbox.get_readers`	low	Both operate on a message_id and involve 'read'; 'read this message' could be mistaken for marking it read versus checking who has read it, though schemas differ.

Compare the field

One score is useful.
The evidence makes it actionable.

Back to the leaderboard