Public assessment

gogabrielordonez/mcp-ragchat (mcp-ragchat)

mcp-ragchat · v1.0.0 · scanned Aug 3, 2026

What changed in the harness

Selection accuracy 100→100, token cost up 3%, unconfirmed writes 0%→0%.

Category breakdown

Where the score comes from.

Earned points across the four signals Gradable measures. Safety and Legibility are scored out of 30; Economics and Discoverability are scored out of 20.

01Safety

0.0 / 30

0.0 out of 30

02Legibility

27.2 / 30

27.2 out of 30

03Economics

20.0 / 20

20.0 out of 20

04Discoverability

11.6 / 20

11.6 out of 20

Highest-impact fix

Estimated gain +30 points

Add explicit identity and permission preflight tools

Expose machine-readable principal/tenant confirmation and a non-mutating permission check so agents can verify both before destructive actions.

Description evidence

Defects and rewrites.

3 defects found across the exposed tool descriptions. Suggested rewrites make purpose, inputs, boundaries, and returns easier for an agent to understand.

Tool	Defect types	Suggested rewrite
`ragchat_setup`	no_return_description	Initialize a domain with a knowledge base from markdown content. Each ## section becomes a searchable document with vector embeddings. This is the first step — run this before testing or serving. Returns a confirmation of the created domain along with the number of documents ingested.
`ragchat_serve`	no_return_description	Start a local HTTP chat server for a domain. The server runs on localhost and handles POST /chat requests. Use ragchat_widget to get the embed code that connects to this server. Returns the server URL (http://localhost:<port>) once the server is listening.
`ragchat_status`	no_return_description	List all configured domains with document counts and config status. Shows what's been set up and what's ready to serve. Returns a list of domains along with their document counts and readiness to serve.

Selection evidence

Confusable tool pairs.

10 pairs where similar names or overlapping descriptions may send an agent toward the wrong tool.

Tool A	Tool B	Confidence	Why they collide
`ragchat_setup`	`ragchat_test`	high	A task like 'verify the knowledge base works for domain X' or 'make sure the chatbot answers questions' could map to either initializing setup or sending a test message; agents frequently confuse configuring a domain with testing it.
`ragchat_serve`	`ragchat_status`	medium	Tasks like 'get the server running and check it' or 'is my chat domain live?' could be read as either starting the HTTP server or checking configuration status, especially without detailed tool descriptions in shortlists.
`ragchat_test`	`ragchat_serve`	medium	A task like 'check that my chat works' or 'see if the chatbot for domain X responds' is ambiguous between sending a test message and starting the serve endpoint, since both involve generating/verifying a chat response.
`ragchat_setup`	`ragchat_serve`	medium	Tasks like 'get domain X up for chat' or 'make the chatbot available' might be interpreted as initializing the knowledge base or starting the server, as both are prerequisite/serving steps in a natural pipeline.
`ragchat_serve`	`ragchat_widget`	medium	A task like 'add a chat bubble to my site' or 'expose the chat on the web' could plausibly select the widget generation tool or the server start tool, since both are needed and descriptions mention connecting to each other.
`ragchat_test`	`ragchat_status`	low	Sending a test message rarely gets confused with listing configured domains; the action vs. status distinction is usually clear from the tool descriptions.
`ragchat_setup`	`ragchat_widget`	low	Initializing a knowledge base and generating an embed script differ enough in intent that a natural task would rarely select the wrong one; shared 'ragchat' token alone is insufficient.
`ragchat_setup`	`ragchat_status`	low	Setting up a domain vs. listing domains is action vs. observation; ambiguity is not genuinely plausible despite the shared domain/ragchat tokens.
`ragchat_widget`	`ragchat_status`	low	Generating widget code and checking status are semantically distant; no plausible task would confuse the two.
`ragchat_test`	`ragchat_widget`	low	Testing chat responses and generating embed code are distinct actions; no plausible natural-language task merges them.

Compare the field

One score is useful.
The evidence makes it actionable.

Back to the leaderboard