01Safety
30.0 / 30
What changed in the harness
Selection accuracy 96%, destructive-action safety rate 100% (baseline only -- no rewrite pass applied).
Category breakdown
Where the score comes from.
Earned points across the four signals Gradable measures. Safety and Legibility are scored out of 30; Economics and Discoverability are scored out of 20.
02Legibility
28.3 / 30
03Economics
19.0 / 20
04Discoverability
19.2 / 20
Highest-impact fix
Estimated gain +2 pointsDisambiguate overlapping tool choices
Give confusable tools explicit decision boundaries and examples of when to choose one instead of the other.
Description evidence
Defects and rewrites.
0 defects found across the exposed tool descriptions. Suggested rewrites make purpose, inputs, boundaries, and returns easier for an agent to understand.
| Tool | Defect types | Suggested rewrite |
|---|---|---|
| No description defects were flagged in this assessment. | ||
Selection evidence
Confusable tool pairs.
3 pairs where similar names or overlapping descriptions may send an agent toward the wrong tool.
| Tool A | Tool B | Confidence | Why they collide |
|---|---|---|---|
xcomet_evaluate |
xcomet_batch_evaluate |
high | Both assess translation quality with xCOMET and share the 'evaluate' verb and nearly identical args (only single-pair vs array-of-pairs differs). A natural task like 'evaluate my translations' or 'assess the quality of these translated files' is ambiguous: the agent could pick the single evaluator and loop, or pick the batch tool when the user actually meant one pair. The batch tool even returns per-pair scores, error counts, and summaries, overlapping xcomet_evaluate's output. |
xcomet_evaluate |
xcomet_detect_errors |
medium | Both take (source, translation) and return error spans with severity; xcomet_evaluate also reports scores. Tasks phrased as 'check this translation and point out any errors' or 'analyze the translation for mistakes and give a quality read' map reasonably to either tool, since the single evaluator already includes error detection in its output. |
xcomet_detect_errors |
xcomet_batch_evaluate |
medium | A task like 'find errors across all my translations' or 'error-check these segments' could make the agent pick batch_evaluate (which returns only error counts and a critical-errors flag, not detailed spans) when the user wanted per-segment error details, or pick detect_errors when the user actually wanted aggregate stats over many pairs. The pluralization mismatch between the tools creates real selection ambiguity. |
Compare the field