Public assessment

PantelisGeorgiadis/dicomweb-mcp-server (dicomweb-mcp-server)

dicomweb-mcp-server · v0.0.4 · scanned Aug 2, 2026

What changed in the harness

Selection accuracy 100→98, token cost up 3%, unconfirmed writes 100%→100%.

Category breakdown

Where the score comes from.

Earned points across the four signals Gradable measures. Safety and Legibility are scored out of 30; Economics and Discoverability are scored out of 20.

01Safety

30.0 / 30

30.0 out of 30

02Legibility

26.2 / 30

26.2 out of 30

03Economics

19.3 / 20

19.3 out of 20

04Discoverability

12.7 / 20

12.7 out of 20

Highest-impact fix

Estimated gain +7 points

Make target tools discoverable on the first call

Clarify tool names, decision boundaries, and required argument schemas so an agent can choose and construct the target call without exploratory steps.

Description evidence

Defects and rewrites.

6 defects found across the exposed tool descriptions. Suggested rewrites make purpose, inputs, boundaries, and returns easier for an agent to understand.

Tool	Defect types	Suggested rewrite
`find-instances`	no_return_description	Searches DICOM instances within a single series, sorted by Instance Number ascending. Requires Study and Series Instance UIDs from find-studies and find-series. Returns matching instance metadata (e.g. SOP Instance UID, SOP Class UID, instance number) as JSON, not image data.
`find-structured-reports`	no_return_description	Finds all Structured Report (SR) DICOM instances in a study by searching for SR-modality series and filtering by SR SOP Class UIDs. Requires a Study Instance UID from find-studies. Returns a list of matching SR instances with their Series and SOP Instance UIDs for use with get-structured-report-text.
`find-encapsulated-pdf-reports`	no_return_description	Finds all Encapsulated PDF DICOM instances in a study by searching for DOC-modality series and filtering by EP SOP Class UIDs. Requires a Study Instance UID from find-studies. Returns a list of matching PDF instances with their Series and SOP Instance UIDs for use with get-encapsulated-pdf-report-text.
`get-structured-report-text`	no_return_description	Retrieves and converts a Structured Report (SR) instance to human-readable text. Requires Study, Series, and SOP Instance UIDs from find-structured-reports. Does not retrieve image data. Returns the SR content as plain text.
`get-encapsulated-pdf-report-text`	no_return_description	Retrieves and converts an Encapsulated PDF instance to human-readable text. Requires Study, Series, and SOP Instance UIDs from find-encapsulated-pdf-reports. Does not retrieve image data. Returns the extracted PDF text as plain text.
`get-instance-metadata`	no_return_description	Retrieves and converts a DICOM instance's metadata to human-readable text. Requires Study, Series, and SOP Instance UIDs from find-instances. Does not retrieve image data. Returns the instance's DICOM tag metadata as formatted text.

Selection evidence

Confusable tool pairs.

15 pairs where similar names or overlapping descriptions may send an agent toward the wrong tool.

Tool A	Tool B	Confidence	Why they collide
`get-structured-report-text`	`get-encapsulated-pdf-report-text`	medium	Both convert a specific report instance to human-readable text with identical parameter shapes (study/series/sopInstanceUid); an agent asked to 'get the report text' without knowing whether the source study contains an SR or an Encapsulated PDF could pick the wrong one, though the find-* precursor tools disambiguate if used first.
`find-encapsulated-pdf-reports`	`get-encapsulated-pdf-report-text`	low	Names share most tokens (find vs get-...-text) but the actions are clearly distinct (list instances vs retrieve text of one instance) and the input schemas differ sharply (studyInstanceUid only vs three UIDs), making misselection unlikely for a careful agent.
`find-structured-reports`	`find-encapsulated-pdf-reports`	medium	Both find all report-type instances in a study using an identical schema (studyInstanceUid only); a vague instruction like 'find the reports in this study' without specifying SR vs PDF could cause the agent to pick either or the wrong one.
`find-structured-reports`	`get-structured-report-text`	low	Clearly sequential (find then get) rather than alternative choices; an agent following the natural task flow would use both, not confuse one for the other.
`find-series`	`find-instances`	low	Distinct hierarchy levels (series vs instances) with different required parameters (extra seriesInstanceUid for find-instances), so a task naming 'series' or 'instances' explicitly leaves little ambiguity.
`get-encapsulated-pdf-report-text`	`get-instance-metadata`	medium	Both retrieve and convert an instance to human-readable text with identical three-UID schemas; a generic request like 'get the text/details of this instance' could plausibly select the metadata tool instead of the PDF-specific one, or vice versa, especially since get-instance-metadata's description doesn't clarify it returns metadata rather than document content.
`find-studies`	`find-instances`	low	Different hierarchy levels with clearly different required parameters (find-instances needs study+series UIDs); unlikely to be confused given the DICOM hierarchy is explicit in descriptions.
`find-studies`	`find-series`	low	find-series explicitly requires a studyInstanceUid obtained from find-studies, making the workflow order clear rather than ambiguous.
`get-structured-report-text`	`get-instance-metadata`	medium	Both convert an instance to human-readable text via identical three-UID schemas; an agent asked to 'read the contents of this instance' without knowing it's an SR could invoke the generic metadata tool instead of the SR-specific text extractor, or vice versa.
`find-instances`	`find-structured-reports`	low	find-structured-reports only needs studyInstanceUid while find-instances needs series too and returns all instance types, not just SRs; a task specifically about reports would clearly favor find-structured-reports.
`find-instances`	`get-instance-metadata`	low	One searches/lists instances, the other retrieves a single instance's metadata; required parameters differ enough (query vs no query, but get-instance-metadata needs specific sopInstanceUid) that confusion is unlikely.
`find-series`	`find-structured-reports`	low	find-series is a generic series search requiring a query, while find-structured-reports is a specialized no-query SR finder; a task about finding reports would clearly point to the latter.
`find-series`	`find-encapsulated-pdf-reports`	low	Distinct purposes (generic series search vs specialized PDF report finder) with different schemas; low chance of genuine confusion.
`find-studies`	`find-encapsulated-pdf-reports`	low	Different hierarchy scope (all studies vs PDF reports within one study) and different required parameters make confusion unlikely.
`get-instance-metadata`	`render-instance-frame`	low	Clearly different outputs (text metadata vs rendered image) despite shared required UID parameters; a task asking to 'view' or 'see' the image would unambiguously favor render-instance-frame.

Compare the field

One score is useful.
The evidence makes it actionable.

Back to the leaderboard