Public assessment

jasonwilbur/cloud-cost-mcp (cloud-cost-mcp)

cloud-cost-mcp · v1.2.2 · scanned Aug 2, 2026

What changed in the harness

Selection accuracy 100→100, token cost up 10%, unconfirmed writes 0%→0%.

Category breakdown

Where the score comes from.

Earned points across the four signals Gradable measures. Safety and Legibility are scored out of 30; Economics and Discoverability are scored out of 20.

01Safety

0.0 / 30

0.0 out of 30

02Legibility

19.6 / 30

19.6 out of 30

03Economics

18.3 / 20

18.3 out of 20

04Discoverability

12.6 / 20

12.6 out of 20

Highest-impact fix

Estimated gain +30 points

Add explicit identity and permission preflight tools

Expose machine-readable principal/tenant confirmation and a non-mutating permission check so agents can verify both before destructive actions.

Description evidence

Defects and rewrites.

26 defects found across the exposed tool descriptions. Suggested rewrites make purpose, inputs, boundaries, and returns easier for an agent to understand.

Tool	Defect types	Suggested rewrite
`compare_compute`	no_return_description	Compare VM/instance pricing across AWS, Azure, GCP, and OCI. Finds instances matching your vCPU and memory requirements. Returns a list of matching instances per provider with specs and pricing, sorted for comparison.
`compare_storage`	name_restates_behavior no_return_description	Compare object and block storage pricing across all clouds for a given size, tier, and type. Returns per-provider monthly cost estimates so you can identify the cheapest option.
`compare_egress`	no_return_description	Compare data transfer/egress costs. OCI offers 10TB/month free (100x more than others)! Returns per-provider monthly egress cost estimates for the given data volume.
`compare_kubernetes`	name_restates_behavior no_return_description	Compare managed Kubernetes pricing (EKS, AKS, GKE, OKE) for a given node count and per-node vCPU/memory. Returns control plane and worker node cost breakdowns per provider.
`find_cheapest_compute`	name_restates_behavior no_return_description	Search all providers for instances matching the given vCPU, memory, and optional category, and return the single lowest-cost matching instance along with its provider and price.
`calculate_workload_cost`	params_unexplained no_return_description	Estimate total monthly cost for a workload across all cloud providers. 'compute' specifies vCPUs, memory, and instance count; 'storage' specifies object and/or block storage in GB; 'egress' specifies monthly outbound data in GB; 'kubernetes' specifies node count and per-node vCPU/memory for a managed cluster. Returns a per-provider total monthly cost breakdown combining all specified components.
`quick_estimate`	no_return_description	Get instant cost comparison for common deployment presets including GPU workloads. Returns per-provider monthly cost estimates for the selected preset configuration.
`list_presets`	name_restates_behavior no_return_description	Returns the full list of deployment preset names and descriptions accepted by quick_estimate's 'preset' parameter.
`estimate_migration_savings`	params_unexplained no_return_description	Calculate potential savings when migrating from one cloud provider to another. 'compute' specifies vCPUs, memory, and instance count; 'storage' specifies object and/or block storage in GB; 'egress' specifies monthly outbound data in GB, all used to price the workload on both providers. Returns the cost on the current provider, the cost on the target (or cheapest) provider, and the estimated savings.
`get_data_freshness`	no_return_description	Check how recent the pricing data is for each provider. Warns if data is stale (>30 days). Returns a per-provider timestamp of the last data refresh and a staleness flag.
`get_provider_details`	no_return_description	Get detailed pricing data for a specific cloud provider, optionally filtered by category. Returns the raw pricing entries (SKUs, specs, and prices) matching the request.
`get_storage_summary`	no_return_description	Get storage pricing summary by tier (hot, cool, cold, archive) for all providers at the given size. Returns a per-provider, per-tier monthly cost breakdown.
`refresh_azure_pricing`	no_return_description	Fetch latest Azure VM pricing from the public Azure Retail Prices API. No authentication required. Returns the refreshed list of Azure VM SKUs with prices for the given region and filters.
`list_oci_categories`	name_restates_behavior no_return_description	Query OCI's real-time pricing API for the full set of service categories it exposes. Returns the list of category names usable as filters in refresh_oci_pricing.
`check_api_status`	no_return_description	Check if the real-time pricing APIs (Azure, OCI, AWS) are accessible. Returns a per-provider status indicating reachability and any error encountered.
`get_aws_lightsail_pricing`	name_restates_behavior no_return_description	Retrieve AWS Lightsail bundle pricing for the given region — simplified VPS plans with fixed monthly pricing including storage and transfer. Returns the list of Lightsail bundles with their specs and monthly price.
`list_aws_regions`	name_restates_behavior no_return_description	Query the pricing dataset for all AWS regions with available data. Returns the list of AWS region codes usable in other AWS pricing tool calls.
`list_aws_instance_families`	name_restates_behavior no_return_description	List AWS EC2 instance families with counts (e.g., General purpose: 200, Compute optimized: 150). Returns the family names and instance counts for use as a 'family' filter in refresh_aws_ec2_pricing.
`list_gcp_regions`	name_restates_behavior no_return_description	Query the pricing dataset for all GCP regions with available data. Returns the list of GCP region codes and display names usable in other GCP pricing tool calls.
`list_gcp_instance_families`	name_restates_behavior no_return_description	List GCP instance families with counts. Returns family names and instance counts for use as a 'family' filter in refresh_gcp_pricing.
`list_azure_regions`	name_restates_behavior no_return_description	Query the pricing dataset for all Azure regions with available data. Returns the list of Azure region codes usable in other Azure pricing tool calls.
`list_azure_categories`	name_restates_behavior no_return_description	List Azure VM categories with counts (e.g., generalpurpose: 400, computeoptimized: 200). Returns category names and counts for use as a 'category' filter in refresh_azure_full_pricing.
`list_gpu_shapes`	no_return_description	List OCI GPU shapes with pricing. Filter by GPU model (A10, A100, H100, H200, L40S, MI300X) or use case (inference, training). Returns matching GPU shapes with specs and hourly pricing.
`get_gpu_shape_details`	name_restates_behavior no_return_description	Look up a specific OCI GPU shape family (e.g. 'BM.GPU.H100.8') and return its full specifications and pricing details.
`compare_gpu_shapes`	name_restates_behavior no_return_description	Compare multiple named OCI GPU shapes side-by-side on specs, pricing, and price-per-GPU metrics. Returns a comparison table across the requested shapes.
`recommend_gpu_shape`	no_return_description	Get a GPU shape recommendation based on workload type, minimum GPU memory, and budget constraint. Returns the recommended OCI GPU shape(s) with rationale and pricing.

Selection evidence

Confusable tool pairs.

10 pairs where similar names or overlapping descriptions may send an agent toward the wrong tool.

Tool A	Tool B	Confidence	Why they collide
`refresh_azure_pricing`	`refresh_azure_full_pricing`	high	Both fetch Azure VM pricing and a request like 'refresh Azure pricing data' doesn't indicate which source/breadth is wanted; only the description reveals one is the Retail Prices API (100 results) vs the comprehensive vantage.sh dataset (1199 types, spot/Windows).
`refresh_aws_ec2_pricing`	`refresh_aws_rds_pricing`	medium	A generic 'refresh AWS pricing' request doesn't specify compute vs database, and both tools share nearly identical parameter shapes (region, maxResults), risking the agent picking EC2 by default even when RDS/database pricing was intended.
`compare_gpu_shapes`	`recommend_gpu_shape`	medium	A prompt like 'what GPU shape should I use for training within budget' overlaps both: recommend_gpu_shape is designed for exactly this, but an agent might instead try to compare a guessed shortlist via compare_gpu_shapes since it also handles GPU shape selection.
`list_gpu_shapes`	`recommend_gpu_shape`	medium	list_gpu_shapes supports useCase filtering (inference/training) which functionally overlaps recommend_gpu_shape's workloadType-based recommendation, so 'find me a GPU shape for inference' could plausibly invoke either.
`list_gpu_shapes`	`get_gpu_shape_details`	medium	Filtering list_gpu_shapes by gpuModel (e.g., 'H100') returns similar info to requesting details on a specific shape family, so 'give me details on H100 options' is ambiguous between the two.
`compare_compute`	`compare_kubernetes`	medium	A request to compare costs for Kubernetes worker nodes with specific vCPU/memory could be routed to compare_compute (generic instance comparison) instead of compare_kubernetes, which is purpose-built but requires framing around cluster/node structure.
`compare_compute`	`compare_gpu_shapes`	high	compare_compute explicitly supports a 'gpu' category filter for cross-cloud GPU instance comparison, directly overlapping compare_gpu_shapes' OCI-specific shape comparison — 'compare GPU instance pricing across clouds' plausibly maps to either.
`compare_storage`	`get_storage_summary`	high	Both return cross-provider storage pricing organized by tier for a given size; 'show me storage costs by tier for 500GB across providers' has no clear signal favoring one tool's name over the other's near-identical inputSchema and purpose.
`quick_estimate`	`estimate_migration_savings`	low	Both produce cost estimates, but quick_estimate is preset-driven while estimate_migration_savings requires explicit current/target providers, making confusion only plausible for vague prompts like 'estimate savings for my ML training setup' without a migration framing.
`get_provider_details`	`get_storage_summary`	low	get_provider_details with category='storage' for a single provider overlaps get_storage_summary's all-provider tier breakdown, so a prompt like 'get storage pricing details' without specifying scope could go either way.

Compare the field

One score is useful.
The evidence makes it actionable.

Back to the leaderboard