Public assessment

jasonwilbur/cloud-cost-mcp (cloud-cost-mcp)

cloud-cost-mcp · v1.2.2 · scanned Aug 3, 2026

What changed in the harness

Selection accuracy 100→100, token cost up 24%, unconfirmed writes 0%→0%.

Category breakdown

Where the score comes from.

Earned points across the four signals Gradable measures. Safety and Legibility are scored out of 30; Economics and Discoverability are scored out of 20.

01Safety

0.0 / 30

0.0 out of 30

02Legibility

20.6 / 30

20.6 out of 30

03Economics

18.4 / 20

18.4 out of 20

04Discoverability

16.1 / 20

16.1 out of 20

Highest-impact fix

Estimated gain +30 points

Add explicit identity and permission preflight tools

Expose machine-readable principal/tenant confirmation and a non-mutating permission check so agents can verify both before destructive actions.

Description evidence

Defects and rewrites.

26 defects found across the exposed tool descriptions. Suggested rewrites make purpose, inputs, boundaries, and returns easier for an agent to understand.

Tool	Defect types	Suggested rewrite
`compare_compute`	no_return_description	Compare VM/instance pricing across AWS, Azure, GCP, and OCI for instances matching your desired vCPUs and memory in GB, optionally filtered by category (general, compute, memory, storage, gpu, arm). Returns the matching instance types and their prices per provider so you can identify the cheapest option for your workload.
`compare_storage`	name_restates_behavior no_return_description	Compare object and block storage pricing across AWS, Azure, GCP, and OCI. Given a storage size in GB (sizeGB), optionally the tier (hot, cool, cold, archive) and storage type (object or block), returns the estimated monthly cost per provider so you can see which cloud is cheapest for your storage needs.
`compare_egress`	no_return_description	Compare monthly data transfer/egress costs across AWS, Azure, GCP, and OCI given your monthly outbound data in GB (monthlyGB). Returns the estimated monthly egress cost per provider, factoring in free-tier allowances such as OCI's 10TB/month free (100x more than the others).
`compare_kubernetes`	name_restates_behavior no_return_description	Compare managed Kubernetes pricing (EKS, AKS, GKE, OKE) for a cluster of nodeCount worker nodes, each with nodeVcpus vCPUs and nodeMemoryGB memory. Returns the control plane cost, worker node cost, and total monthly price for each provider.
`find_cheapest_compute`	name_restates_behavior no_return_description	Find the cheapest cloud provider for a compute workload given your desired vCPUs (vcpus) and memory in GB (memoryGB), optionally filtered by instance category (general, compute, memory, storage, gpu, arm). Returns the cheapest matching instance options per provider and highlights the lowest-cost provider overall.
`calculate_workload_cost`	params_unexplained no_return_description	Estimate the total monthly cost of a workload on AWS, Azure, GCP, and OCI, including compute, storage, egress, and managed Kubernetes. Describe the workload: compute with vcpus (vCPUs per instance), memoryGB (memory in GB per instance), and count (number of instances, default 1); storage with objectGB (object storage in GB) and blockGB (block storage in GB); egress with monthlyGB (outbound data in GB per month); kubernetes with nodeCount (worker nodes), nodeVcpus (vCPUs per node), and nodeMemoryGB (memory per node in GB). Returns the estimated monthly cost for each provider so you can compare and pick the cheapest.
`quick_estimate`	no_return_description	Get an instant cost comparison for a common deployment preset without specifying resources yourself. Given a preset name (see list_presets for options, which include GPU workloads), returns the estimated monthly cost of that preset for each cloud provider so you can quickly compare providers.
`list_presets`	name_restates_behavior no_return_description	Returns the catalog of deployment presets accepted by quick_estimate, including each preset's name and a description of the deployment it models (e.g., small-web-app, ml-training), so you can choose the closest match for your use case.
`estimate_migration_savings`	params_unexplained no_return_description	Estimate the monthly savings of migrating a workload from your current cloud provider to another. Set currentProvider (aws, azure, gcp, oci); targetProvider is optional and defaults to the cheapest option. Describe your current usage: compute with vcpus (number of vCPUs), memoryGB (memory in GB), and count (number of instances); storage with objectGB (object storage in GB) and blockGB (block storage in GB); egress with monthlyGB (outbound data in GB per month). Returns the estimated monthly cost for the relevant providers and the projected monthly and annual savings of switching.
`get_data_freshness`	no_return_description	Check how recent the cached pricing data is for each provider (AWS, Azure, GCP, OCI). Returns the date each provider's data was last refreshed and flags any provider whose data is stale (older than 30 days), warning when a data refresh is recommended.
`get_provider_details`	no_return_description	Return detailed pricing data for a specific cloud provider (aws, azure, gcp, or oci), optionally filtered to a category (compute, storage, egress, kubernetes, database). Returns the provider's instance types, storage tiers, and other offerings with their prices for use in cost comparisons.
`get_storage_summary`	no_return_description	Get a storage pricing summary for all providers broken down by tier (hot, cool, cold, archive). Given a storage size in GB (sizeGB), returns the estimated monthly cost per provider and per tier, showing how pricing varies with tier across clouds.
`refresh_azure_pricing`	no_return_description	Fetch the latest Azure VM pricing from the public Azure Retail Prices API (no authentication required). Optionally set region (default: eastus), filter by vmSeries (e.g., "D", "E", "F"), and limit output with maxResults (default: 100). Returns the fetched price entries and updates the cached Azure pricing used by the compare tools.
`list_oci_categories`	name_restates_behavior no_return_description	Returns the full list of service categories available from OCI's real-time pricing API, so you can see what OCI pricing data is queryable and pass a valid category value to refresh_oci_pricing.
`check_api_status`	no_return_description	Check whether the real-time pricing APIs for Azure, OCI, and AWS are currently accessible. Returns the availability status of each API, including any errors, so you can tell whether a problem is caused by a provider outage or something local.
`get_aws_lightsail_pricing`	name_restates_behavior no_return_description	Get AWS Lightsail bundle pricing for a region (default: us-east-1). Lightsail offers simplified VPS plans with fixed monthly pricing that includes storage and transfer. Returns the available bundles with their monthly price, vCPUs, memory, storage, and included transfer for the requested region.
`list_aws_regions`	name_restates_behavior no_return_description	Returns the list of AWS regions available in the pricing data, so you know which region value to pass to refresh_aws_ec2_pricing and refresh_aws_rds_pricing.
`list_aws_instance_families`	name_restates_behavior no_return_description	Returns the AWS EC2 instance families available in the pricing data with the number of instance types in each (e.g., General purpose: 200, Compute optimized: 150), so you can pick a valid family filter for refresh_aws_ec2_pricing.
`list_gcp_regions`	name_restates_behavior no_return_description	Returns the GCP regions available in the pricing data along with their display names, so you can pick a valid region value to pass to refresh_gcp_pricing.
`list_gcp_instance_families`	name_restates_behavior no_return_description	Returns the GCP instance families available in the pricing data with the number of instance types in each, so you can pick a valid family filter for refresh_gcp_pricing.
`list_azure_regions`	name_restates_behavior no_return_description	Return the set of Azure regions that have pricing data available, giving each region's identifier so callers can confirm coverage and scope region-based pricing lookups.
`list_azure_categories`	name_restates_behavior no_return_description	Report the coverage of Azure VM family categories in the pricing dataset by returning each category name with a count of the instance types it contains (e.g., generalpurpose: 400, computeoptimized: 200), useful for choosing a category before browsing specific instances.
`list_gpu_shapes`	no_return_description	Return OCI GPU shapes together with their pricing, optionally filtered by GPU model (e.g., A100, H100), use case (inference, training, graphics, general), or a maximum hourly price, so callers can survey available accelerators and their hourly cost before narrowing to a specific shape.
`get_gpu_shape_details`	name_restates_behavior no_return_description	Return the full specifications and pricing for the single OCI GPU shape identified by its shape family (e.g., "BM.GPU.H100.8"), so callers can evaluate capacity and cost before committing to an instance.
`compare_gpu_shapes`	name_restates_behavior no_return_description	Return a side-by-side comparison of the given OCI GPU shape families on specifications, pricing, and price-per-GPU metrics, so callers can weigh options directly against each other before selecting a shape.
`recommend_gpu_shape`	no_return_description	Return a recommended OCI GPU shape family matched to the workload type and budget, along with the chosen shape's specifications and expected hourly price, so users can pick a cost-appropriate accelerator without manually comparing shapes.

Selection evidence

Confusable tool pairs.

8 pairs where similar names or overlapping descriptions may send an agent toward the wrong tool.

Tool A	Tool B	Confidence	Why they collide
`refresh_azure_pricing`	`refresh_azure_full_pricing`	high	Both tools fetch latest Azure VM pricing with overlapping inputs (region, maxResults); a task like 'get the latest Azure VM pricing' matches both descriptions equally, and nothing in the request tells the agent to prefer the Retail Prices API over vantage.sh.
`refresh_aws_ec2_pricing`	`refresh_aws_rds_pricing`	medium	Both are described as 'Fetch real-time AWS ... pricing' with no auth; a generic request like 'refresh the AWS pricing data' is ambiguous between EC2 compute and RDS database unless the user explicitly mentions databases.
`compare_gpu_shapes`	`recommend_gpu_shape`	medium	A task like 'which OCI GPU is best for my training workload' fits recommend_gpu_shape (workload+budget) but also plausibly compare_gpu_shapes; conversely 'compare GPU options' without a prespecified shape list maps poorly to compare_gpu_shapes (requires a shapes array) and could fall to recommend.
`list_gpu_shapes`	`compare_gpu_shapes`	medium	Both concern 'OCI GPU shapes with pricing' and list_gpu_shapes supports filters (GPU model, use case, max price) while compare_gpu_shapes needs explicit shape-family names a user may not know; a request to 'compare/vet GPU shape prices' could select either depending on phrasing.
`get_provider_details`	`get_gpu_shape_details`	medium	'Get detailed OCI GPU pricing' matches get_provider_details (provider=oci) and get_gpu_shape_details (shapeFamily) nearly equally; the provider-details category enum has no 'gpu' option, so a GPU-focused request risks landing on the wrong tool.
`list_gpu_shapes`	`get_gpu_shape_details`	low	A task phrased as 'show me the details/specs of available OCI GPU shapes' could map to either, since get_gpu_shape_details requires a single shapeFamily the user usually won't have, making list_gpu_shapes the plausible but possibly wrong pick.
`get_data_freshness`	`get_provider_details`	low	Both reference 'pricing data' with a get verb; 'check the current pricing data for Azure' risks get_data_freshness (which only reports recency) instead of get_provider_details, though each tool's intent is usually disambiguated by wording like 'how recent' vs 'show data'.
`compare_compute`	`compare_storage`	low	A vague request like 'compare the cost of my workload' with no explicit subject could select either since both are generic cross-cloud compare tools; specification of vCPUs/memory vs GB clearly disambiguates, so confusion is only plausible for under-specified tasks.

Compare the field

One score is useful.
The evidence makes it actionable.

Back to the leaderboard