RateMyKeyVault: Grading Every Azure Key Vault From A to F

After RateMyWAF went out, the question I kept getting was some version of “can you do that for Key Vault?” It makes sense. Key Vault is where the secrets, keys and certificates live, every workload depends on at least one, and the configuration that keeps those vaults safe is spread across half a dozen blades: the permission model, the firewall, soft delete and purge protection, diagnostic settings, and then the contents themselves, where it is remarkably common to find a production password with no expiry date and a certificate that lapsed in March.

Answering “are our key vaults in good shape?” for a tenant with a few dozen vaults means opening every one of them and checking all of that by hand. So I built RateMyKeyVault. It does for Key Vault exactly what RateMyWAF does for the firewalls: point it at a tenant, choose the vaults, and get a letter grade per vault with a ranked list of what is costing you points.

What it does

#

RateMyKeyVault is a read-only Blazor Server app on .NET 10, and it is deliberately a twin of RateMyWAF. Same colours, same layout, same three-step wizard, same A to F badges, same Word reports. If you have used one you already know the other. The flow is:

1. Scope - pick the tenant and the subscriptions. Only subscriptions that actually contain key vaults are highlighted, so you are not guessing.
2. Vaults - every vault found in those subscriptions, grouped by subscription, with its SKU, permission model and network posture shown as chips. Tick the ones you want rated; all of them are selected by default.
3. Grades - an A to F grade per vault, the score breakdown by category, the improvements ranked by points, findings with concrete fixes, a per-vault PowerShell fix script, and summary or detailed-findings reports as Word documents.

The second step is the one people asked for. With the WAF tool you rate everything in a subscription; with Key Vault there are often vaults you do not care about (someone’s dev experiment, a vault left behind by a deleted app) and vaults you care about a great deal, so you get to choose.

As with RateMyWAF there is a Try with sample data button on the home page. It runs the whole flow against a built-in estate of seven Contoso vaults that deliberately covers every grade from A to F, without a single call to Azure.

The scoring model

#

The model follows Microsoft’s own Key Vault best practices, weighted by how much damage each gap does: who can get in, from where, whether data can be lost, whether anyone would notice, and what state the contents are in. 100 points across six categories:

The letter bands are the same as RateMyWAF: A at 90 or above, B at 75, C at 60, D at 45, E at 30, F below that.

And again, the hard caps are the part that matters most, because a weighted total can hide something dangerous. A vault that accepts requests from any network is capped at C no matter how good the rest is, because at that point only credentials stand between the internet and the secrets. Soft delete off is also capped at C: a deleted vault or item is simply gone. Purge protection off caps at B, audit logging off caps at B (a vault nobody can see into is not fully trustworthy), and expired items that are still enabled cap at B, because an expired credential that still works is a credential nobody is rotating.

Every improvement is listed with the points it gains and the grade it would produce if you made just that change. “Switch to Azure RBAC and replace the six access policies with role assignments: +15, takes you to a B” is a lot more actionable than “you got a D”.

The hygiene check, and what it never reads

#

The category I think is most useful is the one that looks inside the vault. The app lists the secrets, keys and certificates through the Key Vault data plane and reads their metadata: name, enabled or disabled, created, updated and expiry dates. It never requests a secret value or key material, and it could not even if it wanted to, because the role it needs is Key Vault Reader, which is metadata-only by design.

From that it finds the three things that always turn up in a real estate: items with no expiry date at all, items that have expired but are still enabled, and items about to expire. It also knows that a certificate comes with a managed key and a managed secret behind it, and rates those once, via the certificate, rather than counting them three times.

Some vaults will not let the app in. A vault with firewall rules will reject the machine’s public IP, a vault that is private-endpoint-only will not resolve from outside the network, and a vault where the identity lacks Key Vault Reader will refuse the list call. Those vaults are still graded on everything else and get partial credit for hygiene, and the Grades page is explicit about it: a warning naming the vault and the error (for a firewall block, including the client IP that was refused), followed by a plain “what it means” and “how to fix” right next to the vault. The first time I ran it against a real tenant one of my own vaults came back with exactly that, which is how that part of the UI ended up as clear as it is.

It’s read-only, and that’s deliberate

#

Everything the grade is built from comes through three read paths:

• Azure Resource Graph for the vaults themselves: SKU, permission model, access policies, soft delete, purge protection, retention, platform access flags, public network access, firewall rules and private endpoint connections.
• ARM GET for diagnostic settings, to see whether the AuditEvent category is enabled and where it goes.
• Key Vault data plane GET on /secrets, /keys and /certificates for the item metadata described above.

Authentication defaults to AzureCliCredential, so on your own machine it is:

az login
cd src/RateMyKeyVault
dotnet run

and open http://localhost:5220. For hosting without the Azure CLI (App Service, a container with a managed identity) set Azure:Credential to Default and it switches to DefaultAzureCredential. The credential needs Reader on the subscriptions and Key Vault Reader on the vaults.

Nothing is ever written. The fix script you get for a vault is PowerShell using the Az.KeyVault and Az.Monitor modules, with one toggle per gap: enable purge protection, switch to RBAC (with the role assignments to make first listed as comments, because access policies stop applying the moment you flip it), set the firewall to deny, add the diagnostic setting, raise the retention, set expiry dates on the items without one, disable the expired ones. You read it, adjust it, and run it yourself. The app holds no write permissions.

How it’s built

#

The structure is lifted straight from RateMyWAF, which is the point: the same shape of problem deserves the same shape of solution.

• A pure rating engine. KeyVaultRatingService is a static, deterministic function from a vault’s facts to a rating. No Azure, no clock it does not control (the “now” is injected, so expiry tests are repeatable), no side effects. Caps are collected as keyed entries so each improvement can say which cap it lifts and what grade results.
• A discovery service that only parses. Resource Graph rows, ARM responses and data-plane lists go through small null-tolerant JSON helpers into plain model classes. Every parser is internal and covered by tests fed with the real JSON shapes, including the exact 403 body Key Vault returns when a firewall blocks you.
• An IAzureApi seam. The one interface that talks to Azure has three methods (Resource Graph query, ARM list, Key Vault list) so the whole scan can be exercised in tests against canned JSON.
• A flow service and scoped session state that the Blazor components talk to, so the UI never knows whether it is looking at Azure or the sample estate.
• Reports from one document model. The HTML preview and the Word download render the same ReportDoc, so what you see in the modal is what the customer gets.

Testing is where I spent the time, for the same reason as last time: a security grade that is wrong in the optimistic direction is worse than no grade. There are 77 xUnit and bUnit tests: every category and cap in the rating engine, the findings builder, the parsers, the data-plane error classification, a full scan against a fake Azure, fix scripts, both reports, the sample estate (which must keep covering A through F) and the components. On top of that a Playwright walkthrough drives the running app in headless Chromium through the whole flow on sample data, including a hard reload, the dark theme and a phone-width viewport, and fails on any browser error.

Caveats

#

It only rates Microsoft.KeyVault/vaults; Managed HSM is not in scope yet. It does not enumerate RBAC role assignments, so in RBAC mode the access category trusts that you review those with the rest of your Azure RBAC rather than grading each assignment. Hygiene depends on being able to reach the vault: locked-down vaults are graded on configuration with partial credit, which is the honest answer, not a full one. And as always the grade is only as complete as the permissions behind it.

Final thoughts

#

If you have ever had to answer “are our key vaults locked down?” and reached for a spreadsheet, this is the tool I wanted to have. Run az login and dotnet run, pick a subscription you think you know, and find out how many secrets have no expiry date. Or click Try with sample data first and walk the demo estate end to end; the vault called kv-contoso-locked will show you what a firewall-blocked vault looks like before you meet one of your own.