A customer asked me last month whether their backups were “alright”. Not whether they existed, they knew they existed, but whether they would actually help on the day something went badly wrong. I opened the first Recovery Services vault and found two VMs with protection stopped since June, a policy with seven days of retention, and locally redundant storage in a region with a paired region sitting right there. That was one vault. They had nineteen.
Backups are the thing everyone has and nobody checks. The portal will happily tell you a vault exists and that jobs are running, and it will tell you almost nothing about whether the setup as a whole would survive a bad day. So after RateMyWAF and RateMyKeyVault, this is the third one: RateMyAzureBackups. Point it at a tenant and get a letter grade per vault, with the one change that lifts the grade the most at the top of the list.
What it does#
It’s the same read-only Blazor Server app on .NET 10 as the other two, same layout, same three-step wizard, same badges and Word reports. The difference this time is that Azure Backup has two vault products that look alike in the portal and are completely different underneath, so the app has two flows and you pick one from the home page.
The Recovery Services vaults flow covers the classic product: Azure VMs, SQL Server and SAP HANA inside VMs, Azure Files, and MARS agent items. The Backup vaults flow covers the newer Data Protection product: blob storage, managed disks, PostgreSQL, MySQL and AKS backup instances. Both get the same scoring model, and both walk the same three steps:
- Scope - pick the tenant and subscriptions. Only subscriptions that actually contain that vault product are highlighted, which saves a lot of ticking boxes in a big tenant.
- Grades - A to F per vault, the score broken down by category, improvements ranked by points gained, findings with fixes, the policy inventory, and summary or detailed reports as Word documents.
- Jobs - roughly the last month of backup jobs for a vault, compared with the month before, with every failing item classified as a persistent failure, a transient one, or something to review.
The third step is new and I’ll come back to it, because it’s where I ended up spending most of the time.
There’s a Try with sample data button on the home page, same as the other two. It runs both flows against a built-in Contoso estate that covers every grade from A to F, so you can see what a vault with stopped protection looks like without owning one.
The scoring model#
The categories follow Microsoft’s Azure Backup best practices in the order I think the damage runs: are the workloads actually protected, are the recovery points recent, would the backups survive a regional outage, could someone (or ransomware) delete them, are the policies sensible, and would anyone notice a failure. 100 points across six categories:
| Category | Points | What earns it |
|---|---|---|
| Protection health | 30 | Every protected item actively protected, none stopped or in error (pro-rated) |
| Backup freshness | 20 | A successful recovery point within the expected window for every active item: 36 hours for daily and hourly policies, 8 days for weekly (pro-rated) |
| Redundancy & restore | 20 | Geo-redundant storage 12 (zone-redundant 8, locally redundant 0) - Cross Region Restore 8 |
| Vault security | 15 | Soft delete 5 - immutable vault locked 5 (unlocked 3) - multi-user authorization 3 - public network access disabled 2 |
| Policy hygiene | 10 | Daily or more frequent schedule 4 - at least 30 days retention 3 - weekly, monthly or yearly retention 3 |
| Monitoring & alerts | 5 | Built-in Azure Monitor alerts for all job failures 3 - diagnostics to Log Analytics 2 |
The bands are the same as before: A at 90 or above, B at 75, C at 60, D at 45, E at 30, F below that.
And again the caps are the part that does the real work, because a weighted total can hide something that should be a red flag. A vault that protects nothing is an F, obviously. Any item with protection stopped or in error caps the vault at D, because a workload with no new recovery points is the backup equivalent of an unprotected endpoint, however good the rest of the vault looks. Any item whose last backup failed caps at C. Soft delete off caps at C, since without it a deleted backup is just gone. And locally redundant storage caps at B, because a regional outage takes the backups down with the workload.
The 36 hour window for daily policies is deliberate. A strict 24 hours flags every vault whose jobs run long or started a bit late, and I’d rather the tool was quiet about jitter and loud about a VM that hasn’t had a recovery point since Tuesday.
Settings the scan can’t read, usually because a vault’s Resource Graph shape predates them or the identity is missing a permission, get half points and never trigger a cap. Guessing in the pessimistic direction is annoying; guessing in the optimistic direction is dangerous, and half points with a note saying “couldn’t read this” felt like the honest middle.
Every improvement comes with the points it gains and the grade it would produce on its own. “Switch rsv-contoso-legacy to geo-redundant storage: +12, lifts the B cap, takes you to an A” is the sentence I wanted to be able to put in front of a customer.
The VMs nobody is backing up#
A vault can score an A and still be hiding the worst problem in the estate, which is the virtual machine that isn’t in any vault at all. So the Recovery Services flow also lists every VM in the scanned subscriptions and calls out the ones no vault protects. That list has been the most useful output on every real tenant I’ve tried it on. It’s always the jump box. It’s always a domain controller someone built by hand in 2022.
Jobs, and what to do about them#
Grades tell you how a vault is configured. The Jobs step tells you whether it’s working. It pulls the vault’s jobs from Resource Graph for a window you choose (Resource Graph keeps about a month, so that’s the ceiling), and the same window before it, and then looks at each item that failed.
An item that has failed on every attempt in the window is a persistent failure. One that failed and then succeeded is transient. Anything else, a single failure or a warning, is marked for review. It also shows what changed against the previous period, so you can see that three items started failing this month that were fine last month, which is usually more useful than a raw count.
For the common error codes it knows what the fix is. UserErrorGuestAgentStatusUnavailable means the VM agent isn’t responding. UserErrorRequestDisallowedByPolicy means an Azure Policy is blocking the snapshot resource group, and you need an exemption for the AzureBackupRG_* groups. UserErrorVmNotFoundV2 means the VM was deleted and someone forgot to stop protection, so the vault has been dutifully failing a job every night since. There are around two dozen of these built in, each with a one-line explanation and what to do.
And for each failing item you can generate a PowerShell script using Az.RecoveryServices or Az.DataProtection. For a persistent failure it runs an on-demand backup once you’ve fixed the cause and then lists the recent jobs so you can confirm the item has a current recovery point again. If the error says the source is gone, it offers to stop protection with the data retained instead. For a transient failure it lists recovery points and recent jobs so you can check it really did recover. You tick the options you want, read the script, and run it yourself. The app never writes anything.
Read-only, and what it needs#
Everything comes through Azure Resource Graph and ARM GET calls. The vaults themselves, with redundancy, Cross Region Restore, soft delete, immutability, multi-user authorization, public network access and the built-in alert settings. The protected items and backup instances from the RecoveryServicesResources table, with protection state, last backup status and policy. The policies, with schedule, retention and SQL log cadence. The jobs. The diagnostic settings on every vault. And every VM in scope, for the unprotected list.
A couple of settings aren’t in the Resource Graph projection for older vaults, so there are ARM fallbacks to backupstorageconfig and backupconfig for those. That took me three goes to get right, because the vault I was testing against was new enough to have everything in Resource Graph and the customer’s weren’t.
The credential needs Reader on the subscriptions. You don’t need Backup Reader. On your own machine it’s:
az login
cd src/RateMyAzureBackups
dotnet runand open http://localhost:5220. For hosting without the Azure CLI (App Service with a managed identity, say) set Azure:Credential to Default and it uses DefaultAzureCredential instead.
One warning that’s in the README in bold and belongs here too: the app has no login of its own. Every visitor uses the credential the app was started with. On a workstation that’s fine. On App Service with a managed identity and nothing in front of it, anyone who can reach the URL can read every subscription that identity can. Put Easy Auth or a reverse proxy in front of it, or don’t host it.
How it’s built#
The shape is lifted from the other two tools, because the problem has the same shape. A pure, static rating engine that takes a vault’s facts and a clock and returns a rating, with no Azure and no side effects, so every cap and improvement path has a test. A discovery service that only parses, through small null-tolerant JSON helpers, fed in tests with JSON captured from real Resource Graph output. One IAzureApi interface so the whole scan runs against canned responses. One ReportDoc model rendered to both the HTML preview and the Word download.
There are 91 xUnit and bUnit tests and they all pass. The perfect vault is the reference case and has to keep producing 100 and an A. On top of that a Playwright walkthrough drives the real app in headless Chromium through both flows on sample data, including the fix script modal, a hard reload, the dark theme and a phone-width viewport, and fails on any browser error.
Caveats#
The freshness check for Backup vault instances is derived from recent jobs rather than a last-backup field, so an instance with no jobs in the window gets half points rather than a verdict. The security category awards points for multi-user authorization being on, but doesn’t check that the Resource Guard is owned by a different person, which is the whole point of MUA; that’s a manual check for now. The unprotected VM list only knows about Recovery Services vaults, so a VM protected by a third-party product will show up as unprotected. And as always the grade is only as good as the permissions behind it: a subscription the identity can’t read is a subscription the tool doesn’t know about.
I’m also not fully settled on the weights. Thirty points for protection health feels right. Five for monitoring might be low given how often the problem on a real tenant turns out to be “it’s been failing for a month and nobody got an email”. If you run it and disagree, I’d like to hear why.
Final thoughts#
Run it against the subscription you’re most confident about, not the one you’re worried about. Click Try with sample data first if you want to see the shape of it, then az login, dotnet run, pick the Recovery Services flow and scroll straight to the unprotected VM list at the bottom of the Grades page. If that list is empty, go and look at the Jobs step. If both are clean, you’re in better shape than most of the tenants I’ve seen this year, and you can go and argue with me about the weights.
