Disaster Recovery

This document provides disaster recovery procedures for the App Store for Intune, including backup strategies, recovery runbooks, and high availability options.

Last updated: May 2026

---

Table of contents

1. Recovery Objectives
2. Built-in Protection (Tier 2)
3. Recovery Procedures
4. Identity and Grant Reconstruction
5. High Availability Setup (Tier 3)
6. Backup Verification
7. Emergency Contacts

---

Recovery objectives

Tier 2 (default deployment)

Metric	Target
RTO (Recovery Time Objective)	1-2 hours
RPO (Recovery Point Objective)	5 minutes (SQL), 0 minutes (Storage with GRS)
Data Loss Risk	Minimal, geo-redundant backups

Tier 3 (high availability)

Metric	Target
RTO (Recovery Time Objective)	15-30 minutes
RPO (Recovery Point Objective)	Near-zero with active geo-replication
Data Loss Risk	Near-zero

---

Built-in protection (Tier 2)

The default ARM template deployment includes these disaster recovery features:

SQL Database backups

Feature	Configuration
Automated Backups	Enabled by default
Point-in-Time Restore	7 days (Basic tier)
Backup Storage	Geo-redundant (cross-region)
Full Backups	Weekly
Differential Backups	Every 12-24 hours
Transaction Log Backups	Every 5-10 minutes

What’s protected:

• All application data (apps, requests, settings)
• User configurations and branding
• Audit logs and history

Storage account (GRS)

Feature	Configuration
Redundancy	Geo-Redundant Storage (GRS)
Replication	6 copies (3 primary + 3 secondary region)
Failover	Automatic with RA-GRS or manual

What’s protected:

• WinGet packaging queue messages
• Package blobs (temporary)

Key Vault

Feature	Configuration
Soft Delete	Enabled (7-day retention)
Purge Protection	Optional (prevents permanent deletion)

What’s protected:

• SQL connection string
• Storage connection string

There is no Entra ID client secret to protect. The App Service authenticates to Microsoft Graph as its own managed identity, so the only secrets in the vault are the two connection strings.

Application code

Feature	Protection
Source Code	GitHub repository
Deployment Package	GitHub Releases (immutable)
ARM Template	GitHub repository

Recovery: Redeploy from ARM template. No code backup needed.

---

Recovery procedures

Scenario 1: Accidental data deletion

Symptoms: User accidentally deleted requests, apps, or settings.

Recovery steps:

1. Identify the deletion time from audit logs or user report

2. Restore SQL Database to point-in-time:

   # Azure CLI
   az sql db restore \
     --resource-group <resource-group> \
     --server <sql-server-name> \
     --name AppRequestPortal \
     --dest-name AppRequestPortal-Restored \
     --time "2026-02-14T10:00:00Z"

3. Verify restored data by connecting to the restored database

4. Option A - Full restore: Swap the restored database

   # Rename original (backup)
   az sql db rename --resource-group <rg> --server <server> \
     --name AppRequestPortal --new-name AppRequestPortal-Old
   
   # Rename restored to production
   az sql db rename --resource-group <rg> --server <server> \
     --name AppRequestPortal-Restored --new-name AppRequestPortal

5. Option B - Selective restore: Copy specific records from restored DB to production

6. Restart App Service to reconnect:

   az webapp restart --resource-group <rg> --name <app-name>

Estimated recovery time: 30-60 minutes

---

Scenario 2: Key Vault secret deleted

Symptoms: Application fails to start, “Key Vault reference failed” errors.

Recovery steps:

1. Check if secret is soft-deleted:

   az keyvault secret list-deleted --vault-name <vault-name>

2. Recover the deleted secret:

   az keyvault secret recover \
     --vault-name <vault-name> \
     --name SqlConnectionString

3. If purged (permanently deleted), recreate the secret:

   # Get the value from its source resource (for example, the SQL Database
   # Connection strings blade, or the Storage account Access keys blade)
   az keyvault secret set \
     --vault-name <vault-name> \
     --name SqlConnectionString \
     --value "<new-secret-value>"

4. Restart App Service:

   az webapp restart --resource-group <rg> --name <app-name>

Estimated recovery time: 15-30 minutes

---

Scenario 3: Complete region failure

Symptoms: All Azure services in primary region unavailable.

Recovery steps:

1. Create new resource group in secondary region:

   az group create --name apprequest-dr --location westus2

2. Redeploy App Store into the secondary region:

   The supported path is the Azure Marketplace offer. Search the Marketplace for App Store for Intune, select the DR resource group, and complete the wizard with the same values you used originally. If you keep your own copy of the ARM template, you can deploy that instead:

   az deployment group create \
     --resource-group apprequest-dr \
     --template-file ./azuredeploy.json \
     --parameters \
       clientId=<app-store-app-registration-client-id> \
       adminGroupId=<admin-group-object-id> \
       sqlAdminLogin=<sql-admin-login> \
       sqlAdminPassword=<password>

   There is no client secret parameter and no certificate parameter. App Store uses one app registration for sign-in, and the App Service’s own managed identity for everything privileged.

   If the original deployment had unattended API access enabled, also pass workloadApiAudience with the same value it had. That is an identifier rather than a credential, and the application role assignment on your Automation Account’s managed identity is unaffected by the rebuild because it belongs to the workload registration, not to anything being recreated.

   Pass the same adminGroupId you use in production. Left empty, the deployment falls back to making whoever ran it the single temporary administrator, which during an outage is rarely the person who should hold that access.

3. Grant Microsoft Graph permissions to the new managed identity:

   The recovered App Service has a new system-assigned managed identity, and Graph permissions belong to that identity rather than to the app registration, so they don’t come across with the template. Run the permission script against the new App Service. Until you do, users can sign in but every Intune operation fails.

4. Restore the database into the deployment you just created:

   The template created an empty database named AppRequestPortal, and the App Service is already pointed at it. If you restore the backup somewhere else, the portal will come up against the empty database, migrate it, and look completely healthy with none of your data. The restore therefore has to end with the recovered data sitting under that exact name on that exact server.

   # Names of the resources the template just created.
   DR_RG=apprequest-dr
   NEW_SERVER=$(az sql server list --resource-group $DR_RG --query "[0].name" -o tsv)
   NEW_APP=$(az webapp list --resource-group $DR_RG --query "[0].name" -o tsv)
   
   # 1. Find the geo-backup of the production database.
   az sql db geo-backup list \
     --resource-group <original-rg> \
     --server <original-server> -o table
   
   # 2. Restore it onto the new server under a temporary name.
   az sql db geo-backup restore \
     --resource-group $DR_RG \
     --dest-server $NEW_SERVER \
     --dest-database AppRequestPortal-restored \
     --geo-backup-id <backup-id>
   
   # 3. Stop the site so nothing writes to the empty database while you swap them.
   az webapp stop --resource-group $DR_RG --name $NEW_APP
   
   # 4. Put the restored database in place of the empty one.
   az sql db delete --resource-group $DR_RG --server $NEW_SERVER \
     --name AppRequestPortal --yes
   az sql db rename --resource-group $DR_RG --server $NEW_SERVER \
     --name AppRequestPortal-restored --new-name AppRequestPortal
   
   # 5. Start it again.
   az webapp start --resource-group $DR_RG --name $NEW_APP

   Caution

   Do not skip the rename by repointing the connection string instead. The connection string lives in a Key Vault secret that the ARM template writes, so the next deployment would point the portal back at a database that no longer exists.

5. Re-establish the grants that were tied to the old identities.

   These are not carried by the template, because they were made to identities that no longer exist. See Identity and grant reconstruction for the full list and why each one is needed. The ones that apply to most deployments:

   • Microsoft Graph permissions (step 3 above). Without them, nothing involving Intune works.
   • Power BI workspace access, if you use Group Automations. Add the new App Service managed identity to the BI for Intune workspace with Build permission. If your Power BI tenant settings are scoped to a security group rather than the whole organization, add the new identity to that group as well.
   • Exchange Online application access policy, if you created one to restrict Mail.Send to a single mailbox. It names the old identity’s application ID, so it neither permits the new identity nor restricts it. Recreate it against the new application ID.

6. Rebuild the Teams app package, if the Teams bot is enabled.

   The template creates a new user-assigned managed identity for the bot, with a new client ID, and the App Service has a new hostname. The Teams app manifest carries both. Rebuild it with the new teamsBotAppId and the new URL from the deployment outputs, then upload the new package to Teams. See Configure Microsoft Teams bot.

7. Recover storage rather than replacing it, where you can.

   Check the existing storage account first. It is geo-redundant by default, so if the region failed but the account did not, the supported move is a customer-initiated account failover, which promotes the secondary and keeps the account name and endpoints. That is preferable to abandoning it: point the recovered App Service at the failed-over account instead of the empty one the template created.

   Failover is asynchronous, so expect a small amount of recent data not to have replicated, and note that the account is locally redundant in its new primary region until you reconfigure redundancy.

   If you genuinely need a new account, nothing of consequence is lost. This container holds the working area of the packaging pipeline, not finished product state. See what is committed and what is in flight.

   • Every app you have already published stays published, and version rollback is unaffected. Both live in the database and in Intune, not here.
   • An operation that was still running when the disaster struck has to be started again. A packaging job mid-flight, or a custom MSI uploaded but not yet packaged, is unfinished work: run it again, or upload the installer again. Nothing that had finished is affected.
   • The PSADT template cache is empty, so the first packaging job re-downloads and re-verifies it. Automatic, given outbound HTTPS to github.com.

   If the original storage account is still readable, copy the containers across with az storage blob copy start-batch before you consider the recovery complete.

8. Update Entra ID redirect URIs:

   • Add the new App Service URL to the App Store app registration’s Single-page application redirect URIs

9. Update DNS (if using custom domain):

   • Point DNS to new App Service
   • Request new SSL certificate or migrate existing

10. Work through the post-recovery validation checklist.

    A portal that loads and lets people sign in is not a recovered portal. Sign-in exercises the app registration, which the rebuild never touched, so it proves almost nothing about the parts that were rebuilt.

Estimated recovery time: 3-6 hours, most of it waiting for the geo-restore. Re-establishing the grants in steps 3, 5 and 6 is quick, but each one needs somebody with the right administrative role, so find out who those people are before you need them.

---

Scenario 4: Storage account failure

Symptoms: WinGet packaging jobs fail, queue errors.

Recovery steps:

1. For GRS accounts, initiate failover. This is the preferred path:

   az storage account failover \
     --name <storage-account-name> \
     --resource-group <rg>

   Failover promotes the geo-replicated secondary and keeps the account name and endpoints, so App Store’s configuration continues to work unchanged. Prefer it over creating a replacement account.

   Note: Failover may take up to 1 hour and makes the secondary region the new primary. Replication is asynchronous, so recent writes may not have reached the secondary. After failover the account is locally redundant in its new primary region until you reconfigure redundancy.

2. Alternative, only if the account cannot be recovered - create a new storage account.

   A new account is empty, and that costs you only unfinished work. Version rollback is unaffected, because it reads the database and Intune rather than stored packages, and every published app stays published. An operation that was still running has to be started again. See What is committed and what is in flight.

   az storage account create \
     --name <new-storage-name> \
     --resource-group <rg> \
     --location <location> \
     --sku Standard_GRS

3. Update Key Vault with new connection string:

   CONNECTION_STRING=$(az storage account show-connection-string \
     --name <new-storage-name> --resource-group <rg> -o tsv)
   
   az keyvault secret set \
     --vault-name <vault-name> \
     --name StorageConnectionString \
     --value "$CONNECTION_STRING"

4. Restart App Service:

   az webapp restart --resource-group <rg> --name <app-name>

Estimated recovery time: 1-2 hours

---

Scenario 5: Application corruption / bad deployment

Symptoms: Application crashes, unexpected behavior after update.

Recovery steps:

1. Roll back the application package:

   # Point the App Service at the previous version on the package host.
   # Each release is published with a versioned filename in its channel folder, e.g.:
   #   https://bi.powerstacks.com/appstoreforintune/bin/latest/AppRequestPortal-1.33.0.zip
   # (preview-channel builds live under .../bin/preview/)
   az webapp config appsettings set \
     --resource-group <rg> \
     --name <app-name> \
     --settings WEBSITE_RUN_FROM_PACKAGE="https://bi.powerstacks.com/appstoreforintune/bin/latest/AppRequestPortal-<version>.zip"

2. Restart App Service:

   az webapp restart --resource-group <rg> --name <app-name>

3. Verify the rollback was successful.

Roll back the database too if the bad release changed the schema

App Store applies Entity Framework Core migrations automatically on startup, and migrations are generally forward-only. If the release you are rolling back from added or changed database schema, the older package may fail to start against the newer schema. In that case, also restore the database to the pre-upgrade point in time, or to the BACPAC export you took before upgrading. See SQL Database backups.

To avoid this, take a database backup before every upgrade that may change the schema: a BACPAC export to the storage account, or simply note the UTC timestamp so you can use point-in-time restore.

Estimated recovery time: 5-15 minutes (add 30-60 minutes if a database restore is required)

---

Identity and grant reconstruction

A recovery that redeploys the ARM template does not restore your deployment. It builds a new one and then puts your data in it. Anything that was granted to a specific identity or resource, rather than written in the template, refers to something that no longer exists.

This section exists because the failure is quiet. Authentication belongs to the app registration, which a rebuild never touches, so the recovered portal signs people in and renders the catalog from the restored database while everything privileged fails.

What changes in a rebuild

Resource names are derived from the resource group, so deploying into a new resource group renames everything: the App Service and its URL, the SQL server, the Key Vault, the storage account, and the bot identity.

Identity or resource	What happens on rebuild
App Service system-assigned managed identity	Destroyed and recreated. New object (principal) ID and new application ID. Nothing granted to the old one carries over
Bot user-assigned managed identity	A new resource with a new name, new client ID and new object ID
App Service URL	New hostname, because the App Service name is derived from the resource group
Key Vault, storage account, SQL server	New resources with new names. Key Vault contains only the secrets the template writes; the storage account is empty
App registration	Unchanged. It is yours, not the template’s, which is why sign-in keeps working and proves nothing

Grants the template restores for you

Grant	Why it survives
Website Contributor on the App Service, held by its own managed identity	The template creates the role assignment against whatever principal ID the new site has. This is what lets the portal update its own package during an in-app update
Key Vault access policy granting the App Service get and list on secrets	Created by the template against the new principal ID. This is what lets the SQL and storage connection strings resolve
Bot registration to bot identity binding	The template sets the bot’s msaAppId from the new user-assigned identity
SQL and storage connection strings	Rewritten by the template to name the new resources

Nothing in this table needs a human. If one of them is wrong, the deployment itself failed.

Grants a human has to re-establish

Grant	Keyed to	Who can do it	What it costs if missed
Microsoft Graph application permissions (the seven required, plus Mail.Send if you use email)	The App Service managed identity’s service principal object ID	Privileged Role Administrator or Global Administrator	Everything involving Intune fails with “Insufficient privileges”. Sign-in is unaffected
Power BI workspace access, Build permission	The managed identity, added as a workspace member	Power BI workspace admin	Group Automations cannot read BI for Intune, so automations match nothing
Power BI tenant settings, when scoped to a security group	The managed identity’s membership of that group	Fabric (Power BI) administrator, or the group’s owner	Same as above, and the failure looks identical
Exchange Online application access policy, if you created one	The managed identity’s application (client) ID	Exchange administrator	The policy no longer describes the portal. Depending on your configuration this either fails to restrict the new identity or, if Mail.Send is otherwise constrained, blocks all notification email
Teams app package	The bot identity’s client ID and the App Service hostname	Teams administrator, to upload the rebuilt package	The bot cannot be reached, so Teams notifications stop
App registration redirect URI	The App Service hostname	Whoever owns the app registration	Sign-in fails with a redirect URI mismatch. This one is loud, and the only one that is

What is committed and what is in flight

Recovery restores everything App Store had finished. Work that was still running when the disaster happened is started again. That line is worth being precise about, because it decides what you should expect to find after a recovery.

A packaging operation is finished when the application exists in Intune and App Store has recorded its identifier. In order, a publish streams the package into Intune’s own storage, has Intune commit the content, finalizes the application, and only then writes the Intune application id and a completed status into the App Store database. The installer content a device eventually downloads is held by Intune from that point on, not by App Store.

	Examples	What recovery does
Committed, and restored	Published apps, version history, requests, approvals, audit history, workflows, settings, licensing, deployment and ring records	Restored with the database, within its documented restore point
External, and never lost	The Intune application objects themselves, and their assignments	Nothing to restore. Intune is a separate service and is unaffected by the loss of an Azure region or subscription
In flight, and restarted	A packaging job that had not finished, a queued job, a custom MSI uploaded but not yet packaged, the working .intunewin of a job still running	Start the operation again. Nothing that had finished is affected

A recovered portal must never show unfinished work as finished. A job that was interrupted appears as the incomplete job it was, and you re-run it.

What rollback actually needs

Rolling an application back to a previously deployed version is a shipped capability, so it is worth being exact about what a recovery has to restore for it to work.

Rollback rebuilds the known-good version from a package App Store kept for it. For every version inside your Number of rollback versions to retain setting, App Store holds the packaged .intunewin and the metadata needed to reconstruct the application definition. A rollback fetches that package, checks it against the digest recorded when it was stored, and republishes it into the product’s install application.

It does not need the old version’s Intune application object. You are free to delete historical Win32 apps in Intune; rollback still works. That is deliberate, because administrators do tidy those up, and the previous design failed in exactly that case.

So rollback depends on three things, and two of them are durable state a recovery must restore:

Requirement	Where it lives	Recovered by
The version history row	Azure SQL	the database restore
The retained rollback package and its metadata	Blob Storage, in the rollback container	the storage account, which a recovery must restore
The product’s install application	Microsoft Intune	nothing; Intune is unaffected by the loss of an Azure region or subscription

Caution

Retained rollback packages are durable product state. A version inside your rollback retention count is only rollback-capable while its package exists. Losing the rollback container costs you the ability to roll back to those versions, and the portal will say so rather than offering a rollback it cannot perform.

This is the one place where the packaging container and the rollback container differ in importance. They are separate containers for exactly this reason.

Packaging work is still transient. The working container holds the intermediate files of in-progress packaging jobs and is swept by age. Losing it costs an unfinished job, which is re-run. That sweep cannot touch the rollback container.

Post-recovery validation checklist

Each item exercises a dependency that a rebuild breaks, using something the portal already does. Do them in order: a failure early on explains failures later.

#	Check	How	Expected
1	The portal is running and reaching its database	Open https://<new-app-url>/health	status is Healthy or Degraded, and the database check is Healthy. A Healthy result here does not mean Graph permissions were restored: this endpoint checks that settings are present, not that any grant works
2	Your data came back, not an empty schema	Sign in and open App Management	Your apps, requests and approval history are present. An empty catalog on a recovered instance means the database swap in step 4 did not take effect
3	Sign-in works	Sign in as a normal user	The portal loads. This proves the redirect URI and the app registration, and nothing else
4	Graph permissions were restored	In App Management, run Sync from Intune	The sync completes. “Insufficient privileges” or a 403 means the permission script has not been run against the new identity
5	Intune write access works	Publish or update one low-risk app	The Win32 app appears in Intune. Reads can succeed while writes fail if only some permissions were granted
6	Email notifications work	Submit a test request that requires approval	The approver receives the email. Nothing arrives if Mail.Send was not granted, or if an Exchange application access policy still names the old identity
7	Teams notifications work, if enabled	Store Administration > Teams settings > Test	A message arrives. A failure here usually means the rebuilt Teams package was not uploaded
8	Group Automations work, if used	Group Automations > Test connection	The connection succeeds. A failure means the new managed identity is not on the Power BI workspace, or the tenant setting does not cover it
9	Nothing drifted while you were down	Open Store Administration > reconciliation view (/api/admin/reconciliation)	Review anything listed. Changes made in Intune during the outage are not automatically reconciled
10	Self-update still works	Store Administration > System	The current version is reported and Install Update is available. This exercises the Website Contributor assignment
11	Rollback capability is intact	Open an app with more than one version in Version history	Previous versions are listed and offered for rollback. This proves the version history survived the database restore and still names Intune application objects. If you can afford to, roll one low-risk app back and forward again
12	Unfinished work is reported honestly	Store Administration > packaging jobs	Any job that was running when the disaster struck shows as incomplete or failed, never as completed. Re-run it. A recovered portal that claims an interrupted job finished is a defect worth reporting

Caution

Do not close a recovery at check 3. Checks 1 to 3 pass on a deployment whose managed identity holds no permissions at all, because they exercise the database, the app registration and the site itself, none of which the rebuild replaced. Check 4 is the first one that touches a grant the rebuild destroyed.

---

High availability setup (Tier 3)

For organizations requiring higher availability, implement these additional measures manually.

SQL active geo-replication

Creates a readable secondary database in another region with continuous replication.

Setup:

1. Create geo-replica:

   az sql db replica create \
     --resource-group <primary-rg> \
     --server <primary-server> \
     --name AppRequestPortal \
     --partner-server <secondary-server> \
     --partner-resource-group <secondary-rg>

2. Configure failover group (recommended):

   az sql failover-group create \
     --resource-group <primary-rg> \
     --server <primary-server> \
     --name apprequest-fog \
     --partner-server <secondary-server> \
     --partner-resource-group <secondary-rg> \
     --failover-policy Automatic \
     --grace-period 1

3. Update connection string to use failover group endpoint:

   Server=tcp:apprequest-fog.database.windows.net,1433;...

Cost: ~$5/month additional (Basic tier replica)

Recovery time: Automatic failover in 1-2 minutes

Traffic Manager / Front Door

Distributes traffic across multiple App Service instances.

Setup:

1. Deploy secondary App Service in another region using ARM template

2. Create Traffic Manager profile:

   az network traffic-manager profile create \
     --resource-group <rg> \
     --name apprequest-tm \
     --routing-method Priority \
     --unique-dns-name apprequest-portal

3. Add endpoints:

   # Primary
   az network traffic-manager endpoint create \
     --resource-group <rg> \
     --profile-name apprequest-tm \
     --name primary \
     --type azureEndpoints \
     --target-resource-id <primary-app-id> \
     --priority 1
   
   # Secondary
   az network traffic-manager endpoint create \
     --resource-group <rg> \
     --profile-name apprequest-tm \
     --name secondary \
     --type azureEndpoints \
     --target-resource-id <secondary-app-id> \
     --priority 2

Cost: $0.75/million queries + secondary App Service ($55/month)

Storage RA-GRS with manual failover

Enables read access to secondary region for faster failover decisions.

Setup:

Change storage redundancy parameter during deployment:

"storageRedundancy": "Standard_RAGRS"

Cost: ~$0.50/month additional over GRS

---

Backup verification

Monthly backup test checklist

• SQL Point-in-Time Restore Test

  • Restore database to 24 hours ago
  • Verify data integrity
  • Delete test database

• Key Vault Recovery Test

  • List soft-deleted secrets
  • Verify recovery capability
  • Document secret expiration dates

• ARM Template Deployment Test

  • Deploy to test resource group
  • Verify all resources created
  • Delete test deployment

• Document Recovery Time

  • Record actual time for each recovery step
  • Update estimates if needed

Automated monitoring

Configure Azure Monitor alerts for:

Alert	Condition	Action
SQL DTU	> 90% for 15 min	Email admins
App Service Health	Unhealthy	Email + webhook
Key Vault Access Denied	Any	Email security team
Storage Availability	< 99%	Email admins

---

Emergency contacts

Role	Contact	Responsibility
Primary Admin	[Your contact]	First responder for incidents
Azure Support	https://portal.azure.com/#blade/Microsoft_Azure_Support	Severity A for production down
Security Team	[Security contact]	Data breach or security incidents

Azure support plans

Plan	Response Time (Sev A)	Cost
Basic	No SLA	Free
Developer	8 hours	~$29/month
Standard	1 hour	~$100/month
Professional Direct	15 minutes	~$1,000/month

Recommendation: Standard support for production deployments.

---

Recovery decision tree

Is the application accessible?
├── YES: Can users sign in?
│   ├── YES: Is data correct?
│   │   ├── YES: No DR needed
│   │   └── NO: → Scenario 1 (Data Deletion)
│   └── NO: Key Vault issue? → Scenario 2
└── NO: Is it a single resource or entire region?
    ├── Single resource:
    │   ├── SQL → Restore from backup
    │   ├── Storage → Scenario 4
    │   ├── App Service → Redeploy from ARM template
    │   └── Key Vault → Scenario 2
    └── Entire region: → Scenario 3 (Region Failure)

---

Appendix: Resource IDs quick reference

After deployment, record these values for emergency use:

Resource	How to Find
Resource Group	Azure Portal → Resource Groups
SQL Server Name	Deployment outputs or Portal
Storage Account Name	Deployment outputs or Portal
Key Vault Name	Deployment outputs or Portal
App Service Name	Deployment outputs or Portal

Store this information securely outside of Azure (e.g., password manager, printed runbook).