Configure the Nerdio Manager Web App for High Availability (HA) or Disaster Recovery (DR) Scenarios

Configure the Nerdio Manager Web App for High Availability (HA) or Disaster Recovery (DR) Scenarios

This article provides an overview of protection and resilience concepts, the requirements and supported configurations for the Nerdio Manager platform, known limitations, and caveats.

All functions used to support a resilient Nerdio Manager environment are Azure native, however the ability to configure Nerdio Manager in a resilient configuration is only supported for the Premium edition of the product. Full guidance for resilience configuration is provided below.

Note: It is strongly recommended that customers use paired regions for SQL deployment Failover Group deployment. See Configure Nerdio Manager Database Resilience for more information.

Concepts

Note: As of Nerdio Manager v6.5, HA is supported for the web app layer only. The SQL database layer may be configured for automatic failover as described in this article. However, this failover is classed as a DR invocation action due to the active/passive nature of SQL failover groups. As SQL is a core functional component of the service, customers should be aware that Nerdio Manager does not support true HA functionality at this time. Brief service outages should be expected and planned for. This limitation and associated processes should be recorded in your DR and business continuity planning.

High Availability (HA) describes a scenario where a service runs using multiple parallel instances of the service. HA expects that, in the event that one instance of the service fails, the service continues to run without downtime or impact. HA solutions are generally more expensive to maintain, because they require multiple services to be running simultaneously, therefore increasing the resource usage and potential cost of the service.

Disaster Recovery (DR) describes a scenario where a service runs using a single instance of the service, however in the event of a failure of that service, a process may be followed to fail the service over to another instance of the service. The secondary instance is often disabled or stopped until disaster recovery or testing is required (known as DR invocation).

DR-enabled services require manual or automatic intervention to perform the invocation and failover actions, which may result in downtime and/or service outages. The time required for these actions varies by service or configuration, and organizations are responsible for deciding what duration of outage may be tolerated.

The Nerdio Manager web app configuration requirements for both HA and DR scenarios are essentially the same. The key difference is that HA mode requires 2 or more web apps to be running simultaneously, whereas DR mode only requires a single web app to be running. The Azure Front Door service acts as the load balancing layer for these web apps.

In HA mode, requests are routed equally between the active web apps. In DR mode, Azure Front Door intelligently detects that one or more web apps are offline and routes requests to another functioning web app.

Functional Overview

The following diagram provides a high level overview of Nerdio Manager’s resilient configuration.

Service components are divided into the following conceptual layers:

Load Balancing Layer

This is where users connect to the Nerdio Manager web app console. In a resilient scenario, these requests are managed by Azure Front Door (or an alternate load balancing service).

Connections are dynamically routed to an available Nerdio Manager web app instance in the web app layer.

Storage Layer

The storage layer is comprised of the Azure storage accounts, which form part of the core Nerdio Manager platform. Only 2 key storage accounts are shown in the diagram above, which are:

  • The Scripted Action storage account (prefixed CSSA by default), which contains all the scripts that are used to complete the core deployment of hosts within a host pool. If this storage account is unavailable, host creation activities, including auto-scale tasks, fail.

  • The Data Protection storage account (prefixed with DPS by default), which contains the encryption keys required to add encrypted secrets to, and read secrets from, the database.

Note: Nerdio Manager currently supports Zone Resilience (ZRS) and Geo-Redundancy (GRS) at the storage layer. Review the Requirements and Limitations sections of this document for more information. In addition, this limitation should be noted in organizational DR/BC documentation.

Web App Layer

The web app layer refers to the running or stopped web apps that constitute a resilient application environment. Users accessing the console connect to these web apps (if available) via the Load Balancing layer.

In an HA scenario, a minimum of two active Nerdio Manager web apps must be running and accessible simultaneously. In a DR scenario, at least one active and one offline Nerdio Manager web app must be present. To invoke DR, an administrator must start one of the offline web apps for services to be resumed.

This layer also contains the Nerdio Manager key vault, which is acceded by the web apps as required to read secrets data. Because key vaults are designed to be globally resilient within the Azure infrastructure, no additional resiliency has been added to this item.

SQL Database Layer

The SQL Database Layer refers to the databases that store Nerdio Manager configuration information. The web apps communicate with the active SQL database.

The SQL database must be configured for resilience before proceeding with any other HA or DR configuration. Please refer to this article for further information.

Read and write requests from the active web apps are routed to the active database in the SQL failover group.

Configuration Guidance

Note: The sections below describe the requirements and processes required to configure the Nerdio Manager for Enterprise web app for resilience. This process does not take account of any network or infrastructure hardening that may exist in your environment. Please speak with a member of the Nerdio Support team or your Technical Account Manager if you wish to discuss this further.

Requirements

  • This feature requires Nerdio Manager Premium Edition licensing.

  • The Microsoft.Cdn content provider must be registered for your subscription.

  • Data Protection Keys must be migrated as part of the configuration process. Check that your environment contains a Data Protection Keys Storage Account prefixed with dps before starting this process. If it does not, please follow the process outlined below. If it does not, please review the section Key migration process (Required for Nerdio Manager environments installed before v5.5),and contact the Nerdio support team as described.

    • Please also ensure that the creation script is configured to create the DPS storage account as a Geo-Redundant Storage account (GRS).

    • Existing DPS accounts configured as LRS must be converted to GRS by following the process described here.

  • This feature must be configured in combination with Nerdio Manager Database Resilience. It is recommended that resilient databases are created in the same regions as the web apps, as described here.

  • Additional web apps and load balancing services must be manually created in Azure to support this feature.

  • Configuration changes should be tested at every stage to ensure that Nerdio Manager is functioning correctly, as described in the guidance. If issues are detected, please follow the suggested roll back process, or contact the Nerdio support team.

  • All Nerdio Manager web app instances must be updated together, and all must be running the same version of Nerdio Manager.

  • If you are planning to use Private Endpoints for your web apps, Azure Front Door Premium is required in combination with the Private Link feature. Please refer to this article for more information.

Limitations

  • Environments configured for HA or DR must be updated using Cloud Shell (Method 2) or Zip Deploy (Method 4). For details, see Update the Nerdio Manager Application.

  • All Nerdio Manager web app resources must be in the same subscription.

  • All Nerdio Manager web app resources must be in a single resource group.

  • In DR mode, Azure Front Door may report warnings related to web app availability. This is expected.

  • In DR mode, invocation requires that standby web apps must be manually started from the Azure portal.

  • If storage account connectivity in the regions or zones containing the CSSA storage account is not possible, associated actions that leverage Nerdio Manager scripts fail. This includes auto-scale operations.

  • If storage account connectivity in the regions or zones containing the DPS storage account is not possible, the Nerdio Manager console is inaccessible until connectivity is restored.

  • GRS replication zones for the storage accounts are selected by Azure and the configuration may not be changed. Please refer to this Microsoft article that describes the paired regions. If the paired region differs from regions where the SQL failover group or web app replicas are configured, this may impact Nerdio Manager performance in the event of a failover.

  • The Nerdio User Cost Attribution web app is not currently supported for either DR or HA scenarios. Failover may lead to non-critical console errors until normal operations are resumed.

Configuration Process

Note: Ensure that all resources are deployed to the same subscription and resource group as the original Nerdio Manager web app.

To create a new app configuration resource:

  1. In the Azure portal, create a new App Configuration resource using the Standard plan:

    • Location: Select the location, which should be the same region as your web app.

    • Name: Provide a name.

    • Pricing tier: Select Standard.

    • Enable replication: Enable this option, and then in the right pane that opens, enter a name in the Replica name field. This can be any name, for example, replica. Select Create.

  2. Change the following options:

    • On the Access settings tab, set the Authentication mode to Pass-through.

    • In the Networking tab, select Enable Azure Resource Manager Private Network Access.

  3. Select Review + create and then select Create.

  4. Open the new App Configuration resource.

  5. In the Operations blade, select Import/export.

  6. Change the following options:

    • Source type: Select App Services.

    • Resource group: Select your existing Nerdio Manager app service.

  7. Once you have made all the changes, select Apply.

  8. In the Operations blade, select Configuration explorer.

  9. For each of the following keys, make a note of the key and value from the Advanced edit tool. These are only required in a roll back scenario.

    • ApplicationInsights:ConnectionString

    • ApplicationInsights:InstrumentationKey

    • Deployment:WebAppName

  10. Select the following keys:

    • ApplicationInsights:ConnectionString

    • ApplicationInsights:InstrumentationKey

    • Deployment:WebAppName

  11. Once you have made the selections, select Delete.

  12. In the Access Control (IAM) blade, select +AddAdd role assignment.

  13. In the Members tab, change the following:

    • Selected role: Select App Configuration Data Reader.

    • Assigned access to: Select Manage identity.

    • Members: Select +Select members and then add the Nerdio Manager App Service object.

  14. Apply the changes.

  15. In the Settings blade, select Access settings.

  16. Copy and save the value of the Endpoint field.

  17. Open the Nerdio Manager App Service resource (nmw-app-#############).

  18. In the Settings blade, select Environment variables.

  19. Delete all the variables except for the following:

    • ApplicationInsights:ConnectionString

    • ApplicationInsights:InstrumentationKey

    • Deployment:WebAppName

  20. On the same page, create a new setting named AppConfiguration:Endpoint.

  21. Paste the Endpoint value that was previously saved.

  22. Apply the changes.

  23. Restart Nerdio Manager app service.

  24. Verify that the service works as expected., then move onto the next section of this guide. If issues are detected, please contact Nerdio Support.

To modify the Data Protection storage account:

Note: If the dps storage account described below does not exist in your environment, skip forward to the supplementary guidance in Key Migration process (Required for Nerdio Manager environments installed before v5.5). Once complete, return to this section and complete the required activities.

  1. In the Azure portal, navigate to Storage accounts.

  2. Select the Nerdio Manager Data Protection Keys Storage Account.

    Note: This has a default resource name in the format dps#############.

  3. In the Data storage blade, select Containers.

  4. Select +Containers.

  5. In Name, type locks.

  6. Select Create.

  7. Right-click on the new container and select Generate SAS.

  8. Change the following options:

    • Permissions: From the drop-down list, select Read, Write, and Create.

    • Expiry: Select a long-term date (for example, 2099 or 2999).

  9. Once you have made the changes, select Generate SAS token and URL.

  10. Copy and save the value from Blob SAS URL.

  11. Navigate to Key vaults.

  12. Open the Nerdio Manager application key vault.

    Note: This has a default name off nmw-app-kv-#############.

  13. In the Objects blade, select Secrets.

  14. Select +Generate/Import.

  15. Enter the following information:

    • Name: Type Deployment--LocksContainerSasUrl.

    • Secret value: Paste the SAS URL that you previously saved.

  16. Once you have entered the desired information, select Create.

Key migration process (Required for Nerdio Manager environments installed before v5.5)

Note: This is supplementary guidance and is only required if your Nerdio Manager instance does not have a Data Protection Keys Storage Account prefixed with dps. This must be created and keys migrated prior to moving forward.

  1. Run the script migrate-dataprotection.ps1 from a Cloud Shell instance.

    Note: This script can be obtained from the Nerdio support team.

  2. The required script parameters are:

    ./migrate-dataprotection.ps1 -resourceGroupName <ResourceGroupOfYourWebAppName> -webAppName <YourWebAppName>

  3. Restart the Nerdio Manager web app.

  4. After running the script, review and confirm the following items:

    • A storage account prefixed with dps has been created in the deployment resource group and it has a container named dataprotectionkeys.

    • A Key Vault prefixed with nmw-app-kv exists and has:

      • A key named DataProtection-XXXXXXXXXXXXX.

        Note: For example, the Key Identifier (without version) is https://nmw-app-kv-XXXXXXXXXXXXX.vault.azure.net/keys/DataProtection-XXXXXXXXXXXXX.

      • A secret named DataProtection--Storage--Path exists, and its value is a valid.

        Note: Test this by copying the secret value, then paste this into a new browser tab. You should see that the XML contains all keys from the steps above, and contains the comment This key is encrypted with Azure Key Vault.

    • The App Service configuration has the following environmental variables defined:

      • DataProtection:Storage:Type is set to AzureBlobStorage.

      • The DataProtection:Protect:KeyIdentifier value contains a key identifier.

  5. Once all of the above has been confirmed, move forward to the next steps.

  6. Restart the App Service.

  7. Navigate to Storage accounts.

  8. Select the Nerdio Manager storage account that contains the locks container you previously created.

  9. In the Data storage blade, select Containers.

  10. Select the locks container.

  11. Verify that the new blob files background.loop and web.startup were created in the container.

Configure the Azure Front Door

Nerdio recommends the use of the Azure Front Door service to provide load balancing and intelligent session routing to the web app. Other load balancing services may be validated in the future.

To configure the Azure Front Door:

  1. In Azure Portal, navigate to Front Door and CDN profiles.

  2. Select +Create.

  3. Select Custom create and then select Continue to create a Front Door.

    Note: Front Door Premium may be required if used in combination with Private Endpoints.

  4. Enter the following information:

    • In the Basics tab:

      • Resource group: Select the resource group where your Nerdio Manager application is deployed.

      • Name: Type a name.

    • In the Endpoints tab:

      • Select Add an endpoint.

      • Type a preferred name, and then select Add.

  5. Once you have entered, all the desired information, select Review + create and then Create.

  6. Select the new Front Door.

  7. In the Routes box, select + Add a route.

  8. Enter the following information:

    • Name: Type Default-Route or any name you want.

    • Accepted protocols: From the drop-down list, select HTTPS only.

    • Forwarding protocol: Select HTTPS only.

    • Select Add a new origin group.

      • Name: Type Default-Origin-Group or any name you want.

      • Enable session affinity: Select this option.

      • Health probes > Protocol: Select HTTPS.

      • Select +Add an origin and enter the following information for each origin you create.

        Note: An origin must be added for each instance of the Nerdio Manager web app.

        • Name: Type Nerdio-Webapp-[number].

          Note: Any name may be used, but each name must be unique.

        • Origin type: From the drop-down list, select App services.

        • Health probes > Protocol: Select HTTPS.

        • Health probe method: Select GET.

        • Health probe path: Set to /public/health/ping.

        • When all the options have been entered, select Add.

  9. Save all changes and complete the Azure Front Door profile creation.

  10. Select the new Azure Front Door.

  11. In Overview, copy and save the value of the Endpoint hostname.

    Note:The suffix here is *.azurefd.net, as shown above.

  12. In the Settings blade, and select Properties.

  13. Copy and save the Resource ID.

  14. Navigate to App Configuration.

  15. Select the resource you created in the Configuration Process section of this document.

  16. In the Operations blade, select Configuration explorer.

  17. Create the following key names:

    • Deployment:MultiInstance:ProxyHostname: The key value should be the Endpoint hostname previously saved.

    • Deployment:MultiInstance:FrontDoorProfileId: The key value should be the Resource IDpreviously saved.

  18. Navigate to App registrations.

  19. Select your Nerdio Manager app registration.

  20. In the Manage blade, select Authentication.

  21. Delete any existing automatically created Redirect URIs.

  22. Add the following rules, replacing AFD_ENDPOINT_HOSTNAME with the previously saved value for the load balanced address:

    • https://AFD_ENDPOINT_HOSTNAME/signin-oidc

    • https://AFD_ENDPOINT_HOSTNAME/

  23. Modify Front-channel logout URL to https://AFD_ENDPOINT_HOSTNAME/signout-oidc.

  24. Save all the changes.

  25. Restart the app service.

    After the restart, Nerdio Manager should be available on the load balanced Endpoint hostname URL.

Create Additional App Services - Part 1

Note: Repeat the process below for each additional required instance. This process requires application down time.

The prerequisites have now been completed and load balancing functionality has been tested and verified. The next step is to create the additional app services as required.

To create additional app services:

  1. In the Azure portal, navigate to App Services.

  2. Select the Nerdio Manager app service.

  3. In the Settings blade, select WebJobs.

  4. Select the Run option, and then select Stop.

  5. In the Development Tools blade, select Clone App.

  6. Enter your desired Name, Region, and App Service Plan.

  7. Select Clone.

    This operation takes about 5-10 minutes.

  8. Open to the newly created App Service.

  9. In the Settings blade, select Identity.

  10. In the System assigned tab, set Status to On.

  11. Select Save.

  12. In the Settings blade, select Environmental variables.

  13. Change Deployment:WebAppName to the newly created Web App resource name created above.

  14. Navigate to the App Configuration.

  15. Select the newly created resource.

  16. In the Access Control (IAM) blade, select +AddAdd role assignment.

  17. In the Members tab, change the following:

    • Selected role: Select App Configuration Data Reader.

    • Assigned access to: Select Manage identity.

    • Members: Select +Select members and then add the Nerdio Manager App Service object.

  18. Apply the changes.

  19. Navigate to Key vaults.

  20. Select the Nerdio Manager Key Vault resource with the standard prefix name nmw-app-kv.

  21. In the Access policies blade, select +Create.

  22. Enter the following information:

    • In the Permissions tab:

      • Cryptographic Operations: From the drop-down list, select Unwrap Key and Wrap Key.

      • Secret Management Operations: From the drop-down list, select Get, List, Set, and Delete.

    • In the Principal tab, search for and select the Nerdio Manager Web App Service Principal.

  23. Once you have entered all the desired information, select Review + create and then Create.

App Services - Part 2

Note: The steps below should only be completed after all additional required Web Apps are created.

  1. Ensure all app services (web apps and web jobs) are started.

  2. Navigate to Front Door and CDN profiles.

  3. Select the new Azure Front Door profile.

  4. In the Settings blade, select Origin groups.

  5. Select the newly created Origin Group.

  6. Select +Add an origin.

  7. Enter the same values as for the previously created origin, but select the newly copied app service in Host Name.

  8. Save all changes.

    The operation can take up to 5 minutes.

Testing

The following tests should be performed:

  • Open the Load balanced Azure Front Door URL multiple times, ideally from multiple endpoints. The Nerdio Manager console should open without any errors.

  • Attempt to stop each app service alternately. Services are unavailable for the configured Azure Front Door Health probes interval (default is 100s) during a failover event.

  • Ensure that the Nerdio Manager is accessible after the above timeout, even if only a single instance is running.

  • Review the Azure Front Door traffic logs for any errors or misconfigurations.

Optional: Restrict Access to the Azure Front Door Endpoint Only

Optionally, configure the app services to block any attempts to contact the web app using its native web address. This ensures that all communications pass through the Azure Front Door service.

To restrict the Azure Front Door endpoint:

  1. For each web app, in the Settings blade, select Networking.

  2. Select Enabled with no access restrictions.

  3. Enter the following information:

    • Public network access: Select Enabled from selected virtual networks and IP addresses.

    • Unmatched rule action: Select Deny.

    • In the Site access and rules section, select +Add and then enter the following information:

      • Name: Type any value (for example, AFD).

      • Priority: Type 1.

      • Type: From the drop-down list, select Service Tag.

      • Service Tag: From the drop-down list, select AzureFrontDoor.Backend.

      • X-Azure-FDID: Optionally, include the unique headed value.

        Note: This information can be found in the Overview section of the Front Door profile under Front Door ID.

      • Save the new rule.

  4. Once you have entered all the desired information, save your changes.

Was this article helpful?

0 out of 0 found this helpful
Have more questions? Submit a request

Comments (0 comments)

Article is closed for comments.