Pexip Version Upgrades

This a guide to the upgrade process for our Pexip Infinity infrastructure.

Prerequisites

In order to proceed with an upgrade, you must:

1. Have a user that is a member of the Entra ID Role - MAX - Video Conferencing Users Group, providing administrative access to the Pexip Management interface.
2. Have an Azure account with administrative privileges to the Verify-by-Video Platform subscription. The IAM groups with that authorization are (in increasing scope of privilege):
   • Azure Roles - Verify-by-Video Subscription Contributors
   • Azure Roles - Tenant Root Contributors, and
   • Azure Roles - Tenant Root Owners
3. Have access to the Azure administrative portal with write access to VM Services in the resource groups for the environment of Pexip Infinity that you are updating.
4. Recommended software to avoid Windows entering sleep during the large file upload process: Mouse Jiggler.

Promotion pathway

Exceptions to this promotion pathway may be required for unforseen reasons. If a diversion from this pathway is required, that should be documented and this process guide updated if required.

1. If you plan to upgrade Test, you should intend to upgrade Production to the same version in a reasonable time frame (e.g., within a few weeks after testing that the new version running in Test is stable).
   • Test must not be promoted any more than 1 major version ahead of Production.
   • Test should not be promoted any more than 2 minor versions than Production.
2. Release notes review must occur before an upgrade to a new version. If possible, a vendor-led socialization session introducing the new version should be attended ahead of an upgrade.
3. If a breaking change is expected, a mitigation response must precede the upgrade to Test. The mitigation process must be documented so as to apply the same changes to Production when required.
4. The platform must be tested to ensure the functionality of the Verified by Video (VbV) service is in no way impaired by an upgrade. Checking all major functions through the PostMan API is required. Checking the VbV service using the Dev or Q/A version of the BC Services Card (BCSC) app is also required.
5. Production can only be updated to the same version as Test. Pexip provides SHA256 Checksum for their downloadable Infinity Platform upgrade files at their download page: [https://www.pexip.com/help-center/platform-download].

Snapshotting VMs

To snapshot VMs, the VM has to be turned off. You can either wait for the VM status pipeline to turn these VMs off outside of business hours, or pause that pipeline and manually turn the VMs off.

Be aware that when pausing the pipeline, the schedule will still operate, but the stages will not start until the pipeline is resumed.

1. In Azure, access the VMs involved in the Pexip environment to be upgraded.

2. For each VM (e.g., the VM for the Management node, and any VMs used for the conference nodes).

   For example, if upgrading Test, access the VM overview screens for:

   • vm-node-conf-core
   • vm-node-mgmt-core

   and if upgrading Prod, access the VM overview screens for:

   • vm-node-mgmt-test
   • vm-node-conf-test

3. In each VM, access Backup + Discovery > Backup from the options blade.

4. Within that screen, click Backup now, choose the date to retain the backup till (using the default of 1 month is normally sufficient), and press OK.

Upgrade Process

1. Download an upgrade file from [https://www.pexip.com/help-center/platform-download].
2. Ensure there is a recent snapshot of the VM. If there is not, take a snapshot once the VM is turned off and before proceeding with any next steps of the upgrade process.
3. To avoid the VM being shut down by the Virtual Machine - Start Stop Azure DevOps Pipeline, pause the schedule on that pipeline through the pipeline settings interface ((here)[https://dev.azure.com/MaximusDevOps/Verify-by-Video%20Platform/_build?definitionId=107]) by navigating to ⋮ > Settings > Paused and pressing Save.
   • If the upgrade is occuring outside of service hours, manually trigger the environment to ensure the VMs are Started and running.
4. In the Pexip Infinity management node administrative interface, uncheck the Enable management web interface session timeout checkbox located under Platform > Global Settings > Management web interface configuration.
5. Proceed with the upgrade process described at [https://docs.pexip.com/admin/upgrading.htm].
   • while uploading the file, activate Mouse Jiggler if installed to avoid Windows entering a lockscreen or sleeping and potentially interrupting the file upload process.

Post-Upgrade process

1. Log in as an administrator and verify the system appears functional
2. Test endpoints for the VbV service are functional through Postman
3. If upgrading Test, check that the BCSC app continues to work to make a call to the VbV service (this will require someone or yourself being in the VBV Test Queue in Genesys and being logged in to genesys and on queue to take your own call).
4. Turn Enable management web interface session timeout back on.
5. Resume the Virtual Machine - Start Stop pipeline.

Risks

1. Possible loss of log files stored in local storage if the upgrade fails and a rollback to the snapshot VM is required. This is mitigated by the fact that we use a remote syslog to collect logs from Pexip Infinity.
2. Upgrade delays are possible due to session timeouts or network outages while performing an upload of the upgrade file to the Pexip Infinity host. Network outages can briefly occur and cause the upload process to fail when Windows enters a lockscreen or goes to sleep due to inactivity while the upload is occurring (Mouse Jiggler software can prevent that occurance).