Tuesday Morning Maintenance
Scheduled Maintenance Report for Warwick Business School
Postmortem

Summary

Each Tuesday morning we perform scheduled maintenance on my.wbs. Normally this is during the period 07.00 - 08.15, in order to match with wider University and ISP ‘at risk’ periods.

Today, May 18th, the scheduled maintenance overran until 09.14.

We would like to apologise for this outage. The following is a brief description of what went wrong and what we are doing to try to prevent it happening again.

Impact

Anyone trying to access my.wbs during the period was unable to visit the site and was redirected to mini-my.wbs.ac.uk instead. The latter site was not affected by this outage.

Root cause

In addition to our regular maintenance we also used this morning to replace one of file servers that stores information required by my.wbs. It was important we replaced this server to maintain our security.

As part of this replacement process we needed to perform a final synchronisation of data between the old and new servers. When performed on test servers this completed within the time we would have available during our regular maintenance window. When this process was started this morning we quickly realised this was going much more slowly than expected. We traced the cause to an additional protection we have in place for the production environment that limits bandwidth available to prevent us accidentally damaging performance during back-end works. This misconfiguration was identified, resolved and the syncronisation restarted. This led to the entire synchronisation taking longer than anticipated.

Actions to reduce the risk of a re-occurrence

  1. We recognise that we need to improve our documenting of differences between our test and production environments
  2. Risky changes should always be reviewed by a colleague to maximise our chances of anticipating issues
Posted May 18, 2021 - 17:17 BST

Completed
The scheduled maintenance has been completed. Apologies that this took longer than expected.
Posted May 18, 2021 - 09:13 BST
Update
We're doing final checks now.
Posted May 18, 2021 - 09:09 BST
Update
Apologies for the continued disruption. We are migrating some file servers (part of an essential upgrade package of work) which is taking longer than expected. We will keep you updated.
Posted May 18, 2021 - 08:46 BST
Update
The scheduled maintenance will take a little longer this morning. We'll keep you updated.
Posted May 18, 2021 - 08:13 BST
In progress
Scheduled maintenance is currently in progress. We will provide updates as necessary.
Posted May 18, 2021 - 07:00 BST
Scheduled
We use this opportunity to do some routine maintenance and are normally finished by 08:15 Warwick time.
Posted Apr 01, 2021 - 17:13 BST
This scheduled maintenance affected: my.wbs : Full login check and my.wbs : Site availability check.