Summary

Each Tuesday morning we perform scheduled maintenance on my.wbs. Normally this is during the period 07.00 - 08.15, in order to match with wider University and ISP ‘at risk’ periods.

Today, May 18th, the scheduled maintenance overran until 09.14.

We would like to apologise for this outage. The following is a brief description of what went wrong and what we are doing to try to prevent it happening again.

Impact

Anyone trying to access my.wbs during the period was unable to visit the site and was redirected to mini-my.wbs.ac.uk instead. The latter site was not affected by this outage.

Root cause

In addition to our regular maintenance we also used this morning to replace one of file servers that stores information required by my.wbs. It was important we replaced this server to maintain our security.

As part of this replacement process we needed to perform a final synchronisation of data between the old and new servers. When performed on test servers this completed within the time we would have available during our regular maintenance window. When this process was started this morning we quickly realised this was going much more slowly than expected. We traced the cause to an additional protection we have in place for the production environment that limits bandwidth available to prevent us accidentally damaging performance during back-end works. This misconfiguration was identified, resolved and the syncronisation restarted. This led to the entire synchronisation taking longer than anticipated.

Actions to reduce the risk of a re-occurrence

1. We recognise that we need to improve our documenting of differences between our test and production environments
2. Risky changes should always be reviewed by a colleague to maximise our chances of anticipating issues