Extended Maintenance for Server Migrations

Currently, both the github.ncsu.edu application server and the backup server for the service are hosted out of the VMWare environment in Data Center One (DC1). There is an ongoing effort to migrate services out of DC1 and into the new Eastern Data Center (EDC) in order to decommission the DC1 data center. Therefore, we will be migrating the application server and backup server out of DC1 on Thursday, March 12th.

As a part of this migration we are physically separating the application server from the backup server so that in the case of a physical disaster at one of the data centers, the other is less likely to be affected. The backup server will be moving from DC1 to EDC, and the application server will be moved from DC1 to Data Center 2 (DC2). Since DC2 is physically closer to most of the campus infrastructure, the application server will have slightly less latency, and will be less likely to be isolated from campus if fiber lines are accidentally cut somewhere.

We have chosen the date of March 12th for the migration to be the least impactful to students and staff that use the service. The target date for decommissioning DC1 is June of 2020, and so Spring Break is the best time for changes considering the academic schedule between now and June. We wanted to avoid Friday, so that if anything goes sideways after the migration there will be more staff on-hand to diagnose and fix issues, both on-premises and at GitHub. Therefore, Thursday March 12 was chosen for the migration date.

Because of the amount of data that needs to move from DC1 to EDC and DC2, the migration period will take longer than the usual maintenance time. The estimate given was one hour per 250GB of data, and so with 2TB of allocated space it will take approximately eight hours to migrate. We have scheduled the maintenance window for nine hours, from 5:00PM to 2:00AM on March 12th.

In order to perform the migration in VMWare, the server needs to be offline during the migration. Instead of the usual 503 maintenance page in the web interface, the browser will instead fail to connect. Similarly, SSH connections to the service will fail.

Users that have automation connected to GitHub are advised to take these differences into account for this maintenance period. Automation that previously ran smoothly through upgrade maintenance may behave differently when the service is totally unavailable. Users are advised to pause/disable automation during the migration window if possible.

The maintenance is scheduled for March 12th, from 5:00PM to 2:00AM. Contact github@help.ncsu.edu if you have any questions or concerns.

EDIT – 3/12/2020: Updated maintenance window based on dry-run migration time.

https://sysnews.ncsu.edu/news/5e679c61