Today I moved the FreshPorts website from one server to another. My goal is for nobody to notice.
In preparation for this move, I have:
- DNS TTL reduced to 60s
- Posted to Twitter
- Updated the status page
- Put the website put in offline mode:
Sorry, the website is offline from 2017-11-05 17:01, for approximately 3 hours. It is now 2017-11-05 17:01.
That works for both http and https:
[dan@air01:~] $ wget -S http://www.freshports.org --2017-11-05 12:02:15-- http://www.freshports.org/ Resolving www.freshports.org... 206.127.23.230 Connecting to www.freshports.org|206.127.23.230|:80... connected. HTTP request sent, awaiting response... HTTP/1.1 503 Service Unavailable Date: Sun, 05 Nov 2017 17:02:17 GMT Server: Apache/2.4.27 (FreeBSD) PHP/5.6.32 OpenSSL/1.0.2m X-Powered-By: PHP/5.6.32 Content-Length: 149 Connection: close Content-Type: text/html; charset=UTF-8 2017-11-05 12:02:17 ERROR 503: Service Unavailable. [dan@air01:~] $ wget -S https://www.freshports.org --2017-11-05 12:02:23-- https://www.freshports.org/ Resolving www.freshports.org... 206.127.23.230 Connecting to www.freshports.org|206.127.23.230|:443... connected. HTTP request sent, awaiting response... HTTP/1.1 503 Service Unavailable Date: Sun, 05 Nov 2017 17:02:23 GMT Server: Apache/2.4.27 (FreeBSD) PHP/5.6.32 OpenSSL/1.0.2m X-Powered-By: PHP/5.6.32 Content-Length: 149 Connection: close Content-Type: text/html; charset=UTF-8 2017-11-05 12:02:23 ERROR 503: Service Unavailable. [dan@air01:~] $
NOTE: those wget commands were issued on my laptop which was on EST not UTC.
It took a while to get the website to respond with 503, but now that I have that, I hope to make it a PRESS THIS BUTTON feature.
Now I’m ready to start the transfer.
What was missed
I turned off commit processing on the new server, but I did not do this on the old server. I should have:
sudo svc -d /var/service/freshports
That stops processing of incoming commits. No data is lost, but it keeps the two databases at the same spot in history. Commit processing could continue during the database dumping, but that does not affect the dump, which will be consistent regardless.
The offline code
Here is the basic stuff I used to put the website into offline mode. The main points are:
- header(“HTTP/1.1 503 Service Unavailable”);
- ErrorDocument 404 /index.php
I move the DocumentRoot to a new directory, containing only index.php. Every error invokes index.php, which returns a 503 code.
The dump
The database dump just started (Sun Nov 5 17:07:22 UTC 2017).
root@pg96:~ # /usr/bin/time pg_dump -h 206.127.23.226 -Fc -U dan freshports.org > freshports.org.9.6.dump
That should take about 30 minutes. I have set a timer to remind me.
Total time was:
1464.82 real 1324.96 user 37.22 sys
The MD5 is:
MD5 (freshports.org.9.6.dump) = 5249b45a93332b8344c9ce01245a05d5
It is now: Sun Nov 5 17:34:07 UTC 2017
The rsync
The rsync should take about 10-20 minutes. I have already done an rsync of yesterday’s dump file. The rsync today should copy over only the deltas (i.e. differences).
The rsync started at about Sun Nov 5 17:36:05 UTC 2017
That took 2m9.091s
The MD5 matches.
The restore
The restore should take about 30 minutes. I ran this test yesterday.
It is now Sun Nov 5 17:40:03 UTC 2017.
$ createdb -T template0 -E SQL_ASCII freshports.testing $ time pg_restore -j 16 -d freshports.testing freshports.org.9.6.dump
Done.
real 25m21.108s user 1m57.508s sys 0m15.172s
It is now Sun Nov 5 18:06:22 UTC 2017.
Insert break here
About here, I took a 30 minute break to run an errand. It was worth it.
Changing DNS
I’m ready to change DNS now.
It is Sun Nov 5 19:49:20 EST 2017
Done.
And nearly immediately, traffic started.
How many misses?
During this process, XXXXX requests were declined:
$ grep -c '" 503 ' /usr/websites/log/freshports.org-access.log XXXXX
That’s it, we’re done
Total elapsed time: 1 hour 48 minutes.
There are still a number of things to follow up on, but that was the transfers.
