Nov 152017
 

Today I moved the FreshPorts website from one server to another. My goal is for nobody to notice.

In preparation for this move, I have:

  1. DNS TTL reduced to 60s
  2. Posted to Twitter
  3. Updated the status page
  4. Put the website put in offline mode:
    Sorry, the website is offline from 2017-11-05 17:01, for approximately 3 hours.
    
    It is now 2017-11-05 17:01.

    That works for both http and https:

    [dan@air01:~] $ wget -S http://www.freshports.org
    --2017-11-05 12:02:15--  http://www.freshports.org/
    Resolving www.freshports.org... 206.127.23.230
    Connecting to www.freshports.org|206.127.23.230|:80... connected.
    HTTP request sent, awaiting response... 
      HTTP/1.1 503 Service Unavailable
      Date: Sun, 05 Nov 2017 17:02:17 GMT
      Server: Apache/2.4.27 (FreeBSD) PHP/5.6.32 OpenSSL/1.0.2m
      X-Powered-By: PHP/5.6.32
      Content-Length: 149
      Connection: close
      Content-Type: text/html; charset=UTF-8
    2017-11-05 12:02:17 ERROR 503: Service Unavailable.
    
    [dan@air01:~] $ wget -S https://www.freshports.org
    --2017-11-05 12:02:23--  https://www.freshports.org/
    Resolving www.freshports.org... 206.127.23.230
    Connecting to www.freshports.org|206.127.23.230|:443... connected.
    HTTP request sent, awaiting response... 
      HTTP/1.1 503 Service Unavailable
      Date: Sun, 05 Nov 2017 17:02:23 GMT
      Server: Apache/2.4.27 (FreeBSD) PHP/5.6.32 OpenSSL/1.0.2m
      X-Powered-By: PHP/5.6.32
      Content-Length: 149
      Connection: close
      Content-Type: text/html; charset=UTF-8
    2017-11-05 12:02:23 ERROR 503: Service Unavailable.
    
    [dan@air01:~] $ 

NOTE: those wget commands were issued on my laptop which was on EST not UTC.

It took a while to get the website to respond with 503, but now that I have that, I hope to make it a PRESS THIS BUTTON feature.

Now I’m ready to start the transfer.

What was missed

I turned off commit processing on the new server, but I did not do this on the old server. I should have:

sudo svc -d /var/service/freshports

That stops processing of incoming commits. No data is lost, but it keeps the two databases at the same spot in history. Commit processing could continue during the database dumping, but that does not affect the dump, which will be consistent regardless.

The offline code

Here is the basic stuff I used to put the website into offline mode. The main points are:

  1. header(“HTTP/1.1 503 Service Unavailable”);
  2. ErrorDocument 404 /index.php

I move the DocumentRoot to a new directory, containing only index.php. Every error invokes index.php, which returns a 503 code.

The dump

The database dump just started (Sun Nov 5 17:07:22 UTC 2017).

root@pg96:~ # /usr/bin/time pg_dump -h 206.127.23.226 -Fc -U dan freshports.org > freshports.org.9.6.dump

That should take about 30 minutes. I have set a timer to remind me.

Total time was:

1464.82 real      1324.96 user        37.22 sys

The MD5 is:

MD5 (freshports.org.9.6.dump) = 5249b45a93332b8344c9ce01245a05d5

It is now: Sun Nov 5 17:34:07 UTC 2017

The rsync

The rsync should take about 10-20 minutes. I have already done an rsync of yesterday’s dump file. The rsync today should copy over only the deltas (i.e. differences).

The rsync started at about Sun Nov 5 17:36:05 UTC 2017

That took 2m9.091s

The MD5 matches.

The restore

The restore should take about 30 minutes. I ran this test yesterday.

It is now Sun Nov 5 17:40:03 UTC 2017.

$ createdb -T template0 -E SQL_ASCII freshports.testing
$ time pg_restore -j 16 -d freshports.testing freshports.org.9.6.dump

Done.

real    25m21.108s
user    1m57.508s
sys     0m15.172s

It is now Sun Nov 5 18:06:22 UTC 2017.

Insert break here

About here, I took a 30 minute break to run an errand. It was worth it.

Changing DNS

I’m ready to change DNS now.

It is Sun Nov 5 19:49:20 EST 2017

Done.

And nearly immediately, traffic started.

How many misses?

During this process, XXXXX requests were declined:

$ grep -c '" 503 ' /usr/websites/log/freshports.org-access.log
XXXXX

That’s it, we’re done

Total elapsed time: 1 hour 48 minutes.

There are still a number of things to follow up on, but that was the transfers.

Website Pin Facebook Twitter Myspace Friendfeed Technorati del.icio.us Digg Google StumbleUpon Premium Responsive