Taking your nginx website offline for maintenance? I have an idea.

Leave a Comment / FreeBSD, nginx, Open Source / By /

From time to time, I need to take an nginx webserver or website offline for whatever reason. I might be migrating the database behind the website, the hardware might be powered off for work, etc.

In my case, these points might help you follow along with what I’m doing:

  • FreeBSD 15
  • nginx-1.28.2
  • there is an nginx proxy in front of the website – this nginx instance has no real content; it uses proxy_pass to get content from other nginx instances
  • I might want to take all the websites offline, or just one – so far, my solution is based on a website-by-website

Sample website configuration

This is a slightly simplified version of a website on the proxy nginx instance:

# As taken from http://kcode.de/wordpress/2033-nginx-configuration-with-includes

server {
  listen 10.0.0.29:80;
  listen 10.0.0.29:443 ssl;
  http2 on;

  # this has stuff like ssl_protocols, ssl_ciphers, etc
  include /usr/local/etc/nginx/includes/ssl-common.inc;

  # this server is a proxy for this host:
  server_name dev.freshports.org;

  # the logs - I should put ${server_name} in there:
  error_log  /var/log/nginx/dev.freshports.org.error.log  info;
  access_log /var/log/nginx/dev.freshports.org.access.log combined;

  # the certificate and key specific to this host. Again, ${server_name} might be useful here
  ssl_certificate     /usr/local/etc/ssl/dev.freshports.org.fullchain.cer;
  ssl_certificate_key /usr/local/etc/ssl/dev.freshports.org.key;

  # take the content from this backend server, which (otherwise) is not publicly available.
  location / {
    proxy_pass https://dev-freshports.int.unixathome.org/;

    include /usr/local/etc/nginx/includes/proxy_set_header.inc;
  }
}

Every proxy server looks like that. Just change the server name. Imagine file for both test.freshports.org and stage.freshports.org ; they look very similar.

Allowing for maintenance

I think my inspiration for this solution came from this ServerFault post.

This is what I have for test.freshports.org. I have already implemented the maintenance solution over here.

The added lines are highlighted.

# As taken from http://kcode.de/wordpress/2033-nginx-configuration-with-includes
server {
  listen 10.0.0.29:80;
  listen 10.0.0.29:443 ssl;
  http2 on;

  # this has stuff like ssl_protocols, ssl_ciphers, etc
  include /usr/local/etc/nginx/includes/ssl-common.inc;

  # this server is a proxy for this host:
  server_name test.freshports.org;

  # this directory is usually empty - content is served from here only if the site is in maintenance mode
  root        /usr/local/www/offline;

  # the logs - I should put ${server_name} in there:
  error_log  /var/log/nginx/test.freshports.org.error.log  info;
  access_log /var/log/nginx/test.freshports.org.access.log combined;

  # the certificate and key specific to this host. Again, ${server_name} might be useful here
  ssl_certificate     /usr/local/etc/ssl/test.freshports.org.fullchain.cer;
  ssl_certificate_key /usr/local/etc/ssl/test.freshports.org.key;

  error_page 503 @maintenance;

  # if this file exists, we are in maintenance mode
  if (-f $document_root/${server_name}-maintenance.html) {
    return 503;
  }

  # take the content from this backend server, which (otherwise) is not publicly available.
  location / {
    proxy_pass https://test-freshports.int.unixathome.org/;

    include /usr/local/etc/nginx/includes/proxy_set_header.inc;
  }

  # when in maintenance mode, provide this file to the user
  location @maintenance {
    rewrite ^(.*)$ /${server_name}-maintenance.html break;
  }

}

The first set of new lines looks for a file on disk – if it exists, the site is in maintenance mode.

Putting a site into maintenance mode

To put test.freshports.org into maintenance mode, I create this file:

[12:45 r720-02-proxy01 dvl /usr/local/www/offline] % cat test.freshports.org-maintenance.html     
<html>
<head>
<title>Error 503 Service Unavailable</title>
</head>
<body>
<h1>503 Service Unavailable</h1>
Server is offline for maintenance.
</body>
</html>
</pre>

When that file exists, the backend website is never contacted and only the above file is presented to the user, regardless of what content they are requesting.

To take the host out of maintenance, remove the file. Or rename it, say to test.freshports.org-maintenance.html.disabled

Future work

I want to make more use of ${server_name} in my proxy hosts.

I want a script to create the *-maintenance.html and include:

  1. The time the site went into maintenance
  2. The expected maintenance duration

I would also like to include a Retry-After header, but I think that requires more than just simple HTML.

Add an option to take all the websites offline at once. Perhaps with a all-sites.html file.

The future is now: Using more variables and global off-line

I added this section after first publishing the post.

Here is the configuration for dev.freshports.org which uses more ${server_name} and allows me to take all the websites offline. Which would be great right now, because the power is out at home (they are replacing poles nearby). Inspiration for the global solution came from Nginx implements the AND, OR multiple judgment in the IF statement.

If either files exists, it goes into maintenance mode:

  1. ${document_root}/${server_name}-maintenance.html
  2. $document_root/site-wide-maintenance.html

The site-wide file always takes priority, in terms of what is displayed to the user.

The newly added code (highlighted below) checks for each file, and then displays the last-one found to the user.

# As taken from http://kcode.de/wordpress/2033-nginx-configuration-with-includes

include /usr/local/etc/nginx/includes/blocked_IP.inc;

server {
  listen 10.0.0.29:80;
  listen 10.0.0.29:443 ssl;
  http2 on;

  # this has stuff like ssl_protocols, ssl_ciphers, etc
  include /usr/local/etc/nginx/includes/ssl-common.inc;

  # this server is a proxy for this host:
  server_name dev.freshports.org;

  # this directory is usually empty - content is served from here only if the site is in maintenance mode
  root        /usr/local/www/offline;

  # the logs
  error_log  /var/log/nginx/${server_name}.error.log  info;
  access_log /var/log/nginx/${server_name}.access.log combined;

  # the certificate and key specific to this host.
  ssl_certificate     /usr/local/etc/ssl/${server_name}.fullchain.cer;
  ssl_certificate_key /usr/local/etc/ssl/${server_name}.key;

  # The first of the maintenance code.
  error_page 503 @maintenance;

  set $maint '';
  if (-f ${document_root}/${server_name}-maintenance.html) {
    set $maint "${server_name}-maintenance.html";
  }

  if (-f $document_root/site-wide-maintenance.html) {
    set $maint "site-wide-maintenance.html";
  }

  # if either file exits, it's maintenance mode
  # site-wide always takes precedence
  if ($maint != '') {
    return 503;
  }
  # The last of the maintenance code.

  location / {
    proxy_pass https://dev-freshports.int.unixathome.org/;

    include /usr/local/etc/nginx/includes/proxy_set_header.inc;
  }

  # but wait, there's more!
  location @maintenance {
    rewrite ^(.*)$ /${maint} break;
  }

}

I’m confident that this is not the most efficient way to do this (if you were serving millions of pages a day, for example). And if you’re doing that, you have much better proxies in place than I have.

Hope this helps. I’m looking forward to updating Ansible (when the power comes back one) and deploying these changes for all the websites.

Website Pin Facebook Twitter Myspace Friendfeed Technorati del.icio.us Digg Google StumbleUpon Premium Responsive

Related Posts

Leave a Comment

Scroll to Top