Turning Mailman 2.1 into static archives, at the same URLs

I have been running Mailman 2.1 since at least 2006. It is long past end of life, it still needs Python 2.7, and the lists it carries have all gone quiet. Some moved elsewhere (BSDCan went in April 2024), the rest just stopped. Time to turn it off.

The one thing I did not want to lose was the archives. Lots of old links point at them, from mailing list posts, from blog posts, and from search engines. The first step of decommissioning was therefore to replace Mailman with plain static archives, at exactly the same URLs.

What was there

The host is mailman, a FreeBSD 15.1 jail running mailman-2.1.39_4, python27 and Apache 2.4. It served lists on 7 hostnames, each hostname on its own IP address:

Those bsdcan mailing lists… you can’t get to them. That hostname doesn’t resolve to this host. You’d need to jigger your DNS.

Mailman keeps each list's settings in a Python pickle, so you cannot just grep them. Mailman's own config_list -o – prints a list's settings as Python assignments, one per line. This loop runs it for every list and keeps the two settings I needed: the hostname the list belongs to, and whether its archive is private.

M=/usr/local/mailman
for l in $(ls $M/lists); do
  printf "%-24s " $l
  $M/bin/config_list -o - $l | grep -E "^(host_name|archive_private) " | tr "\n" " "
  echo
done

A few of those hostnames no longer resolve at all (lists.phorum.org, lists.nz.freebsd.org), and lists.bsdcan.org already points at another server. Those archives were still reachable, because Mailman serves /pipermail/ from one shared directory on every hostname: lists.freshports.org/pipermail/pgcon-announce/ works just as well as the "right" URL. I kept that behaviour, so nothing that works today breaks.

The archives are already static

This is what makes the job easy. Pipermail, the Mailman 2 archiver, writes plain HTML files. Mailman only gets involved in three places:

• /pipermail/<list>/ is an Apache Alias to archives/public/, which is a set of symlinks into archives/private/. No Mailman code at all.
• /mailman/private/<list>/ is a CGI which checks the member's list password, then serves those same HTML files.
• /mailman/listinfo/<list> is a CGI. Every archive page links to it ("More information about the … mailing list").

So the static site needs a copy of the HTML, a static page for each listinfo URL, and something to replace the password check on the private archives.

Private archives: keep the passwords

freebsddiary-announce was private, but it was an announcement list, so I made it public. The other private archives stay private, at the same URLs. Apache basic auth does that. The question was who gets a login.

Mailman 2.1 keeps member passwords in clear text in each list's config.pck. That is a terrible idea in general, but here it is useful: everyone can keep their existing login. A small script, run as root, reads each private list's members and pipes each password into htpasswd -i -B, so they become bcrypt hashes and never appear on a command line. Mailman passwords are per list, so one address can have a different password on each list. That means one htpasswd file per list, not one for the whole site.

Each private list gets this:

<Directory "/usr/local/www/mailman-archives/private/biking">
  AuthType Basic
  AuthName "biking archives"
  AuthBasicProvider file
  AuthUserFile "/usr/local/etc/apache24/mailman-archives/htpasswd/biking"
  Require valid-user
</Directory>

The same goes for biking.mbox/, so the full mbox download still works, behind the same login.

The rest of the URLs

The shared include, which replaces mailman.common.conf.include, is short:

Alias /pipermail "/usr/local/www/mailman-archives/pipermail"

# Mailman also served public lists to members under /mailman/private/
RedirectMatch 301 "^/mailman/private/(bsdcan-announce|...|racingsystem-help)(/.*)?$" "/pipermail/$1$2"

Alias /mailman/private "/usr/local/www/mailman-archives/private"
Include "/usr/local/etc/apache24/mailman-archives/private.conf"

AliasMatch "^/mailman/listinfo/([^/]+)/?$" "/usr/local/www/mailman-archives/listinfo/$1.html"

# the rest of the Mailman web interface is gone
RedirectMatch 301 "^/mailman/?$" "/mailman/listinfo"
RedirectMatch 410 "^/mailman/(?!listinfo|private/)"

Each listinfo page is generated from the list's own description and info text. It says the list is closed, and links to the archives. Each vhost also gets its own /mailman/listinfo index, listing only the advertised lists on that hostname, which is what Mailman showed.

Everything else under /mailman/ (admin, subscribe, options, roster, confirm) now returns 410 Gone.

Three scripts

• build-archives.sh uses rsync --delete to copy the archives into /usr/local/www/mailman-archives, skipping pipermail's database/ and pipermail.pck. Public lists are exactly the symlinks in archives/public. It is safe to re-run.
• gen-config.py reads the list configs (read only, with MailList(name, lock=0)) and writes the listinfo pages and all the Apache config. A short MAKE_PUBLIC list in it overrides Mailman's private setting, which is how freebsddiary-announce became public.
• make-htpasswd.py writes the htpasswd files described above.

The result is 245 MB:

$ du -sh /usr/local/www/mailman-archives/*
 14K    /usr/local/www/mailman-archives/listinfo
 21M    /usr/local/www/mailman-archives/pipermail
224M    /usr/local/www/mailman-archives/private

Side by side, then a DNS switch

I did not touch the Mailman vhosts. The static site went onto a new IP address, 162.208.116.67, with the same hostnames and the same certificates. Before changing any DNS, I tested every URL against the new address with curl --resolve:

curl -s -o /dev/null -w '%{http_code}\n' \
  --resolve lists.langille.org:443:162.208.116.67 \
  https://lists.langille.org/mailman/private/biking/

That caught one bug. With no password, a private archive gave 401, as it should. With a wrong password, it gave 500. The error log said:

AH01620: Could not open password file: /usr/local/etc/apache24/mailman-archives/htpasswd/biking

The files were root:www 0640, but my script had created their directory as root:wheel 0750, so www could not get into it. Apache only opens the file once it has a password to check, which is why the no-password case looked fine. A chgrp www on the directory fixed it, and the script now sets that itself.

Then the switch. Every hostname became a CNAME to a new name, archived.unixathome.org, which points at the new address. I did this with nsupdate against the hidden master. Each zone has a prereq, so the update only applies if the record is still what I saw when I wrote the file:

server dns-hidden-master.int.unixathome.org

zone unixathome.org
prereq nxdomain archived.unixathome.org
update add archived.unixathome.org 600 IN A 162.208.116.67
send

zone pgcon.org
prereq yxrrset lists.pgcon.org IN A 162.208.116.82
update delete lists.pgcon.org A
update delete lists.pgcon.org MX
update add lists.pgcon.org 300 IN CNAME archived.unixathome.org.
send

... and the same for the other five zones

I also wrote the reverse file first, putting back the old A and MX records, just in case. I have not needed it.

Note the update delete … MX. A CNAME cannot share a name with any other record, so the MX records had to go. That is fine here: the point is to stop mail. With no MX, mail falls back to the address of the name, and nothing listens on port 25 at 162.208.116.67, so senders get connection refused and eventually bounce.

Once DNS had settled, I moved the old vhost files out of Includes/ and did a graceful restart. Apache now listens only on *:80 and *:443, and every hostname answers from the static site:

for h in lists.freebsddiary.org lists.freshports.org lists.freshsource.org \
         lists.langille.org lists.pgcon.org lists.racingsystem.com lists.unixathome.org; do
  printf "%-24s %s\n" $h "$(curl -s -o /dev/null -w '%{remote_ip} %{http_code}' https://$h/mailman/listinfo)"
done

lists.freebsddiary.org   162.208.116.67 200
lists.freshports.org     162.208.116.67 200
lists.freshsource.org    162.208.116.67 200
lists.langille.org       162.208.116.67 200
lists.pgcon.org          162.208.116.67 200
lists.racingsystem.com   162.208.116.67 200
lists.unixathome.org     162.208.116.67 200

What's left

• Stop Mailman, and remove the lists.* domains from Postfix.
• Keep a tarball of the Mailman archives/ and lists/ directories offline, then remove the package. The build scripts need Mailman, so any final rebuild has to happen first.
• The certificates for these names are issued by acme.sh on another host. If that host uses a DNS challenge, the CNAMEs make no difference. I will know for sure in mid-November, when they renew.
• Eventually, pkg delete mailman && pkg autoremove
• Adjust monitoring to remove items no longer installed (postfix, mailman)
• Configure dma for mail delivery from this jail to the host for eventual relay to me

After that, mailman is just an Apache server with some HTML files on it. That is much easier to look after than a Python 2 application.