Missing dependencies from mail/mailman3

Leave a Comment / Mail, Mailman / By

This started off as a migration. It went down a rabbit hole of incorrect and undeclared dependencies. I’m going to post this as documentation for the required ports update.

I’m migrating from Mailman 2.1 to Mailman 3 not because I want to but because Python 2.7 is deprecated. I’m sick of seeing these monitoring messages:

jail: ioc-mailman_example_org
mailman-2.1.34: Tag: expiration_date Value: 2020-12-31
python27-2.7.18_1: Tag: expiration_date Value: 2020-12-31
mailman-2.1.34: Tag: deprecated Value: Uses Python 2.7 which is EOLed upstream
python27-2.7.18_1: Tag: deprecated Value: EOLed upstream

In this post:

  1. FreeBSD 12.1
  2. mailman-2.1.34 (source)
  3. py37-mailman-3.3.1 (destination)

Snapshots

I’m taking a snapshot of this FreeBSD iocage jail:

[dan@zuul:~] $ zfs list | grep mailman
system/iocage/jails/mailman.example.org          5.38G  62.2G    92K  /iocage/jails/mailman.example.org
system/iocage/jails/mailman.example.org/root     5.38G  62.2G  5.16G  /iocage/jails/mailman.example.org/root
[dan@zuul:~] $ sudo zfs snapshot -r system/iocage/jails/mailman.example.org@mailman-2.1.34 
[dan@zuul:~] $

I am sure not to get any emails during this time, but to be sure you could turn off your incoming mail. I won’t be.

Stopping stuff

I made sure stuff was stopped, just because.

$ sudo service mailman stop
Stopping mailman.
Waiting for PIDS: 11430.
$ sudo sysrc mailman_enable="NO"
mailman_enable: YES -> NO

$ sudo service apache24 stop
Stopping apache24.
Waiting for PIDS: 11460.
$ sudo sysrc apache24_enable="NO"
apache24_enable: YES -> NO

Both mailman and apache.

Installing

This is what I did:

sudo pkg install py37-mailman

Then, to get rid of packages no longer required:

$ sudo pkg autoremove
Checking integrity... done (0 conflicting)
Deinstallation has been requested for the following 4 packages:

Installed packages to be REMOVED:
	libiconv: 1.16
	py27-dnspython: 1.16.0
	py27-setuptools: 44.0.0
	python27: 2.7.18_1

Number of packages to be removed: 4

The operation will free 77 MiB.

Proceed with deinstalling packages? [y/N]: y
[mailman.unixathome.org] [1/4] Deinstalling py27-dnspython-1.16.0...
[mailman.unixathome.org] [1/4] Deleting files for py27-dnspython-1.16.0: 100%
[mailman.unixathome.org] [2/4] Deinstalling py27-setuptools-44.0.0...
[mailman.unixathome.org] [2/4] Deleting files for py27-setuptools-44.0.0: 100%
[mailman.unixathome.org] [3/4] Deinstalling libiconv-1.16...
[mailman.unixathome.org] [3/4] Deleting files for libiconv-1.16: 100%
[mailman.unixathome.org] [4/4] Deinstalling python27-2.7.18_1...
[mailman.unixathome.org] [4/4] Deleting files for python27-2.7.18_1: 100%

Run stuff as mailman

From the docs:

This, and in general all, mailman commands should be run as the Mailman user and not as root. Running some of these commands as root can create files owned by root that can’t be read by the Mailman user.

Keep that in mind as you read this.

But you can’t just do this:

$ sudo su -l mailman
This account is currently not available.

You could, if you changed the user shell.

WARNING: before you do this, make sure you already have root access and not via sudo.

$ grep mailman /etc/passwd
mailman:*:91:91:Mailman User:/usr/local/mailman:/usr/sbin/nologin

I may just try that.

$ sudo pw usermod mailman -s /usr/local/bin/bash 
$ grep mailman /etc/passwd
mailman:*:91:91:Mailman User:/usr/local/mailman:/usr/local/bin/bash

OK, let’s try it again:

$ sudo su -l mailman
sudo: you do not exist in the passwd database
$ su
su: who are you?
$ grep dan /etc/passwd
dan:*:1002:1002:Dan Langille:/usr/home/dan:/bin/sh

To solve this, I got onto the host system, and went into the jail console:

$ sudo iocage console mailman
#

Next, I ran vipw, then typed :wq and saw:

vipw: password list updated

That update ran pwd_mkdb, which I have rarely run manually.

Now, back in the jail, I tried again:

$ sudo su -l mailman
[mailman@mailman ~]$

Another snapshot

[dan@zuul:~] $ sudo zfs snapshot -r system/iocage/jails/mailman.unixathome.org@py37-mailman-3.3.1-before-first-migration
[dan@zuul:~] $

Migrate the list[s]

I did this for one list, it worked, then I did it for every other list.

[mailman@mailman ~]$ mailman import21 biking@lists.langille.org /usr/local/mailman/lists/biking/config.pck
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 583, in _build_master
    ws.require(__requires__)
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 900, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 791, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (zope.interface 4.6.0 (/usr/local/lib/python3.7/site-packages), Requirement.parse('zope.interface>=5.0'), {'mailman'})

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/mailman", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3251, in <module>
    @_call_aside
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3235, in _call_aside
    f(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3264, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 585, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 598, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 786, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'zope.interface>=5.0' distribution was not found and is required by mailman
[mailman@mailman ~]$

Well, isn’t that just grand. :/

$ pkg info -x zop
py37-zope.component-4.2.2
py37-zope.configuration-4.1.0
py37-zope.event-4.1.0
py37-zope.i18nmessageid-3.6.1_1
py37-zope.interface-4.6.0
py37-zope.schema-4.2.2_1

A known issue so it seems.

Attempts to get around this

I modified /usr/local/lib/python3.7/site-packages/mailman-3.3.1-py3.7.egg-info/requires.txt to change:

zope.interface>=5.0

to

zope.interface>=4.0

Then I tried again:

[mailman@mailman ~]$ mailman import21 biking@lists.langille.org /usr/local/mailman/lists/biking/config.pck
Traceback (most recent call last):
  File "/usr/local/bin/mailman", line 6, in 
    from pkg_resources import load_entry_point
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3251, in 
    @_call_aside
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3235, in _call_aside
    f(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3264, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 583, in _build_master
    ws.require(__requires__)
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 900, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 786, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'gunicorn' distribution was not found and is required by mailman
[mailman@mailman ~]$

Well, that’s intriguing.

Oh wait, I need to have mailman running first.

Starting mailman

Let’s try starting mailman.

$ sudo sysrc mailman_enable="YES"
mailman_enable: NO -> YES

$ sudo service mailman start
/usr/local/etc/rc.d/mailman: WARNING: /usr/local/mailman/etc/mailman.cfg is not readable.

Oh, it is not?

$ ls -l  /usr/local/mailman/etc/mailman.cfg
ls: /usr/local/mailman/etc/mailman.cfg: No such file or directory

$ cd  /usr/local/mailman/etc
bash: cd: /usr/local/mailman/etc: No such file or directory

WTF is going on here?

We do have this file:

$ pkg info -l py37-mailman | grep mailman.cfg
	/usr/local/lib/python3.7/site-packages/mailman/config/mailman.cfg

But what’s this:

$ grep cfg mailman
# mailman_config (path):	Set to /usr/local/mailman/etc/mailman.cfg
: ${mailman_config="/usr/local/mailman/etc/mailman.cfg"}
required_files="/usr/local/${name}/etc/${name}.cfg"

The file is required, but can be set to something else? OK, I’ll play your silly game:

$ sudo sysrc mailman_config=/usr/local/lib/python3.7/site-packages/mailman/config/mailman.cfg
mailman_config:  -> /usr/local/lib/python3.7/site-packages/mailman/config/mailman.cfg

$ sudo service mailman start
/usr/local/etc/rc.d/mailman: WARNING: /usr/local/mailman/etc/mailman.cfg is not readable.

OK, time to hack on /usr/local/etc/rc.d/mailman.

Let’s change this line in that file:

required_files="${mailman_config}"

And try starting:

$ sudo service mailman start
Traceback (most recent call last):
  File "/usr/local/bin/mailman", line 6, in 
    from pkg_resources import load_entry_point
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3251, in 
    @_call_aside
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3235, in _call_aside
    f(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3264, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 583, in _build_master
    ws.require(__requires__)
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 900, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 786, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'gunicorn' distribution was not found and is required by mailman

OK, installing gunicorn.

sudo pkg install py37-gunicorn

Trying again:

$ sudo service mailman start
Traceback (most recent call last):
  File "/usr/local/bin/mailman", line 6, in 
    from pkg_resources import load_entry_point
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3251, in 
    @_call_aside
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3235, in _call_aside
    f(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3264, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 583, in _build_master
    ws.require(__requires__)
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 900, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 786, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'importlib_resources>=1.1.0' distribution was not found and is required by mailman

Next, I build and installed devel/py-importlib-resources as sudo pkg install py37-importlib-resources.

Next up was 

pkg_resources.DistributionNotFound: The 'authres>=1.0.1' distribution was not found and is required by mailman

Do you ever get the feeling you are the first time to try something?

Next: mail/py-authres to build py37-authres-1.2.0

Next was authheaders>=0.9.2, which is not in the tree.

I built mail/py-authheaders.

Next:

$ sudo service mailman start
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/authheaders/dmarc_lookup.py", line 30, in 
    from publicsuffix2 import PublicSuffixList
ModuleNotFoundError: No module named 'publicsuffix2'

I built dns/py-publicsuffix2 and then installed py37-publicsuffix2-2.20191221_8.

Next: 

ModuleNotFoundError: No module named 'dkim'

It’s as if nobody ever tested this port.

Let’s try mail/py-dkimpy

sudo pkg install py37-dkimpy

SUCCESS!

$ sudo service mailman start
Starting Mailman's master runner
Generating MTA alias maps

Here is what I see:

$ ps auwwx | grep mailman
mailman 48167  0.0  0.4 83344 69944  -  IsJ  23:04    0:01.29 /usr/local/bin/python3.7 /usr/local/bin/master -C /usr/local/lib/python3.7/site-packages/mailman/config/mailman.cfg
mailman 48173  0.0  0.4 81664 69964  -  SJ   23:04    0:01.29 /usr/local/bin/python3.7 /usr/local/bin/runner -C /usr/local/lib/python3.7/site-packages/mailman/config/mailman.cfg --runner=archive:0:1
mailman 48174  0.0  0.4 83436 71524  -  IJ   23:04    0:01.36 /usr/local/bin/python3.7 /usr/local/bin/runner -C /usr/local/lib/python3.7/site-packages/mailman/config/mailman.cfg --runner=bounces:0:1
mailman 48175  0.0  0.4 81664 69996  -  SJ   23:04    0:01.30 /usr/local/bin/python3.7 /usr/local/bin/runner -C /usr/local/lib/python3.7/site-packages/mailman/config/mailman.cfg --runner=command:0:1
mailman 48176  0.0  0.4 81664 69984  -  SJ   23:04    0:01.29 /usr/local/bin/python3.7 /usr/local/bin/runner -C /usr/local/lib/python3.7/site-packages/mailman/config/mailman.cfg --runner=in:0:1
mailman 48177  0.0  0.4 83880 70136  -  SJ   23:04    0:01.29 /usr/local/bin/python3.7 /usr/local/bin/runner -C /usr/local/lib/python3.7/site-packages/mailman/config/mailman.cfg --runner=lmtp:0:1
mailman 48178  0.0  0.4 81664 69996  -  SJ   23:04    0:01.29 /usr/local/bin/python3.7 /usr/local/bin/runner -C /usr/local/lib/python3.7/site-packages/mailman/config/mailman.cfg --runner=nntp:0:1
mailman 48179  0.0  0.4 81664 70048  -  SJ   23:04    0:01.29 /usr/local/bin/python3.7 /usr/local/bin/runner -C /usr/local/lib/python3.7/site-packages/mailman/config/mailman.cfg --runner=out:0:1
mailman 48180  0.0  0.4 81664 69980  -  SJ   23:04    0:01.29 /usr/local/bin/python3.7 /usr/local/bin/runner -C /usr/local/lib/python3.7/site-packages/mailman/config/mailman.cfg --runner=pipeline:0:1
mailman 48181  0.0  0.5 88560 76412  -  SJ   23:04    0:01.39 python3.7: gunicorn: master [gunicorn] (python3.7)
mailman 48182  0.0  0.4 81664 69984  -  IJ   23:04    0:01.29 /usr/local/bin/python3.7 /usr/local/bin/runner -C /usr/local/lib/python3.7/site-packages/mailman/config/mailman.cfg --runner=retry:0:1
mailman 48183  0.0  0.4 81664 69984  -  SJ   23:04    0:01.30 /usr/local/bin/python3.7 /usr/local/bin/runner -C /usr/local/lib/python3.7/site-packages/mailman/config/mailman.cfg --runner=virgin:0:1
mailman 48184  0.0  0.4 81664 69984  -  SJ   23:04    0:01.29 /usr/local/bin/python3.7 /usr/local/bin/runner -C /usr/local/lib/python3.7/site-packages/mailman/config/mailman.cfg --runner=digest:0:1
mailman 48229  0.0  0.5 88464 76456  -  SJ   23:04    0:00.01 python3.7: gunicorn: worker [gunicorn] (python3.7)
mailman 48230  0.0  0.5 88560 76440  -  SJ   23:04    0:00.01 python3.7: gunicorn: worker [gunicorn] (python3.7)
dan     48352  0.0  0.0 11376  2744  2  S+J  23:05    0:00.00 grep mailman

I’ll continue this in another blog post.

Website Pin Facebook Twitter Myspace Friendfeed Technorati del.icio.us Digg Google StumbleUpon Premium Responsive

Leave a Comment

Scroll to Top